Commit Graph

955 Commits

Author SHA1 Message Date
Simon Michael
b7c4dc3b53 fix:journal: cyclic include error messages now show the correct line 2025-07-11 13:00:51 -07:00
Simon Michael
801a7adaa4 imp:include: better errors, eg for missing argument; more debug output 2025-07-11 12:13:52 -07:00
Simon Michael
c8a5b8eb37 dev: includedirectivep: cleanup 2025-07-11 12:13:12 -07:00
Simon Michael
f5d3b7bd38 fix:journal: include directive error messages now show the correct line
They were showing the line after the include directive, confusingly.
2025-07-11 11:55:29 -07:00
Simon Michael
2815a1865f dev: includedirectivep: cleanups, docs 2025-07-11 11:38:50 -07:00
Simon Michael
4a5775da71 imp: commodities/payees/tags: used/declared flags, like accounts
And general cleanup of options and help across
the accounts, commodities, payees, tags commands.
2025-06-15 07:32:40 -10:00
Simon Michael
de4d637def dev: transaction balancing: cleanup [#2402] 2025-06-12 17:55:13 -10:00
Simon Michael
bf90b20f2c imp: --txn-balancing flag to select transaction balancing precision [#2402] 2025-06-12 17:55:13 -10:00
Simon Michael
442bd24bc5 ;dev: commodityStyleFromRawOpts: cleanup 2025-06-12 17:55:13 -10:00
Simon Michael
e360e50497 imp:csv: more --debug=2 output for if rules
Also, in debug output show records more like what matchers are seeing,
ie with quotes removed.
2025-05-22 17:05:45 -10:00
Simon Michael
820a44eb07 imp:lib:Hledger.Utils.Debug: simpler, more consistent dbg* names 2025-05-21 22:54:00 -10:00
Simon Michael
2371f677e5 imp:journal: include directive now allows a same-line comment 2025-04-27 08:30:18 -10:00
samahri
4f9473240f cli: "balanced" check is enabled only with "check" command 2025-04-26 20:05:00 -10:00
Simon Michael
bf40d28c8b imp: config files (and maybe timedot files) no longer require a final newline 2025-04-25 13:59:18 -10:00
Simon Michael
fea127fcb5 ;doc: note check-related bug in rawOptsToInputOpts [#2377] 2025-04-24 10:44:34 -10:00
Simon Michael
98b40b2b0e ;dev: fix a warning 2025-04-22 12:26:55 -10:00
Simon Michael
9340b73aae imp: improve/format errors for various failures [#2367]
These now call error' and show errors in the standard style:

- reading a nonexistent data file
- reading an unsafe dotted file name on windows
- web: using --socket on windows
- demo: demo not found
- demo: error while running asciinema
- diff: bad arguments
- print --match: no match found
- register --match: no match found
- roi: no investment transactions found
2025-04-11 08:06:47 -10:00
Michael Rees
62071bc4c2 feat: timeclock: Add support for multiple clocked in sessions (#2141)
We now support having multiple sessions clocked in. These are paired by
account name if given on the out entry, and otherwise an out closes the
most recent in entry.

Note that this breaks some backwards compatibility, in that we
previously ignored the description on the clock out entry. To mitigate
this, a new hidden flag --timeclock-old has been added, which reverts to
the old behavior.
2025-04-03 11:19:18 -10:00
Simon Michael
133560aa93 ;dev: csv: no need to test for unsupported feature [#2352] 2025-03-12 20:40:36 -10:00
Thomas Miedema
a8a0d3ee30 fix: csv: fix regression in parsing rules containing & (#2352) 2025-03-12 20:35:59 -10:00
Dmitry Astapov
1fc7006919 run: cache input files by (iopts, name), allows commands with different iopts 2025-03-06 11:54:00 -10:00
Thomas Miedema
2faceb8e1b feat: csv: allow multiple matchers on the same line
`If blocks` and `If tables` now allow multiple matchers on the same line
separated by `&&` (AND) or `&& !` (AND NOT).

Example `if block` with two matchers on the same line:

	if %description amazon && %date 2025-02-22
	    account2 expenses:books

Example `if table` with two matchers on the same line:

	if,account2
	%description amazon && %date 2025-02-22, expenses:books
2025-03-01 11:21:00 -10:00
Joschua Kesper
5114962b2a feat:csv: add an encoding rule, allowing non-UTF8 CSV to be read [#2319]
Previously, hledger could read CSV files containing non-ascii
characters only if they are UTF8-encoded.  Now there is a new CSV
rule, encoding ENCODING, which allows reading CSV files with other
encodings.

This adds a dependency on the encoding library, which supports fewer
encodings than text-icu but does not require a third-party C library.
To avoid build issues on various platforms, we require version 0.10+.

This adds some use of the ImplicitParams language extension, required
by encoding's API, but only in a small code region.

This also changes the type of Reader's rReadFn; it now takes
a `Handle` rather than a `Text`, allowing more flexibility.
2025-02-15 14:48:30 -10:00
Simon Michael
39f3b2c7ba ;dev: doc 2025-01-31 02:04:40 -10:00
Simon Michael
f648903b37 imp:print:beancount: don't add account tags to postings
When print is generating beancount output, turn off the usual
inheritance of account tags by postings; it would generate excessive
metadata in the journal. Beancount can do or not do that kind of
inheritance itself.
2024-12-06 05:55:42 -10:00
Simon Michael
29349458b3 imp:csv:if: go back to accepting unknown csv field names [#2289]
It makes life easier when reusing common rules with different CSVs.
2024-12-04 17:45:28 -10:00
Simon Michael
99fc4cd61f imp:csv:if: show the problematic field name when warning 2024-12-04 16:47:01 -10:00
Simon Michael
054a204aa0 imp:csv:if: support & ! (AND NOT) 2024-12-03 17:25:43 -10:00
Simon Michael
3d55f260b3 imp:csv:if: warn on invalid csv field names; improve doc [#2289] 2024-12-03 16:07:57 -10:00
Simon Michael
a47dce073d dev:csv: refactor/document isBlockActive, matcherMatches 2024-12-03 13:51:32 -10:00
Simon Michael
1010e3dee6 imp:csv: improve debug=7 output from isBlockActive 2024-12-03 10:45:16 -10:00
Simon Michael
1ad9fbb4a8 imp:print: clean up special tags; show them more often with --verbose-tags
- These special hidden tags, used internally, have been renamed:
  - `_modified`           -> `_modified-transaction`
  - `_cost-matched`       -> `_cost-posting`
  - `_conversion-matched` -> `_conversion-posting`

- All special hidden tags now have a similarly-named visible tag,
  and `--verbose-tags` now shows those more often, which is useful
  when troubleshooting `--infer-equity`, `--infer-costs`,
  or the matching of redundant costs and conversion postings.

- The `generated-posting:` tag added by `--infer-equity` is now valueless.

- The `modified-transaction:` tag added by `--auto` now appears on its own line.
2024-11-16 14:40:03 -10:00
Simon Michael
445e80fd41 dev:clarify: rename jcommodities to jdeclaredcommodities 2024-11-02 15:52:17 -10:00
Simon Michael
5e0a35b1da fix:journal:P directives: require a space after the symbol [#2280]
This prevents surprising parses, like
`P 2024-10-31 a0 1` parsed as `P 2024-10-31 a 01`.
2024-11-02 15:24:25 -10:00
Simon Michael
c66e901d8b dev: save the parse positions of PriceDirectives 2024-11-02 15:00:47 -10:00
Simon Michael
4d38c63ec8 dev: move/rename nullsourcepos 2024-11-02 14:59:43 -10:00
Simon Michael
e44cbbf1a4 dev: refactor, clarify detection of cost/conversion postings 2024-10-20 09:13:58 -10:00
Simon Michael
b940254025 imp:cli: detect --color more safely in most cases
--color now also works in a config file, like --pager, except for two
cases: it does not affect colouring of debug output, or the colouring
helpers used in the check recentassertions error message.
2024-10-19 16:28:36 -10:00
Simon Michael
c92b601028 dev: fix warnings with ghc 9.10 / base 4.20
Older ghc versions should also still build cleanly (tested with 9.8 so far).

I don't like enabling CPP in so many modules but it's easier that
figuring out how to do it with base-compat; hopefully no noticeable
compilation impact.
2024-09-30 17:20:13 -10:00
Simon Michael
823be7c565 fix: csv: tags on following lines, and posting dates, also work now [#2241]
Follow-on work from #2214.
2024-09-28 18:54:43 -10:00
Simon Michael
b28468e651 dev: clarify some confusing comment parsers a bit [#2241] 2024-09-28 18:53:46 -10:00
Simon Michael
375fb07ede ;dev: cleanups 2024-08-29 10:07:02 +01:00
Henning Thielemann
14b5a1f82a imp: Hledger.Read.CsvUtils -> Write.Csv 2024-08-16 16:57:38 +02:00
Simon Michael
40620666f8 imp: cli: rename --rules-file to --rules; tweak options help
For brevity, and consistency with --conf.
--rules-file remains supported, as a hidden option.

hledger's main mode now supports the hidden legacy flags,
as the command modes do.
2024-06-25 18:37:55 +01:00
Simon Michael
969b5a89d1 dev: comment parsing tests/refactoring, fix indented timedot comments 2024-06-25 18:37:54 +01:00
Simon Michael
713c3f4067 imp: TimedotReader: trace parsing at debug level 9 2024-06-25 18:37:54 +01:00
Simon Michael
e1991be46f feat: config file: add a real parser, support command-specific options 2024-06-25 18:37:54 +01:00
Simon Michael
f5c2ec681c dev: refactor: merge Text.Megaparsec.Custom into Hledger.Utils.Parse 2024-06-25 18:37:54 +01:00
Simon Michael
07a4b21620 dev: refactor: move emptyorcommentlinep'; hlint 2024-06-25 18:37:54 +01:00
Simon Michael
605f8446e5 fix:pkg: fix a doctest failure with ghc 8.10 2024-05-17 15:08:26 -10:00
Simon Michael
e0f017f544 ;dev: journalFinalise: doc update 2024-05-08 12:40:06 -10:00
Simon Michael
b7e5c05da2 imp: -I can now be overridden by -s or the check command
This enables a "relaxed" workflow where you delay balance assertions
checking until strict mode is turned on: always run hledger -I, and
add -s when you're ready.
2024-05-07 15:44:06 -10:00
Simon Michael
243a96b656 dev: clarify how readJournalFiles defers checks, and use a dedicated flag 2024-05-02 21:36:32 -10:00
Simon Michael
ea58a7ccd7 imp: check: check ordereddates before balance assertions, when both are enabled 2024-05-02 21:11:04 -10:00
Simon Michael
490a46fcd2 fix: journal: parse include directives with trailing whitespace
[https://github.com/adept/full-fledged-hledger/issues/29]
2024-05-02 07:26:12 -10:00
Simon Michael
d3634cf4dd dev: update failng doctests 2024-04-08 00:08:48 -10:00
Simon Michael
fae6e49407 imp: simpler, clearer date parse error messages
When the error message repeated the invalid date at the end,
it was possible to misinterpret that as a suggested fix (reported in chat).

Instead, date errors (most of them) now rely on the highlighted data
excerpt above. This is also preferable since it shows the original
date as written, not a reconstruction with a possibly different format.

Should this be the policy for all error messages going forward ?
It would be easier.
Can we assume the data excerpt is always visible along with the error message ?
It isn't shown by flycheck-hledger in emacs, eg.
2024-03-26 10:05:57 -10:00
Dmitry Astapov
b0b9e69e4f ;dev:lib allow comment lines in the "if" table body 2024-03-08 07:42:58 -10:00
Jonathan Dowland
3b416a76ef ;cln:import: clarify haddock for getEffectiveAssignment
Signed-off-by: Jonathan Dowland <jon@dow.land>
2024-03-02 20:56:35 +00:00
Jonathan Dowland
c5079d4f1e dev:import: call hledgerFieldValue rather than re-implementing it
Signed-off-by: Jonathan Dowland <jon@dow.land>
2024-03-01 18:02:29 +00:00
Jonathan Dowland
1424a1f2f1 ;cln:import: update some Haddock strings to reflect #2158
Signed-off-by: Jonathan Dowland <jon@dow.land>
2024-03-01 17:58:28 +00:00
Jonathan Dowland
71684f5611 ref:import: simplify renderTemplate and friends
renderTemplate and its ancillary functions did not need the
HledgerFieldName argument, so remove it.

Signed-off-by: Jonathan Dowland <jon@dow.land>
2024-02-29 21:44:42 +00:00
Jonathan Dowland
b7027c8bbb feat:import: resolve matchgroup references in ConditionalBlock scope (#2158)
Adjust getEffectiveAssignment to compute an intermediary form of the
active assignments (with an additional Either wrapper to distinguish
top-level and conditional assignments) and move the remaining work to
its only caller, hledgerField.

Rework hledgerFieldValue. Instead of calling hledgerField, call
getEffectiveAssignment and--in the conditional block case--construct
a CsvRules scoped just to the active ConditionalBlock before calling
renderTemplate.

Adjust regexMatchValue to use rconditionalblocks to access conditional
blocks from the CsvRules, rather than rblocksassigning, since we haven't
narrowed the scope of that field.

The result is match group references are only expanded for match groups
that occur within the in-scope ConditionalBlock. Fixes: #2158.

Signed-off-by: Jonathan Dowland <jon@dow.land>
2024-02-29 21:44:40 +00:00
Jonathan Dowland
ac7f726282 ;ref:import: consistently use hledgerField
hledgerField is an alias to the function getEffectiveAssignment: both
names are used in various parts of RulesReader.

Treat hledgerField as the canonical name, and getEffectiveAssignment
as an implementation detail of hledgerField.

Replace all uses of getEffectiveAssignment with hledgerField (except the
one in hledgerField.)

Signed-off-by: Jonathan Dowland <jon@dow.land>
2024-02-29 21:44:39 +00:00
Jonathan Dowland
8f514ac16d ;test:import: test case for match groups (#2158)
Add a test which captures the issue of overlapping scope described
in GitHub issue #2158.

Signed-off-by: Jonathan Dowland <jon@dow.land>
2024-02-29 21:44:36 +00:00
Jonathan Dowland
bd5652c1c2 ;cln:import: remove superfluous comment lines
Signed-off-by: Jonathan Dowland <jon@dow.land>
2024-02-29 21:44:32 +00:00
Simon Michael
8f1ae401f4 dev: fix some partial head/tails, silence ghc 9.8's new warnings
Note the headErr/tailErr calls will print stack traces if they fail
(small ones: five lines, one of which is the useful location info),
which may or may not be best UX.
2024-02-28 15:58:21 -10:00
Simon Michael
fac3ee89af feat:allow other kinds of unicode space as digit group separators
Based on feedback in chat, I added support for several more kinds of
Unicode space character for separating digit groups, both when reading
and when displaying numbers. These are the spaces currently supported,
which are just my best guess at the ones that might show up in CSV files
now and then:

space,
no-break space,
en space,
em space,
punctuation space,
thin space,
narrow no-break space,
medium mathematical space
2024-02-28 09:12:17 -10:00
Simon Michael
5a36362b33 imp:journal: use a symlink's target's directory for relative include paths
When reading a symbolically-linked journal file,
relative paths in include directives are now evaluated
relative to the directory of the real linked file,
not the directory of the symlink.

This also seems to fix an obscure case where stats did not report
absolute included file paths in certain circumstances (stdin, maybe no
terminal..)
2024-02-22 08:48:31 -10:00
Simon Michael
60a1adc5ba lib: refactor, extract parseBalanceAssertionType 2024-02-20 20:55:27 -10:00
Michael Rees
d4ecdb3fea imp: Support tsv and ssv prefixes (#2164) 2024-02-08 06:44:44 -10:00
Simon Michael
a38af98c9e fix: a slowdown with many txns and many accounts since 1.29 [#2153]
When processing costs and equity postings in transactions during
journal finalisation, we now pass just the conversion account name(s)
rather than the entire map of account types. This slowdown was severe
for some users/data/machines.
2024-01-25 13:49:49 -10:00
Simon Michael
0cb382cf0e dev: rename AmountDisplayOpts -> AmountFormat, and related constants
noColour          -> defaultFmt
noCost            -> noCostFmt
oneLine           -> oneLineFmt
csvDisplay        -> machineFmt
2024-01-23 21:35:06 -10:00
Simon Michael
323f87b3e9 dev: Amount: aprice -> acost
Acked-by: Simon Michael <simon@joyful.com>
2024-01-23 21:35:06 -10:00
Simon Michael
8102bd9c2b dev: AmountPrice,UnitPrice,TotalPrice -> AmountCost,UnitCost,TotalCost; related renames 2024-01-23 21:35:06 -10:00
Simon Michael
41711d8ab5 fix:--anon: now hidden, gives an error, still usable as --obfuscate [#2133] 2024-01-21 11:59:11 -10:00
Simon Michael
8b45d4ba8c fix:csv: fix %FIELD interpolation in assignments using \n [#2134]
In field assignment values we now parse %FIELD references, \MATCHGROUP references
and "\n" newline markers more carefully, so all can coexist.
Parsing these values might be slower than before, but hopefully not noticeably so.
2023-12-23 19:25:34 -10:00
Simon Michael
20c299684b dev:csv: clarify renderTemplate [#2134] 2023-12-23 19:25:22 -10:00
Simon Michael
aaf4a38161 fix:timedot: accept numbers beginning with a decimal point again [#2130] 2023-12-08 15:08:35 -10:00
Simon Michael
6ae64c8f3e imp: allow declaring the empty payee name with "" (#2119) 2023-12-07 08:30:55 -10:00
Simon Michael
b6a46f637f feat:timedot: tagged time logging with letters 2023-11-25 21:28:23 -10:00
Simon Michael
96f9c1d2b2 ;dev: lotcostp: cleanup 2023-11-23 09:14:54 -10:00
Simon Michael
b4a6212453 imp:ledger-compat: accept lot costs with spaces after {, like Ledger 2023-11-23 08:31:57 -10:00
Simon Michael
2b18715885 fix:csv: fix tag: queries on CSV data (#2114) 2023-11-20 21:55:11 -10:00
Simon Michael
e92ab28cce imp:reading: better timing of strict checks and .latest writing (#2113)
Strict checks now run only once, at end of the high level read operation,
and not for each individual file; this fixes some spurious --strict failures,
like account declarations not affecting a sibling file as they should.

And .latest file writing now happens as the last step, after passing
strict checks. This is mainly for the import command, but it also
means that hledger print --new now does not update .latest files
if strict checks are failing.

The file reading API has been improved and documented in more detail.
2023-11-16 21:48:43 -10:00
Simon Michael
f8ffd9cdda imp: more precision handling fixes, debug output, test updates (precisiongeddon)
This and the preceding commits were "work in progress" that got out of control.
There's more to do, but this one brings these precision-related improvements
(at least):

When "infinite decimals" arise, they are now generally shown with
8 decimal digits rather than 255.

print and prices no longer add trailing decimal zeros unnecessarily.

Some code has been refactored or given more debug output.

All tests have been updated to match the recent changes.
2023-11-08 13:59:18 -08:00
Jonathan Dowland
8bfa382c68 feat: import: interpolate regex matches in field templates (#2009)
Replace occurrences of '\N' (where N is a positive number) in field
templates with the corresponding regular expression match group, if it
exists.

E.g. Warp the date to the first of the month for the second posting

    if %date (....-..)-..
        comment2 date:\1-01

E.g. Strip a prefix from an imported account name

    if %account1 liabilities:jon:(.*)
        account1 \1

Fixes #2009.

Signed-off-by: Jonathan Dowland <jon@dow.land>
2023-11-08 13:49:39 -08:00
Jonathan Dowland
c619e387ea ;fix: import: minor typo 2023-11-08 13:49:39 -08:00
Simon Michael
54b4c2ed22 fix: auto postings: detect redundant costs properly (fix #2110) 2023-11-07 04:38:39 -08:00
Peter Sagerson
efcea0600a feat: cli: Add tsv output (#869)
All commands that suport csv output now also support tsv output. The
data is identical, but the fields are separated by tab characters and
there is no quoting or escaping. Tab, carriage return, and newline
characters in data are converted to spaces (this should rarely if ever
happen in practice).
2023-11-06 16:46:04 -08:00
Simon Michael
63d41b30db ;doc: haddock, comment fixes 2023-10-21 17:58:43 +01:00
Simon Michael
5a72b9e9ea feat:print: add --round option for more control of precisions (#2085) 2023-10-18 13:56:24 +01:00
Simon Michael
c13c13ab1f lib!: use styleAmounts in more places; add rounding strategies 2023-10-18 13:56:24 +01:00
Simon Michael
94ab8b1ed7 dev: journalApplyCommodityStyles -> journalStyleAmounts 2023-10-18 13:56:24 +01:00
Simon Michael
f8027abb44 dev:print: refactor, add AmountStyle "rounding strategy"
Changes to enable more control of "rounding" behaviour
(ie, choosing display precisions for amounts).

This reverts 1.31's change of asprecision, making it a non-Maybe
again, and adds a new asrounding field providing more control over how
a target display precision is applied to existing amounts (two options
for now, more later). Functionality is in an interim state (reports do
no rounding).
2023-10-18 13:56:24 +01:00
Simon Michael
9ead8a64e1 fix:timedot: parse unitful quantities more accurately (fix #2096)
A quantity with a unit like "15m" was being parsed internally as
0.249999999... rather than 0.25 (and since hledger 1.21, printed that
way also). Now we round such quantities to two places during parsing,
to get exact quarter-hour amounts.
2023-10-17 12:05:10 +01:00
Simon Michael
e8da1f9f25 fix: balance-assigned amounts affect styles again (fix #2091) 2023-10-14 08:38:20 +01:00
bobobo1618
9fb5740045 Add support for negating a Matcher
https://github.com/simonmichael/hledger/issues/2054
2023-10-05 10:22:01 +01:00
Simon Michael
97be1646f1 dev: AmountStyle: make setting the precision optional
This simplifies the code for styling amounts with or without precision.
But it complicates the semantics (Nothing is useful only when setting style).
Not sure if it's the best way.
2023-09-02 06:46:14 +01:00
Simon Michael
85845e51b2 dev: AmountStyle: rename, reorder fields more mnemonically
Since this type is about to change anyway.
2023-09-02 06:46:14 +01:00