Strict checks now run only once, at end of the high level read operation,
and not for each individual file; this fixes some spurious --strict failures,
like account declarations not affecting a sibling file as they should.
And .latest file writing now happens as the last step, after passing
strict checks. This is mainly for the import command, but it also
means that hledger print --new now does not update .latest files
if strict checks are failing.
The file reading API has been improved and documented in more detail.
With valuation now preserving more decimal digits, roi could show
excessively precise decimals if there was no known display precision
for the valuation commodity. Now in that situation it limits the
precision to a maximum of 8 digits.
This and the preceding commits were "work in progress" that got out of control.
There's more to do, but this one brings these precision-related improvements
(at least):
When "infinite decimals" arise, they are now generally shown with
8 decimal digits rather than 255.
print and prices no longer add trailing decimal zeros unnecessarily.
Some code has been refactored or given more debug output.
All tests have been updated to match the recent changes.
- The prices comand now more accurately lists the prices that hledger
uses when calculating value reports (similar to what you'd see with
eg `hledger bal -V --debug=2`).
- The prices command's --infer-reverse-prices flag was confusing since
we always infer and use reverse prices; it has been renamed to --show-reverse.
- --infer-market-prices and --show-reverse combine properly.
- --show-reverse now ignores all zero prices rather than giving an error.
- Reverse prices (which can be infinite decimals) are now displayed
with at most 8 decimal digits (rather than the internal precision of
255 digits).
- Filtering prices by cur: or amt: now works properly.
- Price amounts are styled, but all decimal digits are shown.
Cost/value conversion now applies the standard display style, and
sets the display precision equal to the internal decimal precision
(or 8 if the decimal appears to be infinite).
This means value reports and especially `print -V` now show amounts
with more accurate and standard style and precision.
New tests have been added describing and explaining various
style/precision behaviours in print cost/value reports.
Replace occurrences of '\N' (where N is a positive number) in field
templates with the corresponding regular expression match group, if it
exists.
E.g. Warp the date to the first of the month for the second posting
if %date (....-..)-..
comment2 date:\1-01
E.g. Strip a prefix from an imported account name
if %account1 liabilities:jon:(.*)
account1 \1
Fixes#2009.
Signed-off-by: Jonathan Dowland <jon@dow.land>
All commands that suport csv output now also support tsv output. The
data is identical, but the fields are separated by tab characters and
there is no quoting or escaping. Tab, carriage return, and newline
characters in data are converted to spaces (this should rarely if ever
happen in practice).
Changes to enable more control of "rounding" behaviour
(ie, choosing display precisions for amounts).
This reverts 1.31's change of asprecision, making it a non-Maybe
again, and adds a new asrounding field providing more control over how
a target display precision is applied to existing amounts (two options
for now, more later). Functionality is in an interim state (reports do
no rounding).
A quantity with a unit like "15m" was being parsed internally as
0.249999999... rather than 0.25 (and since hledger 1.21, printed that
way also). Now we round such quantities to two places during parsing,
to get exact quarter-hour amounts.
Add a flag --summary-only for multi-column balance reports, which does
not display the main date columns for a report, but only displays the
summary columns (--row-total, --average). This is useful when there are
many columns (a weekly summary over many years) where you're only
interested in the average (or some other summary).
Eg "1,000" (with , as a thousands separator and no decimal digits) is
now displayed with a decimal mark: "1,000.".
"1 000" (where space is a thousands separator) is less ambiguous,
but we do the same thing (eg "1 000.") for consistency, and also to
help disambiguate when forgetting to quote a numeric commodity symbol
(eg "1234 0" where 1234 is a symbol that should have been in double quotes).
This simplifies the code for styling amounts with or without precision.
But it complicates the semantics (Nothing is useful only when setting style).
Not sure if it's the best way.
print now shows zero posting amounts with their original commodity
symbol and the corresponding style (instead of stripping the symbol).
If an inferred amount has multiple zeroes in different commodities,
a posting is displayed for each of these.
Possible breaking changes:
showMixedAmountLinesB, showAmountB, showAmountPrice now preserve
commodityful zeroes when rendering. This is intended to improve print output,
but it seems possible it might also affect balance and register reports,
though our tests show no change in those.
Equity conversion postings and cost amounts were being matched up too
exactly, causing valid entries with redundant conversion postings and
costs to be rejected. Now the amounts are compared with the precision
(number of decimal places) used in the conversion posting's amount.
Eg, here the first posting's 209.60495 GEL cost is recognised as a
match for the third posting's -209.60 GEL, using the latter's two
digit precision:
2023-01-01
Assets -84.01 USD @ 2.495 GEL ; 209.60495 GEL
Equity:Conversion 84.01 USD
Equity:Conversion -209.60 GEL
Assets 209.60 GEL
The suggested sample balance assertion now uses the same commodity
symbol as in the failing posting (the first, if there are more than
one). Also the cleared mark has been removed.
hledger check recentassertions now reports the error at the first
posting that's more than 7 days later than the latest balance
assertion (rather than at the balance assertion). This is the thing
actually triggering the error, and it is more likely to be visible or
at least closer when you are working at the end of a journal file.
Since 1.29, we unconditionally run part of the --infer-cost logic to
identify redundant costs/equity postings. This was too strict, raising
an error whenever it could not find postings matching the equity
postings. Now we do this (and also the explicit --infer-costs
operation) as a best effort, leaving transactions unchanged if we
can't detect matching postings. This is consistent with
--infer-equity, --infer-market-prices, -B and -V.
Transaction balancing is supposed to balance costs, but these were
being stripped when calculating balance assignments, causing us to
wrongly reject this transaction when the last amount is left implicit,
unlike Ledger:
2023-01-01
Assets AAA -1.1 @@ CCC 2
Assets BBB -1.2 @@ CCC 3
Expenses:Fees CCC 0.2
Assets = CCC 4.9
I'm not sure why costs were being stripped. I seem to have added it
in 2019 (to Journal.balanceNoAssignmentTransactionB in 3b47b58ae),
but this bug seems to be present even before that.
CSV rules files can now be read directly, eg you have the option of
writing `hledger -f foo.csv.rules CMD`. By default this will read data
from foo.csv in the same directory. But you can also specify a
different data file with a new `source FILE` rule. This has some
convenience features:
- If the data file does not exist, it is treated as empty, not an
error.
- If FILE is a relative path, it is relative to the rules file's
directory. If it is just a file name with no path, it is relative
to ~/Downloads/.
- If FILE is a glob pattern, the most recently modified matched file
is used.
This helps remove some of the busywork of managing CSV downloads.
Most of your financial institutions's default CSV filenames are
different and can be recognised by a glob pattern. So you can put a
rule like `source Checking1*.csv` in foo-checking.csv.rules,
periodically download CSV from Foo's website accepting your browser's
defaults, and then run `hledger import checking.csv.rules` to import
any new transactions. The next time, if you have done no cleanup, your
browser will probably save it as something like Checking1-2.csv, and
hledger will still see that because of the * wild card. You can choose
whether to delete CSVs after import, or keep them for a while as
temporary backups, or archive them somewhere.
Inner empty lines were not being skipped automatically, contrary to
docs. Now all empty lines are skipped automatically, and the `skip`
rule is needed only for non-empty lines, as intended.
This may be a breaking change: it's possible that the `skip` count
might need to be adjusted in some CSV rules files.
This can be useful to override defaults in scripts.
These flags will now toggle when repeated on the command line:
--invert
--transpose
-r/--related
-%/--percent
-E/--empty
-N/--no-total
-T/--row-total
-A/--average
-S/--sort-amount
Since hledger 1.25, "every Nth day of month" period rules with N > 28
could be off by a couple of days if given certain forecast start dates.
Eg `~ every 31st day of month` with `--forecast='2023-03-30..'`.
Breaking change: previously timeclock descriptions could contain
semicolons. Now a semicolon in the description will end it and
start a comment (which may contain tags).
I found at least one user for whom this would be a breaking change
(they generate forecast txns, and have auto posting rules, but don't
want the latter applied to the former). I guess it's better to keep
things as they were for now: if you need auto postings on your
forecast txns you must use two flags, --forecast --auto.
When doing a type: match we now also check the original unaliased,
unpivoted posting, as when doing an acct: match. This is effectively
how things worked with the older account type detection in hledger <1.27.
Previously, similarity completely outweighed recency, so a
slightly-more-similar transaction would always be selected no matter
how old it was. Now similarity and recency are more balanced,
and it should produce the desired transaction more often.
There is also new debug output (at debug level 1) for
troubleshooting.
Boolean queries are now prefixed with an 'expr:' prefix, making them
completely separable from old queries and making the addition of them a
little more migration proof.
The tests are updated accordingly, changes made to the tests previously
are removed and extra cautious documentation is also removed.
This commit changes some of the functions in the Query module and
changes the overall way to parse queries. Instead of using the words''
split function, this commit starts to fully parse the query, as it's
seen as a type of expression.
AND, OR, NOT, and space operators can be used. The space operator
simulates the behaviour from before, leaving a minimal amount of tests
that need to be adjusted to comply to the new behaviour.
Eg, where previously -p 'monthly from 1/15' or -M -b 1/15 would always
adjust the report start date to 1/1,
unless you used the special -p 'every 15th day of month from 1/15' form,
now the start date will not be adjusted. (It is still adjusted if
the report date is not specified explicitly, eg inferred from the journal).
This keeps behaviour consistent between report periods and periodic transactions.
'in' period expressions, like 'in 2023-01', are a grey area; they
do specify a start date (2023-01-01), although they look a bit implicit.
So previously, -p 'weekly in 2023-01' would adjust the start date to
the preceding monday (2022-12-26), but now it will start exactly on
2023-01-01 (a sunday, which also causes ugly verbose column headings).
To ensure monday based weeks and simple report headings here,
you would have to explicitly specific a start date that is a monday,
eg -p 'weekly from 2022-12-26 to 2023-02'.
You can now write both @/@@ costs and corresponding equity conversion postings
in a transaction at any time, not just when using --infer-costs or --infer-equity.
hledger will recognise the redundancy and ignore it.
One thing has become more strict: hledger now requires conversion postings
to occur in adjacent pairs; an odd number of them is not allowed.
(Conversion postings are postings to accounts of type `V`/`Conversion`,
or named `equity:conversion`, `equity:trade`, `equity:trading`,
or subaccounts of these.)
And, --infer-costs now works in transactions with an implicit amount
(inferring costs from equity now happens after transaction balancing,
not before).
Previously, the accounts passed to account directives would be stripped
of their surrounding brackets, but the required behaviour is to have
account directives plain reject bracketed accounts. This change ensures
that accounts in account directives may not start with a bracket
character.
Currently an account name like "a:(aa)" will not have (aa) unbracketed.
However, this seems reasonable since the full name is unbracketed and
thus will not be confused with virtual or virtual-balanced posting.
"" is equivalent to writing no symbol.
But it's now possible to declare market prices for the no-symbol
commodity, which occurs eg with timedot data.
P 2022-01-01 "" $100
Hledger.Utils.Debug's "trace or log" functions are now controlled as
follows: to enable logging, append ",logging" to the program name at
startup (using withProgName). This also works when running in GHCI.
And they log to PROGNAME.log, not debug.log.
All (hopefully) debug logging in the hledger packages is now "trace or
log" capable.
This means that hledger-ui should now log all debug output to
./hledger-ui.log, with none of it appearing on the console.
Moved from Hledger.Utils.Debug to Hledger.Utils.Print:
pshow
pshow'
pprint
pprint'
colorOption
useColorOnStdout
useColorOnStderr
outputFileOption
hasOutputFile
Previously, CSV date-times with a different time zone from yours
(with or without explicit timezones in the CSV) could give off-by-one
dates, because the CSV timezone was ignored.
Now,
1. you can use the `timezone` rule to indicate which other
timezone a CSV is implicitly using
2. CSV date-times with a timezone - whether declared by rule or
parsed with %Z - are localised to the system time zone
(or another set with the TZ environment variable).
dlog has been replaced by more reliable functions for debug-logging
to a file, useful for debugging TUI apps like hledger-ui:
dlogTrace
dlogTraceAt
dlogAt
dlog0
dlog1
dlog2
dlog3
dlog4
dlog5
dlog6
dlog7
dlog8
dlog9
Monochrome pprint' and pshow' have been added.
New dependency: deepseq
This replaces the old journal*AccountQuery with the new Type query. This
enables uniform treatment of account type, and fixes a subtle bug
(#1921).
Note that cbcsubreportquery no longer takes Journal as an argument.
hledger check recentassertions (or flycheck-hledger if you enable this
check) will complain if any balance-asserted account does not have a
balance assertion within 7 days before its latest posting. This aims
to prevent the situation where you are regularly updating your
journal, but forgetting to check your balances against the real world,
eventually requiring you to dig back through months of data to find
the error.
transaction prices.
When given --infer-costs, hledger will now separately infer transaction
prices for different prices. Given a pair of adjacent conversion
postings, hledger will check if there is a single posting with a
transaction price which matches both the amounts. If so, it associates
those conversion postings to that priced post.
If it can't find any transaction price postings which match, it will
find the first non-transaction price post which matches one of the two
amounts, and will add a transaction price to that, and associate them.
When given --infer-equity, hledger will change transaction prices to balancing equity postings. This introduces the inverse operation, --infer-costs, which will match balancing equity postings and transform them into a transaction price, allowing --cost to work properly with them. This is only a partial inverse as it needs to use some heuristics to match the postings which will not work in complicated cases.
Specifically, when hledger finds exactly two conversion postings in a transaction (by default, subaccounts of equity:conversion or equity:trad(e|ing)), it will find the first posting in the transaction whose amount is negative one of the conversion posting amounts, and inserts the corresponding transaction price.
May also fix#1154, #1033, #708, #536, #73: testing is needed.
This aims to solve all problems where misconfigured locales lead to
parsers failing on utf8-encoded data. This should hopefully avoid
encoding issues, but since it fundamentally alters how encoding is dealt
with it may lead to unexpected outcomes. Widespread testing on a number
of different platforms would be useful.
And remove the last vestiges of older more complex behaviour.
ordereddates now always checks all transactions in each file,
unaffected by a query. (But still affected by --date2).
By default, all account declarations and all transactions are searched;
but when there's a query involving transaction fields,
account declarations unrelated to the matched transactions are not searched.
added:
queryIsCode
queryIsTransactionRelated
parseAndFinaliseJournal' has been removed. In the unlikely event you
needed it in your code, you can replace it with:
parseAndFinaliseJournal' parser iopts fp t =>
initialiseAndParseJournal parser iopts fp t
>>= liftEither . journalApplyAliases (aliasesFromOpts iopts)
>>= journalFinalise iopts fp t
Some parsers have been generalised from JournalParser to TextParser.
This increases composability and avoids some ugly case handling. We
re-export runExceptT in Hledger.Read.
The final return types of the following functions has been changed from
IO (Either String a) to ExceptT String IO a. If this causes a problem,
you can get the old behaviour by calling runExceptT on the output:
readJournal, readJournalFiles, readJournalFile
Or, you can use the easy functions readJournal', readJournalFiles', and
readJournalFile', which assume default options and return in the IO
monad.
total prices. (#1813)
Also reduce duplication for inferring market prices (previously it was
done separately in both Hledger.Data.Journal and
Hledger.Cli.Commands.Prices), and remove *TotalPriceToUnitPrice
functions, since unit prices cannot represent all total prices.
Add a helper function numDigitsInt to get the number of digits in an
integer, which has a surprising number of ways to get it wrong.
This results in big speedups in cases when we have many transaction prices, like in examples/10000x1000x10.journal. This can be disabled with the show_costs_ option in ReportOpts.
^assets?(:.+)?:(cash|bank|che(ck|que?)(ing)?|savings?|current)(:|$)
This should have no effect outside the cash report.
The rule for auto-detecting "Cash" (liquid assets) from account names
has changed in this release - see [Account types]. This is to simplify
how account types are detected and configured. If you use the cashflow
report without explicitly declaring accounts as Cash, you might notice a
change, and might need to add explicit account type: tags.
