print now shows zero posting amounts with their original commodity
symbol and the corresponding style (instead of stripping the symbol).
If an inferred amount has multiple zeroes in different commodities,
a posting is displayed for each of these.
Possible breaking changes:
showMixedAmountLinesB, showAmountB, showAmountPrice now preserve
commodityful zeroes when rendering. This is intended to improve print output,
but it seems possible it might also affect balance and register reports,
though our tests show no change in those.
Equity conversion postings and cost amounts were being matched up too
exactly, causing valid entries with redundant conversion postings and
costs to be rejected. Now the amounts are compared with the precision
(number of decimal places) used in the conversion posting's amount.
Eg, here the first posting's 209.60495 GEL cost is recognised as a
match for the third posting's -209.60 GEL, using the latter's two
digit precision:
2023-01-01
Assets -84.01 USD @ 2.495 GEL ; 209.60495 GEL
Equity:Conversion 84.01 USD
Equity:Conversion -209.60 GEL
Assets 209.60 GEL
The suggested sample balance assertion now uses the same commodity
symbol as in the failing posting (the first, if there are more than
one). Also the cleared mark has been removed.
hledger check recentassertions now reports the error at the first
posting that's more than 7 days later than the latest balance
assertion (rather than at the balance assertion). This is the thing
actually triggering the error, and it is more likely to be visible or
at least closer when you are working at the end of a journal file.
Since 1.29, we unconditionally run part of the --infer-cost logic to
identify redundant costs/equity postings. This was too strict, raising
an error whenever it could not find postings matching the equity
postings. Now we do this (and also the explicit --infer-costs
operation) as a best effort, leaving transactions unchanged if we
can't detect matching postings. This is consistent with
--infer-equity, --infer-market-prices, -B and -V.
Transaction balancing is supposed to balance costs, but these were
being stripped when calculating balance assignments, causing us to
wrongly reject this transaction when the last amount is left implicit,
unlike Ledger:
2023-01-01
Assets AAA -1.1 @@ CCC 2
Assets BBB -1.2 @@ CCC 3
Expenses:Fees CCC 0.2
Assets = CCC 4.9
I'm not sure why costs were being stripped. I seem to have added it
in 2019 (to Journal.balanceNoAssignmentTransactionB in 3b47b58ae),
but this bug seems to be present even before that.
CSV rules files can now be read directly, eg you have the option of
writing `hledger -f foo.csv.rules CMD`. By default this will read data
from foo.csv in the same directory. But you can also specify a
different data file with a new `source FILE` rule. This has some
convenience features:
- If the data file does not exist, it is treated as empty, not an
error.
- If FILE is a relative path, it is relative to the rules file's
directory. If it is just a file name with no path, it is relative
to ~/Downloads/.
- If FILE is a glob pattern, the most recently modified matched file
is used.
This helps remove some of the busywork of managing CSV downloads.
Most of your financial institutions's default CSV filenames are
different and can be recognised by a glob pattern. So you can put a
rule like `source Checking1*.csv` in foo-checking.csv.rules,
periodically download CSV from Foo's website accepting your browser's
defaults, and then run `hledger import checking.csv.rules` to import
any new transactions. The next time, if you have done no cleanup, your
browser will probably save it as something like Checking1-2.csv, and
hledger will still see that because of the * wild card. You can choose
whether to delete CSVs after import, or keep them for a while as
temporary backups, or archive them somewhere.
Inner empty lines were not being skipped automatically, contrary to
docs. Now all empty lines are skipped automatically, and the `skip`
rule is needed only for non-empty lines, as intended.
This may be a breaking change: it's possible that the `skip` count
might need to be adjusted in some CSV rules files.
This can be useful to override defaults in scripts.
These flags will now toggle when repeated on the command line:
--invert
--transpose
-r/--related
-%/--percent
-E/--empty
-N/--no-total
-T/--row-total
-A/--average
-S/--sort-amount
Since hledger 1.25, "every Nth day of month" period rules with N > 28
could be off by a couple of days if given certain forecast start dates.
Eg `~ every 31st day of month` with `--forecast='2023-03-30..'`.
Breaking change: previously timeclock descriptions could contain
semicolons. Now a semicolon in the description will end it and
start a comment (which may contain tags).
I found at least one user for whom this would be a breaking change
(they generate forecast txns, and have auto posting rules, but don't
want the latter applied to the former). I guess it's better to keep
things as they were for now: if you need auto postings on your
forecast txns you must use two flags, --forecast --auto.
When doing a type: match we now also check the original unaliased,
unpivoted posting, as when doing an acct: match. This is effectively
how things worked with the older account type detection in hledger <1.27.
Previously, similarity completely outweighed recency, so a
slightly-more-similar transaction would always be selected no matter
how old it was. Now similarity and recency are more balanced,
and it should produce the desired transaction more often.
There is also new debug output (at debug level 1) for
troubleshooting.
Boolean queries are now prefixed with an 'expr:' prefix, making them
completely separable from old queries and making the addition of them a
little more migration proof.
The tests are updated accordingly, changes made to the tests previously
are removed and extra cautious documentation is also removed.
This commit changes some of the functions in the Query module and
changes the overall way to parse queries. Instead of using the words''
split function, this commit starts to fully parse the query, as it's
seen as a type of expression.
AND, OR, NOT, and space operators can be used. The space operator
simulates the behaviour from before, leaving a minimal amount of tests
that need to be adjusted to comply to the new behaviour.
Eg, where previously -p 'monthly from 1/15' or -M -b 1/15 would always
adjust the report start date to 1/1,
unless you used the special -p 'every 15th day of month from 1/15' form,
now the start date will not be adjusted. (It is still adjusted if
the report date is not specified explicitly, eg inferred from the journal).
This keeps behaviour consistent between report periods and periodic transactions.
'in' period expressions, like 'in 2023-01', are a grey area; they
do specify a start date (2023-01-01), although they look a bit implicit.
So previously, -p 'weekly in 2023-01' would adjust the start date to
the preceding monday (2022-12-26), but now it will start exactly on
2023-01-01 (a sunday, which also causes ugly verbose column headings).
To ensure monday based weeks and simple report headings here,
you would have to explicitly specific a start date that is a monday,
eg -p 'weekly from 2022-12-26 to 2023-02'.
You can now write both @/@@ costs and corresponding equity conversion postings
in a transaction at any time, not just when using --infer-costs or --infer-equity.
hledger will recognise the redundancy and ignore it.
One thing has become more strict: hledger now requires conversion postings
to occur in adjacent pairs; an odd number of them is not allowed.
(Conversion postings are postings to accounts of type `V`/`Conversion`,
or named `equity:conversion`, `equity:trade`, `equity:trading`,
or subaccounts of these.)
And, --infer-costs now works in transactions with an implicit amount
(inferring costs from equity now happens after transaction balancing,
not before).
Previously, the accounts passed to account directives would be stripped
of their surrounding brackets, but the required behaviour is to have
account directives plain reject bracketed accounts. This change ensures
that accounts in account directives may not start with a bracket
character.
Currently an account name like "a:(aa)" will not have (aa) unbracketed.
However, this seems reasonable since the full name is unbracketed and
thus will not be confused with virtual or virtual-balanced posting.
"" is equivalent to writing no symbol.
But it's now possible to declare market prices for the no-symbol
commodity, which occurs eg with timedot data.
P 2022-01-01 "" $100
Hledger.Utils.Debug's "trace or log" functions are now controlled as
follows: to enable logging, append ",logging" to the program name at
startup (using withProgName). This also works when running in GHCI.
And they log to PROGNAME.log, not debug.log.
All (hopefully) debug logging in the hledger packages is now "trace or
log" capable.
This means that hledger-ui should now log all debug output to
./hledger-ui.log, with none of it appearing on the console.
Moved from Hledger.Utils.Debug to Hledger.Utils.Print:
pshow
pshow'
pprint
pprint'
colorOption
useColorOnStdout
useColorOnStderr
outputFileOption
hasOutputFile
Previously, CSV date-times with a different time zone from yours
(with or without explicit timezones in the CSV) could give off-by-one
dates, because the CSV timezone was ignored.
Now,
1. you can use the `timezone` rule to indicate which other
timezone a CSV is implicitly using
2. CSV date-times with a timezone - whether declared by rule or
parsed with %Z - are localised to the system time zone
(or another set with the TZ environment variable).
dlog has been replaced by more reliable functions for debug-logging
to a file, useful for debugging TUI apps like hledger-ui:
dlogTrace
dlogTraceAt
dlogAt
dlog0
dlog1
dlog2
dlog3
dlog4
dlog5
dlog6
dlog7
dlog8
dlog9
Monochrome pprint' and pshow' have been added.
New dependency: deepseq
This replaces the old journal*AccountQuery with the new Type query. This
enables uniform treatment of account type, and fixes a subtle bug
(#1921).
Note that cbcsubreportquery no longer takes Journal as an argument.
hledger check recentassertions (or flycheck-hledger if you enable this
check) will complain if any balance-asserted account does not have a
balance assertion within 7 days before its latest posting. This aims
to prevent the situation where you are regularly updating your
journal, but forgetting to check your balances against the real world,
eventually requiring you to dig back through months of data to find
the error.
transaction prices.
When given --infer-costs, hledger will now separately infer transaction
prices for different prices. Given a pair of adjacent conversion
postings, hledger will check if there is a single posting with a
transaction price which matches both the amounts. If so, it associates
those conversion postings to that priced post.
If it can't find any transaction price postings which match, it will
find the first non-transaction price post which matches one of the two
amounts, and will add a transaction price to that, and associate them.
When given --infer-equity, hledger will change transaction prices to balancing equity postings. This introduces the inverse operation, --infer-costs, which will match balancing equity postings and transform them into a transaction price, allowing --cost to work properly with them. This is only a partial inverse as it needs to use some heuristics to match the postings which will not work in complicated cases.
Specifically, when hledger finds exactly two conversion postings in a transaction (by default, subaccounts of equity:conversion or equity:trad(e|ing)), it will find the first posting in the transaction whose amount is negative one of the conversion posting amounts, and inserts the corresponding transaction price.
May also fix#1154, #1033, #708, #536, #73: testing is needed.
This aims to solve all problems where misconfigured locales lead to
parsers failing on utf8-encoded data. This should hopefully avoid
encoding issues, but since it fundamentally alters how encoding is dealt
with it may lead to unexpected outcomes. Widespread testing on a number
of different platforms would be useful.
And remove the last vestiges of older more complex behaviour.
ordereddates now always checks all transactions in each file,
unaffected by a query. (But still affected by --date2).
By default, all account declarations and all transactions are searched;
but when there's a query involving transaction fields,
account declarations unrelated to the matched transactions are not searched.
added:
queryIsCode
queryIsTransactionRelated
parseAndFinaliseJournal' has been removed. In the unlikely event you
needed it in your code, you can replace it with:
parseAndFinaliseJournal' parser iopts fp t =>
initialiseAndParseJournal parser iopts fp t
>>= liftEither . journalApplyAliases (aliasesFromOpts iopts)
>>= journalFinalise iopts fp t
Some parsers have been generalised from JournalParser to TextParser.
This increases composability and avoids some ugly case handling. We
re-export runExceptT in Hledger.Read.
The final return types of the following functions has been changed from
IO (Either String a) to ExceptT String IO a. If this causes a problem,
you can get the old behaviour by calling runExceptT on the output:
readJournal, readJournalFiles, readJournalFile
Or, you can use the easy functions readJournal', readJournalFiles', and
readJournalFile', which assume default options and return in the IO
monad.
total prices. (#1813)
Also reduce duplication for inferring market prices (previously it was
done separately in both Hledger.Data.Journal and
Hledger.Cli.Commands.Prices), and remove *TotalPriceToUnitPrice
functions, since unit prices cannot represent all total prices.
Add a helper function numDigitsInt to get the number of digits in an
integer, which has a surprising number of ways to get it wrong.
This results in big speedups in cases when we have many transaction prices, like in examples/10000x1000x10.journal. This can be disabled with the show_costs_ option in ReportOpts.
^assets?(:.+)?:(cash|bank|che(ck|que?)(ing)?|savings?|current)(:|$)
This should have no effect outside the cash report.
The rule for auto-detecting "Cash" (liquid assets) from account names
has changed in this release - see [Account types]. This is to simplify
how account types are detected and configured. If you use the cashflow
report without explicitly declaring accounts as Cash, you might notice a
change, and might need to add explicit account type: tags.
This requires checking parent accounts for any new accounts introduced by auto postings which do not exist in the original journal.
Also refactor journalFinalise to only call journalPostingsAddAccountTags once, and use fewer intermediate variables.
type:TYPES, where TYPES is any of the (case insensitive) letters
ALERXCV, matches accounts by their declared or inferred type.
(See https://hledger.org/hledger.html#account-types.)
This should work with most commands, eg:
hledger bal type:al
hledger reg type:x
API changes:
Journal has a new jaccounttypes map.
The journalAccountType lookup function makes it easy to check an account's type.
The journalTags and journalInheritedTags functions look up an account's tags.
Functions like journalFilterPostings and journalFilterTransactions,
and new matching functions matchesAccountExtra, matchesPostingExtra
and matchesTransactionExtra, use these to allow more powerful matching
that is aware of account types and tags.
Accounts, postings, and transactions can now all be filtered by the
tags in an account's declaration. In particular it's now possible to
more reliably select accounts by type, using their type: tag rather
than their name:
account myasset ; type:Asset
account myliability ; type:Liability
$ hledger accounts tag:type=^a
myasset
Accounts inherit tags from their parents.
API changes:
A finalised Journal has a new jdeclaredaccounttags field
for easy lookup of account tags.
Query.matchesTaggedAccount is a tag-aware version of matchesAccount.
directives.
Previously, you declare an account type with the following format:
account assets A
This has been deprecated since 1.13, and should now be declared with one
of:
account assets ; type:A
account assets ; type:asset
Previously the helper functions splitspan and splitspan' would calculate
each span from the start point of the previous span. This meant we had
to be very careful not to lose any relevant information (e.g. what day
of the week it was) about the original start date. We now calculate each
span from the original start date, so there's no risk of losing
information. This simplifies many of the calculations.
parsers to allow for arbitrary numbers of periods in relative dates.
We now accept smart dates like “in 5 days, 5 weeks ahead, in -6 months, 2 quarters ago”.
Introduce --infer-equity option which will generate conversion postings.
--cost will override --infer-equity.
This means there will no longer be unbalanced transactions, but will be
offsetting conversion postings to balance things out. For example.
2000-01-01
a 1 AAA @@ 2 BBB
b -2 BBB
When converting to cost, this is treated the same as before.
When used with --infer-equity, this is now treated as:
2000-01-01
a 1 AAA
equity:conversion:AAA-BBB:AAA -1 AAA
equity:conversion:AAA-BBB:BBB 2 BBB
b -2 BBB
There is a new account type, Conversion/V, which is a subtype of Equity/E.
The first account declared with this type, if any, is used as the base account
for inferred equity postings in conversion transactions, overriding the default
"equity:conversion".
API changes:
Costing has been changed to ConversionOp with three options:
NoConversionOp, ToCost, and InferEquity.
The first correspond to the previous NoCost and Cost options, while the
third corresponds to the --infer-equity flag. This converts transactions with costs
(one or more transaction prices) to transactions with equity:conversion postings.
It is in ConversionOp because converting to cost with -B/--cost and inferring conversion
equity postings with --infer-equity are mutually exclusive.
Correspondingly, the cost_ record of ReportOpts has been changed to
conversionop_.
This also removes show_costs_ option in ReportOpts, as its functionality
has been replaced by the richer cost_ option.
Together with -E, this shows a balance for both used and declared
accounts (excluding empty parent accounts, which are usually not
wanted in list-mode reports).
This is somewhat consistent with --declared in the accounts and payees
commands, except for the leaf account restriction.
The idea of this is to be able to see a useful "complete" balance
report, even when you don't have transactions in all of your declared
accounts yet. I mainly want this for hledger-ui, but there's no harm
in exposing it in the balance CLI as well.
Together with -E, this allows showing a balance for all accounts, both
used and declared. I mainly want this for hledger-ui, but there's no
harm in exposing it in the balance command as well. This is somewhat
consistent with the accounts and payees commands.
This allows more control over how multicommodity amounts are displayed.
In addition to the default single-line display, and the recent commodity
column display, we now have multi-line display. This is controlled by
the --layout option, which has possible values "wide", "tall", and
"bare". The --commodity-column option has been hidden, but is equivalent
to --layout=bare.
squash
Replace showPosting with a wrapper around postingAsLines.
The functions textConcat(Top|Bottom)Padded are no longer used anywhere
in the code base, and can be removed if desired.
This produces slightly different output for showPosting, in particular
it no longer displays the transaction date. However, this has been
marked as ‘for debugging only’ for a while, and is only used in
hledger-check-fancy assertions. The output there is still acceptable.
Combining valuation with filtration is subtle and error-prone (see e.g. #1625).
We have to do in in both MultiBalanceReport and PostingsReport, where it
is done in slightly different ways. This refactors this functionality
into separate functions which are called in both reports, for uniform
behaviour.
