Since 1.29, we unconditionally run part of the --infer-cost logic to
identify redundant costs/equity postings. This was too strict, raising
an error whenever it could not find postings matching the equity
postings. Now we do this (and also the explicit --infer-costs
operation) as a best effort, leaving transactions unchanged if we
can't detect matching postings. This is consistent with
--infer-equity, --infer-market-prices, -B and -V.
Transaction balancing is supposed to balance costs, but these were
being stripped when calculating balance assignments, causing us to
wrongly reject this transaction when the last amount is left implicit,
unlike Ledger:
2023-01-01
Assets AAA -1.1 @@ CCC 2
Assets BBB -1.2 @@ CCC 3
Expenses:Fees CCC 0.2
Assets = CCC 4.9
I'm not sure why costs were being stripped. I seem to have added it
in 2019 (to Journal.balanceNoAssignmentTransactionB in 3b47b58ae),
but this bug seems to be present even before that.
Since hledger 1.25, "every Nth day of month" period rules with N > 28
could be off by a couple of days if given certain forecast start dates.
Eg `~ every 31st day of month` with `--forecast='2023-03-30..'`.
Breaking change: previously timeclock descriptions could contain
semicolons. Now a semicolon in the description will end it and
start a comment (which may contain tags).
I found at least one user for whom this would be a breaking change
(they generate forecast txns, and have auto posting rules, but don't
want the latter applied to the former). I guess it's better to keep
things as they were for now: if you need auto postings on your
forecast txns you must use two flags, --forecast --auto.
Previously, similarity completely outweighed recency, so a
slightly-more-similar transaction would always be selected no matter
how old it was. Now similarity and recency are more balanced,
and it should produce the desired transaction more often.
There is also new debug output (at debug level 1) for
troubleshooting.
Currently an account name like "a:(aa)" will not have (aa) unbracketed.
However, this seems reasonable since the full name is unbracketed and
thus will not be confused with virtual or virtual-balanced posting.
Hledger.Utils.Debug's "trace or log" functions are now controlled as
follows: to enable logging, append ",logging" to the program name at
startup (using withProgName). This also works when running in GHCI.
And they log to PROGNAME.log, not debug.log.
All (hopefully) debug logging in the hledger packages is now "trace or
log" capable.
This means that hledger-ui should now log all debug output to
./hledger-ui.log, with none of it appearing on the console.
This replaces the old journal*AccountQuery with the new Type query. This
enables uniform treatment of account type, and fixes a subtle bug
(#1921).
Note that cbcsubreportquery no longer takes Journal as an argument.
hledger check recentassertions (or flycheck-hledger if you enable this
check) will complain if any balance-asserted account does not have a
balance assertion within 7 days before its latest posting. This aims
to prevent the situation where you are regularly updating your
journal, but forgetting to check your balances against the real world,
eventually requiring you to dig back through months of data to find
the error.
transaction prices.
When given --infer-costs, hledger will now separately infer transaction
prices for different prices. Given a pair of adjacent conversion
postings, hledger will check if there is a single posting with a
transaction price which matches both the amounts. If so, it associates
those conversion postings to that priced post.
If it can't find any transaction price postings which match, it will
find the first non-transaction price post which matches one of the two
amounts, and will add a transaction price to that, and associate them.
When given --infer-equity, hledger will change transaction prices to balancing equity postings. This introduces the inverse operation, --infer-costs, which will match balancing equity postings and transform them into a transaction price, allowing --cost to work properly with them. This is only a partial inverse as it needs to use some heuristics to match the postings which will not work in complicated cases.
Specifically, when hledger finds exactly two conversion postings in a transaction (by default, subaccounts of equity:conversion or equity:trad(e|ing)), it will find the first posting in the transaction whose amount is negative one of the conversion posting amounts, and inserts the corresponding transaction price.
May also fix#1154, #1033, #708, #536, #73: testing is needed.
This aims to solve all problems where misconfigured locales lead to
parsers failing on utf8-encoded data. This should hopefully avoid
encoding issues, but since it fundamentally alters how encoding is dealt
with it may lead to unexpected outcomes. Widespread testing on a number
of different platforms would be useful.
total prices. (#1813)
Also reduce duplication for inferring market prices (previously it was
done separately in both Hledger.Data.Journal and
Hledger.Cli.Commands.Prices), and remove *TotalPriceToUnitPrice
functions, since unit prices cannot represent all total prices.
Add a helper function numDigitsInt to get the number of digits in an
integer, which has a surprising number of ways to get it wrong.
^assets?(:.+)?:(cash|bank|che(ck|que?)(ing)?|savings?|current)(:|$)
This should have no effect outside the cash report.
The rule for auto-detecting "Cash" (liquid assets) from account names
has changed in this release - see [Account types]. This is to simplify
how account types are detected and configured. If you use the cashflow
report without explicitly declaring accounts as Cash, you might notice a
change, and might need to add explicit account type: tags.
This requires checking parent accounts for any new accounts introduced by auto postings which do not exist in the original journal.
Also refactor journalFinalise to only call journalPostingsAddAccountTags once, and use fewer intermediate variables.
type:TYPES, where TYPES is any of the (case insensitive) letters
ALERXCV, matches accounts by their declared or inferred type.
(See https://hledger.org/hledger.html#account-types.)
This should work with most commands, eg:
hledger bal type:al
hledger reg type:x
API changes:
Journal has a new jaccounttypes map.
The journalAccountType lookup function makes it easy to check an account's type.
The journalTags and journalInheritedTags functions look up an account's tags.
Functions like journalFilterPostings and journalFilterTransactions,
and new matching functions matchesAccountExtra, matchesPostingExtra
and matchesTransactionExtra, use these to allow more powerful matching
that is aware of account types and tags.
Accounts, postings, and transactions can now all be filtered by the
tags in an account's declaration. In particular it's now possible to
more reliably select accounts by type, using their type: tag rather
than their name:
account myasset ; type:Asset
account myliability ; type:Liability
$ hledger accounts tag:type=^a
myasset
Accounts inherit tags from their parents.
API changes:
A finalised Journal has a new jdeclaredaccounttags field
for easy lookup of account tags.
Query.matchesTaggedAccount is a tag-aware version of matchesAccount.
Previously the helper functions splitspan and splitspan' would calculate
each span from the start point of the previous span. This meant we had
to be very careful not to lose any relevant information (e.g. what day
of the week it was) about the original start date. We now calculate each
span from the original start date, so there's no risk of losing
information. This simplifies many of the calculations.
parsers to allow for arbitrary numbers of periods in relative dates.
We now accept smart dates like “in 5 days, 5 weeks ahead, in -6 months, 2 quarters ago”.
Introduce --infer-equity option which will generate conversion postings.
--cost will override --infer-equity.
This means there will no longer be unbalanced transactions, but will be
offsetting conversion postings to balance things out. For example.
2000-01-01
a 1 AAA @@ 2 BBB
b -2 BBB
When converting to cost, this is treated the same as before.
When used with --infer-equity, this is now treated as:
2000-01-01
a 1 AAA
equity:conversion:AAA-BBB:AAA -1 AAA
equity:conversion:AAA-BBB:BBB 2 BBB
b -2 BBB
There is a new account type, Conversion/V, which is a subtype of Equity/E.
The first account declared with this type, if any, is used as the base account
for inferred equity postings in conversion transactions, overriding the default
"equity:conversion".
API changes:
Costing has been changed to ConversionOp with three options:
NoConversionOp, ToCost, and InferEquity.
The first correspond to the previous NoCost and Cost options, while the
third corresponds to the --infer-equity flag. This converts transactions with costs
(one or more transaction prices) to transactions with equity:conversion postings.
It is in ConversionOp because converting to cost with -B/--cost and inferring conversion
equity postings with --infer-equity are mutually exclusive.
Correspondingly, the cost_ record of ReportOpts has been changed to
conversionop_.
This also removes show_costs_ option in ReportOpts, as its functionality
has been replaced by the richer cost_ option.
Together with -E, this shows a balance for both used and declared
accounts (excluding empty parent accounts, which are usually not
wanted in list-mode reports).
This is somewhat consistent with --declared in the accounts and payees
commands, except for the leaf account restriction.
The idea of this is to be able to see a useful "complete" balance
report, even when you don't have transactions in all of your declared
accounts yet. I mainly want this for hledger-ui, but there's no harm
in exposing it in the balance CLI as well.
This allows more control over how multicommodity amounts are displayed.
In addition to the default single-line display, and the recent commodity
column display, we now have multi-line display. This is controlled by
the --layout option, which has possible values "wide", "tall", and
"bare". The --commodity-column option has been hidden, but is equivalent
to --layout=bare.
squash
Replace showPosting with a wrapper around postingAsLines.
The functions textConcat(Top|Bottom)Padded are no longer used anywhere
in the code base, and can be removed if desired.
This produces slightly different output for showPosting, in particular
it no longer displays the transaction date. However, this has been
marked as ‘for debugging only’ for a while, and is only used in
hledger-check-fancy assertions. The output there is still acceptable.
Combining valuation with filtration is subtle and error-prone (see e.g. #1625).
We have to do in in both MultiBalanceReport and PostingsReport, where it
is done in slightly different ways. This refactors this functionality
into separate functions which are called in both reports, for uniform
behaviour.
(SourcePos, SourcePos).
This has been marked for possible removal for a while. We are keeping
strictly more information. Possible edge cases arise with Timeclock and
CsvReader, but I think these are covered.
The particular motivation for getting rid of this is that
GenericSourcePos is creating some awkward import considerations for
little gain. Removing this enables some flattening of the module
dependency tree.
Hledger.Data.Balancing.
Both Hledger.Data.Transaction and Hledger.Data.Journal are massive
module with many things in them. Placing the balancing functions, which
are conceptually related, into a separate module helps keep things more
modular.
It also reduces the risk of import cycles, as right now balancing
functions cannot depend on any functions defined outside of
Hledger.Data.Transaction or Hledger.Data.Journal, respectively, if those
modules require basic transaction or journal functions.
mixedAmount(Looks|Is)Zero now operate directly on the MixedAmount,
rather than converting them to a list of amounts first.
mixedAmountCost no longer reconstructs the entire MixedAmount when there
are amounts with no cost.
transactionCheckBalanced only checks if signs are okay if sums are not
okay. It also only traverses the list of postings once when picking real
and balanced virtual postings.
There are no modules which depend on Hledger.Data.Commodity which don't
also depend on Hledger.Data.Amount. Though Hledger.Data.Amount is a very
large module and might be broken up, Hledger.Data.Commodity only defines
three very small functions which are used, and so can be combined with
little cost.
POSIXTime.
This eliminates old-time, which has been deprecated for a while, from
our dependencies.
This introduces a slight incompatibility, as a small number of functions
now take/return POSIXTime instead of ClockTime. Generally you will be
using the current time, in which case you should use getPOSIXTime from
Data.Time.Clock.POSIX instead of getClockTime.
utcTimeToClockTime has been removed, as it is now equivalent to
utcTimeToPOSIXSeconds from Data.Time.Clock.POSIX.
We can't filter out empty commodity strings since that is a legitimate
group. Simultaneously, we should only include the empty commodity if it
is explicitly used (part of a posting) and not generated as part of
`Amounts.amounts`
A gain report will report on unrealised gains by looking at the
difference between the valuation of an amount (by default, --value=end),
and the valuation of the cost of the amount.
This adds the `--commodity-column` option that displays each commodity
on a separate line and the commodities themselves as a separate column.
The initial design considerations are at
simonmichael.hledger.issues.1559
The single-period balance report with `--commodity-column` does not
interoperate with custom formats.
style amounts according to that argument. journalAddForecast and
journalTransform now return an Either String Journal.
This improves efficiency, as we no longer have to restyle all amounts in
the journal after generating auto postings or periodic transactions.
Changing the return type of journalAddForecast and journalTransform
reduces partiality.
To get the previous behaviour for modifyTransaction, use modifyTransaction mempty.
In Amount, aismultiplier is a boolean flag that will always be False,
except for in TMPostingRules, where it indicates whether the posting
rule is a multiplier. It is therefore unnecessary in the vast majority
of cases. This posting pulls this flag out of Amount and puts it into
TMPostingRule, so it is only kept around when necessary.
This changes the parsing of journals somewhat. Previously you could
include an * before an amount anywhere in a Journal, and it would
happily parse and set the aismultiplier flag true. This will now fail
with a parse error: * is now only acceptable before an amount within an
auto posting rule.
Any usage of the library in which the aismultiplier field is read or set
should be removed. If you truly need its functionality, you should
switch to using TMPostingRule.
This changes the JSON output of Amount, as it will no longer include
aismultiplier.
This change provides more predictable and intuitive behaviour when
using -S/--sort-amount with multiple commodities.
It implements a custom Ord (and Eq) instance for MixedAmount
which substitutes zero for any missing commodities.
As a consequence, all the ways of representing zero with a MixedAmount ([],
[A 0], [A 0, B 0, ...]) are now Eq-ual (==), whereas before they were
not. We have not been able to find anything broken by this change.
* imp: lib: Compare MixedAmounts by substituting zero for any missing commodities. (#1563)
* ;doc: Update docs for new multicommodity sort by amount rules.
Also corrects a regression introduced in
8ab29f84b32288a34ae7627a2204081fae31900f where transaction modifier
postings without multipliers would incorrectly be filtered by commodity.
transactions are balanced possibly using explicit prices, but without
inferring any prices. This is included in --strict mode.
Renames check autobalanced to check balancedwithautoconversion.
instead of a list of Amounts. No longer export Mixed constructor, to
keep API clean (if you really need it, you can import it directly from
Hledger.Data.Types). We also ensure the JSON representation of
MixedAmount doesn't change: it is stored as a normalised list of
Amounts.
This commit improves performance. Here are some indicative results.
hledger reg -f examples/10000x1000x10.journal
- Maximum residency decreases from 65MB to 60MB (8% decrease)
- Total memory in use decreases from 178MiB to 157MiB (12% decrease)
hledger reg -f examples/10000x10000x10.journal
- Maximum residency decreases from 69MB to 60MB (13% decrease)
- Total memory in use decreases from 198MiB to 153MiB (23% decrease)
hledger bal -f examples/10000x1000x10.journal
- Total heap usage decreases from 6.4GB to 6.0GB (6% decrease)
- Total memory in use decreases from 178MiB to 153MiB (14% decrease)
hledger bal -f examples/10000x10000x10.journal
- Total heap usage decreases from 7.3GB to 6.9GB (5% decrease)
- Total memory in use decreases from 196MiB to 185MiB (5% decrease)
hledger bal -M -f examples/10000x1000x10.journal
- Total heap usage decreases from 16.8GB to 10.6GB (47% decrease)
- Total time decreases from 14.3s to 12.0s (16% decrease)
hledger bal -M -f examples/10000x10000x10.journal
- Total heap usage decreases from 108GB to 48GB (56% decrease)
- Total time decreases from 62s to 41s (33% decrease)
If you never directly use the constructor Mixed or pattern match against
it then you don't need to make any changes. If you do, then do the
following:
- If you really care about the individual Amounts and never normalise
your MixedAmount (for example, just storing `Mixed amts` and then
extracting `amts` as a pattern match, then use should switch to using
[Amount]. This should just involve removing the `Mixed` constructor.
- If you ever call `mixed`, `normaliseMixedAmount`, or do any sort of
amount arithmetic (+), (-), then you should replace the constructor
`Mixed` with the function `mixed`. To extract the list of Amounts, use
the function `amounts`.
- If you ever call `normaliseMixedAmountSquashPricesForDisplay`, you can
replace that with `mixedAmountStripPrices`. (N.B. this does something
slightly different from `normaliseMixedAmountSquashPricesForDisplay`,
but I don't think there's any use case for squashing prices and then
keeping the first of the squashed prices around. If you disagree let
me know.)
- Any remaining calls to `normaliseMixedAmount` can be removed, as that
is now the identity function.
existing representation is small enough.
Previously the JSON representation of Decimal was rounded to 10 points
of precision before serialising. This sometimes results in an
unnecessary increase of precision.
It now uses the same JSON representation as Maybe Word8. This means that
the JSON serialisation is now broadly compatible with that used before the
commit f6fa76bba7, differing only in
how it handles numbers outside Word8 and that it can now produce null
for NaturalPrecision.
Comparing two Quantity (either with (==) or compare) does a lot of
normalisation (calling roundMax) which is unnecessary if we're comparing
to zero. Do things more directly to save work.
For reg -f examples/10000x10000x10.journal, this results in
- A 12% reduction in heap allocations, from 70GB to 62GB
- A 14% reduction in (profiled) time, from 79s to 70s
Results for bal -f examples/10000x10000x10.journal are of the same order
of magnitude.
rather than lists. This is probably not an enormous performance sink in real
situations, but it takes a huge amount of time and memory in our
benchmarks (specifically 10000x10000x10.journal).
For bal -f examples/10000x10000x10.journal, this results in
- A 23% reduction in heap allocation, from 27GiB to 21GiB
- A 33% reduction in (profiled) time running, from 26.5s to 17.9s
former being a simple wrapper around the latter.
This removes the need for the showNormalised option, as showMixedAmountB
will always showNormalised and showAmountsB will never do so.
We also strip prices from MixedAmount before displaying if not displaying prices.
Exceptions are for dealing with the pamount field, which is really just
dealing with an unnormalised list of amounts.
This creates an API for dealing with MixedAmount, so we never have to
access the internals outside of Hledger.Data.Amount.
Also remove a comment, since it looks like #1207 has been resolved.
supplant the old interface, which relied on the Num typeclass.
MixedAmount did not have a very good Num instance. The only functions
which were defined were fromInteger, (+), and negate. Furthermore, it
was not law-abiding, as 0 + a /= a in general. Replacements for used
functions are:
0 -> nullmixedamt / mempty
(+) -> maPlus / (<>)
(-) -> maMinus
negate -> maNegate
sum -> maSum
sumStrict -> maSum
Also creates some new constructors for MixedAmount:
mixedAmount :: Amount -> MixedAmount
maAddAmount :: MixedAmount -> Amount -> MixedAmount
maAddAmounts :: MixedAmount -> [Amount] -> MixedAmount
Add Semigroup and Monoid instances for MixedAmount.
Ideally we would remove the Num instance entirely.
The only change needed have nullmixedamt/mempty substitute for
0 without problems was to not squash prices in
mixedAmount(Looks|Is)Zero. This is correct behaviour in any case.
price directives after the last transaction/posting date if using
--value=end.
Also enlarges the reportspan to encompass full intervals for budget
goals.
both the quantity and the cost are zero. This is usually what you want,
but if you do only want to check whether the quantity is zero, you
can run mixedAmountStripPrices (or similar) before this.
(multiply|divide)(Mixed)?Amount now also multiply or divide the
TotalPrice if it is present, and the old
(multiply|divide)(Mixed)?AmountAndPrice functions are removed.
internally, closing a big space leak.
This also now combines Amounts with TotalPrices in the same commodity
when normalising; amounts with TotalPrices were previously never
combined.
aquantity.
Journal entries still require a positive @@ price, but now the sign is
set after parsing, rather than when converting in amountToCost.
The reason for this change is that, if we're going to perform arithmetic
on Amount with TotalCost, then the presence of aquantity=0 means that
amountToCost would render the total cost as 0, because signum 0 == 0.
This makes journal entries like the following impossible to balance:
2000-01-01
a 0 @@ 10 A
b -10 A
add --debug=1 shows the top hits for similar past transactions.
added:
Hledger.Cli.Utils.journalSimilarTransaction
provides --debug=1 output
changed:
Hledger.Cli.Commands.Add.transactionsSimilarTo -> Hledger.Data.Journal.journalTransactionsSimilarTo
now takes an extra number-of-results argument
It would not add the tag when a comment already existed.
This affected hledger-print-location.hs and probably
the generated-transaction: tag in periodic transactions.
For clarity; infer-value was too vague. The old spelling remains
supported for compatibility, but is now deprecated.
When typing, --infer-market or even --infer (for now) is sufficient.
On the accounts screen and register screen we round amounts according
to commodity styles, but when you drill down to a transaction you
probably want to see the unrounded amounts.
Ensures parseable and more sensible-looking output in more cases, and behaves more like Ledger's print.
There is still an issue with adding trailing zeroes, which would be nice to prevent.
independently.
You can now combine costing and valuation, for example "--cost
--value=then" will first convert to costs, and then value according to
the "--value=then" strategy. Any valuation strategy can be used with or
without costing.
If multiple valuation and costing strategies are specified on the
command line, then if any of them include costing
(-B/--cost/--value=cost) then amounts will be converted to cost, and for
valuation strategy the rightmost will be used.
--value=cost is deprecated, but still supported and is equivalent to
--cost/-B. --value=cost,COMM is no longer supported, but this behaviour can be
achieved with "--cost --value=then,COMM".
costing and valuation.
This currently is given a dummy NoCost argument and is equivalent to
"maybe id (*ApplyValuation ...)", but provides a constant interface so
that internal behaviour can be changed freely.
Also adds a postingDate argument to amountApplyValuation, and re-orders
the ValuationType and (Transaction/Posting) arguments to
(transaction/posting)ApplyValuation, to be consistent with
amountApplyValuation.
Searching for prices during valuation no longer now properly excludes
price loops, avoiding near infinite looping with certain
configurations of market prices. Also we now always use a direct price
when available, rather than searching unnecessarily.
Price searching progress info, useful for troubleshooting, is now
displayed with --debug=2.
There could still be some corner cases we don't handle correctly. We
now give up with an error message if the searched price chains get too
long (> 1000). More importantly, we should also give up if the search
iterates too many times, but this is not done yet.
