SMorgan:
This PR aims to accomplish two major goals:
- Get boring parent ellision working for multiBalanceReport
- Remove the special BalanceReport code, and just use multiBalanceReport
I believe it does both, with the following additional benefits:
A refactor of multiBalanceReportWith, to make the structure easier to follow, and with a clearer division of responsibilities
All decisions for how an account name is to be displayed are now made in multiBalanceReport, rather than scattered around the code base
Some miscellaneous improvements in account name rendering, including --drop now working with MultiBalanceReports, and addressing some of #373
Algorithmic changes:
- Using HashMap AccountName (Map DateSpan Account) instead of [[MixedAmount]] is new. I admit I didn't profile this change (though given the nubs and lookups, I thought it was appropriate), so I'm glad it produces a speedup.
- Producing the starting balances no longer calls the whole balanceReport, just the first few functions to get what it needs.
- displayedAccounts is completely rewritten. Perhaps one subtle thing to note is that in tree mode it no longer excludes nodes with zero inclusive balance unless they also have zero exclusive balance.
SMichael:
I'll mark the passing of the old multiBalanceReport, into which I poured many an hour :). It is in a way the heart (brain ?) of hledger - the key feature of ledgerlikes (balance report) and a key improvement introduced by hledger (tabular multiperiod balance reports). You have split that 300-line though well documented function into modular parts, which could be a little harder to understand in detail but are easier to understand in the large and more amenable to further refactoring. Then you fixed some old limitations (boring parent eliding in multi period balance reports, --drop with tree mode reports), allowing us to drop the old balanceReport and focus on just the new multiBalanceReport. And for representing the tabular data you replaced the semantically correct but inefficient list of lists with a map of maps, speeding up many-columned balance reports significantly (~40%). Last and not least you made it really easy to review. Thanks @Xitian9, great work.
This works with glob patterns too, applying the prefix to each path.
This can be useful when included files don't have the standard file
extension, eg:
include timedot:2020*.md
As mentioned by netvor on IRC, the unbalanced transaction error was
not too clear when postings all have the same sign.
Some other wording has been clarified, and the main error message is
now shown on multiple lines for readability (at the cost of
predictability/grepability..)
There's also a probably unnoticeable change: selecting which parts of
the error to show is now based on display precisions (reusing the
balanced check logic), rather than original precisions.
The ordering of journalfieldnames is changed in order to comply with the requirement stated in the comment:
"Names must precede any other name they contain, for the parser".
If left unchanged, "account1" would precede "account11", "account12", and so on, which would break the parsing.
With the new ordering, "account11" precedes "account1".
We now accept (but still ignore) a fixed or nonfixed ({=} or {}) lot
price following a posting amount, and it may appear before or after a
transaction price (@ or @@). And it may no longer appear after a
balance assertion.
Also: fixedlotpricep renamed to lotpricep, now also parses non-fixed
lot prices. A bit of amount parsers cleanup.
orgstruct-mode was dropped from org 9.2, and I shouldn't have been
forcing it on anyway.
The new config allows its "replacement", outshine-mode, to do similar
code folding when you press tab on any of the lines matching
outline-regexp. But only if you patch it as mentioned at
https://github.com/alphapapa/outshine/issues/77.
Enable it by, eg: (add-hook 'haskell-mode-hook 'outshine-mode)
The change for hledger 1.17.1 broke one of my csv rules, where I used
`amount` but then tried to override it with `amountN`s in a
conditional block; the two clashed. Now in that situation any
`amountN`s take precedence, causing `amount` to be ignored entirely.
Also clarified the "too many non-zero amounts" error message a bit.
A rewrite and simplification of the posting-generating code. The
"special handling for pre 1.17 rules" should now be less noticeable.
amount1/amount2 no longer force a second posting or explicit amounts
on both postings. (Only amount/amount-in/amount-out do that.)
Error messages and handling of corner cases may be more robust, also.
The include directive now tries just one reader, based on the file
extension and defaulting to journal, like the rest of hledger.
(It doesn't yet handle a reader prefix.)
Reader-finding utilities have moved from Hledger.Read to
Hledger.Read.JournalReader so the include directive can use them.
Reader changes:
- rExperimental flag removed
- old rParser renamed to rReadFn
- new rParser field provides the actual parser.
This seems to require making Reader a higher-kinded type, unfortunately.
For a long time hledger has auto-detected the file format when it's
not known, eg when reading from a file with unusual extension (like
.dat or .txt), or from standard input (-f-), or when using the include
directive (which currently ignores file extensions).
Auto-detecting has been done by trying all readers until one succeeds.
This could guess wrong in some cases, but it was so rare that it has
been working fine.
Recently, more conveniences have been added to timedot format,
increasing its overlap with journal format, which makes this kind of
auto-detection unreliable.
Auto-detection and auto-detection failures are (probably) still pretty
rare in practice. But when it does happen it's confusing, giving
misleading errors or false successes (eg printing timedot entries
instead of a journal error).
For predictability and to minimise confusion, hledger no longer tries
to guess; when there's no file extension or reader prefix, it assumes
journal format. To specify one of the other formats, you must use a
standard file extension (.timeclock, .timedot, .csv, .ssv, .tsv), or a
reader prefix (-f csv:foo.txt, -f timedot:-).
For now, the include directive still tries to autodetect
(journal/timeclock/timedot), and this can't be overridden; it will be
fixed later.
Experimental; testing and feedback welcome.
Now, org headlines before the first day entry are ignored,
regardless of content.
Note, blank lines inside a day entry are not allowed, currently.
It's now easier to be both valid journal and valid timedot at the same
time, so guessing the format of stdin is unreliable, and some tests
are failing. See following commit.
Amounts in JSON are now rendered as simple Numbers with up to 10
decimal places, instead of Decimal objects which would in some cases
have 255 digits, too many for most JSON parsers to handle.
A provisional fix, see the comment in Json.hs for more detail.
Renamed: writeValidJournal -> writeJournalTextIfValidAndChanged
Added comments clarifying line ending behaviour of:
add, import, appendToJournalFileOrStdout, readFilePortably,
writeFileWithBackupIfChanged, writeJournalTextIfValidAndChanged
Summary of current behaviour:
- hledger add and import commands will append with (at least some)
unix line endings, possibly causing the file to have mixed line
endings
- hledger-web edit and upload forms will write the file with
the current system's native line endings, ie changing all
line endings if the file previously used foreign line endings.
D directives are now fully equivalent to commodity directives for
setting a commodity's display style. (Previously it was equivalent to
a posting amount, so it couldn't limit the number of decimal places.)
When both kinds of directive exist, commodity directives take precedence.
When there are multiple D directives in the journal, only the last one
affects display style.
Stop exporting journalAmounts, overJournalAmounts, traverseJournalAmounts.
Rename journalAmounts helper to journalStyleInfluencingAmounts.
D directives are now a little better at influencing amount
canonicalisation, eg in the updated test case.
Fix presumably copy-paste errors
timeclock format has only timeclock lines or empty/comment lines
Update test format to v3, add new tests
Throw error on unexpected clock codes in timeclock format
Fix missing case in pattern matching
(max|min)imum(By)?Def are being replaced by (max|min)imumBound(By)?
but the old functions have the semantics I want and
I don't understand the new ones yet.
This can hide other deprecation warnings in this file,
and presumably the old functions will be removed later,
so this is temporary.
Cf https://github.com/ndmitchell/safe/issues/26
Issue #457 pointed out that commands such as
hledger ui 'amt:>200'
failed. This was becasue the process of dispatching from `hledger ui`
to `hledger-ui` (note addition of `-`) lost the quotes around
`amt:>20` and the `>` character was interpreted as a shell redirection
operator, rather than as part of the argument.
The machinery for quoting or escaping arguements which cointain
characters which require quoting or escaping (thus far whitespace and
quotes) already existed. This solution simply adds shell stdio
redirection characters to this set.
Fixes#457
Hledger.Util.Tests helpers have been cleaned up, and test names are
now shown.
Tests have been cleaned up a bit. Some groups of unnamed tests have
been collapsed into a single named test containing a sequence of
assertions. The test command counts named tests, not assertions, so
the reported unit test count has dropped from 199 to 188.
easytest is not actively maintained and requires an old version of
hedgehog which does not support base-compat 0.11 & ghc 8.8.
This is still using the old easytest helpers, and not displaying test
names properly.
It seems we don't use it at all, and ony says it's not robust with
prices either.
Merge remote-tracking branch 'ony/chores/drop-elide-in-showTransaction'
This commit introduces the commandline argument -%/--percent to show
percentages of the column's total instead of the absolute amounts for
each account in reports. The signs of the values are preserved.
This option is especially useful for the balance and incomestatement
commands.
If there are multiple commodities involved in a report hledger bails
with an error message. This can be avoided by using --cost. Also note
that if one uses -% with the balance command the chances are high that
all numbers are 0. This is due to the fact that by default balance sums
up to zero. If one wants to use -% in a meaningful way with balance one
has to add a query.
In order to keep the implementation as simple as possible --tree has no
influence over how the percentages are calculated, i.e., the percentages
always represent the fraction of the columns total. If one wants to know
the percentages relative to a parent account, one has to use a query to
narrow down the accounts.
This behavior is highly depends on journal. If we want to re-introduce
it we'd better re-consider how transaction entry can be "simplified".
I.e. besides dropping last amount we may drop prices that can be assumed
implicitly.
Note that there is no need to knit it into showTransaction since it
easily achievable with pre-processing (similar to implicit balances
etc).
Invalid transactions generated from CSV will now be rejected.
I updated some csv tests to avoid this, except for 21, which
probably needs more cleanup.
Sometimes trailing empty fields are omitted entirely (including the
commas) in CSV records. (I see this in exported Google spreadsheets.)
Now we don't raise an error in this case, instead we automatically pad
any "short" records with empty fields. Not yet well tested.
Certain journal entries could trigger a bug where we displayed amounts
with the same character for digit group mark and decimal mark. Now if
a comma or period digit group mark is detected, that forces the
decimal mark to be the other character.
And if they did, the stats command would now throw an error.
Changed:
journalApplyCommodityStyles
journalInferCommodityStyles
commodityStylesFromAmounts
fail is moving out of Monad and into it's own MonadFail class.
This will be enforced in GHC 8.8 (I think).
base-compat/base-compat-batteries 0.11.0 have adapted to this,
and are approaching stackage nightly
(https://github.com/commercialhaskell/stackage/issues/4802).
hledger is now ready to build with base-compat-batteries 0.11.0, once
all of our deps do (eg aeson). We are still compatible with the older
0.10.x and GHC 7.10.3 as well.
For now we are using both fails:
- new fail (from Control.Monad.Fail), used in our parsers, imported
via base-compat-batteries Control.Monad.Fail.Compat to work with
older GHC versions.
- old fail (from GHC.Base, exported by Prelude, Control.Monad,
Control.Monad.State.Strict, Prelude.Compat, ...), used in easytest's
Test, since I couldn't find their existing fail implementation to update.
To reduce (my) confusion, these are imported carefully, consistently,
and qualified everywhere as Fail.fail and Prelude.fail, with clashing
re-exports suppressed, like so:
import Prelude hiding (fail)
import qualified Prelude (fail)
import Control.Monad.State.Strict hiding (fail)
import "base-compat-batteries" Prelude.Compat hiding (fail)
import qualified "base-compat-batteries" Control.Monad.Fail.Compat as Fail
-V (and -X) now respects a report end date set with -e/-p/date: when
choosing the valuation date, similar to hledger 1.14 and Ledger.
This means that -V/-X aren't exactly like either --value=end or
--value=now. The "Effect of --value on reports" doc has been extended
accordingly, and much of it has been reworded and made more accurate.
On Windows, ensureJournalFileExists now rejects file paths
containing any problematic trailing dots, to prevent data loss.
This affects the add command and hledger-web's add form.
Errors involving a record like:
2000-01-01,a,"1"
displayed the record with extra spaces:
the CSV record is: "2000-01-01", "a", "1"
which was not accurate or valid RFC-4180.
dropped journalPrices
renamed Price to AmountPrice, AKA "transaction price"
renamed MarketPrice to PriceDirective.
added new MarketPrice (more pure form of PriceDirective without the amount style information)
Prices is now a more efficient data structure, but not used yet.
To reduce confusion, multiperiod balance reports using -H/--historical
or --cumulative, which show end balances, no longer show a Totals
column since summing end balances generally doesn't make sense.
Also the underlying MultiBalanceReport now returns zero for those
totals when in cumulative or historical mode.
This feature turns out to be quite involved, as valuation interacts
with the many report variations. Various bugs/specs have been
fixed/clarified relating to register's running total, balance totals
etc. Eg register's total should now be the sum of the posting amount
values, not the values of the original sums. Current level of support
has been documented.
When valuing at transaction date, we once again do early valuation of
all posting amounts, to get more correct results. variants. This means
--value-at=t can be slower than other valuation modes when there are
many transactions and many prices. This could be revisited for
optimisation when things are more settled.
We need this for choosing a valuation date, otherwise, report
functions would have to be in IO or we'd have to pass in yet another
argument.
It's optional because it's useful to be able to create report opts
purely (I think ?) This is not ideal but maybe not a problem.
Instead of converting all journal amounts to value early on, we now
convert just the report amounts to value, before rendering.
This was basically how it originally worked (for the balance command),
but now it's built in to the four basic reports used by print,
register, balance and their variants - Entries, Postings, Balance,
MultiBalance - each of which now has its own xxValue helper.
This should mostly fix -V's performance when there are many
transactions and prices (the price lookups could still be optimised),
and allow more flexibility for report-specific value calculations.
+------------------------------------------++-----------------+-------------------+--------------------------+
| || hledger.999.pre | hledger.999.1sort | hledger.999.after-report |
+==========================================++=================+===================+==========================+
| -f examples/1000x1000x10.journal bal -V || 1.08 | 0.96 | 0.76 |
| -f examples/2000x1000x10.journal bal -V || 1.65 | 1.05 | 0.73 |
| -f examples/3000x1000x10.journal bal -V || 2.43 | 1.58 | 0.84 |
| -f examples/4000x1000x10.journal bal -V || 4.39 | 1.96 | 0.93 |
| -f examples/5000x1000x10.journal bal -V || 7.75 | 2.99 | 1.07 |
| -f examples/6000x1000x10.journal bal -V || 11.21 | 3.72 | 1.16 |
| -f examples/7000x1000x10.journal bal -V || 16.91 | 4.72 | 1.19 |
| -f examples/8000x1000x10.journal bal -V || 27.10 | 9.83 | 1.40 |
| -f examples/9000x1000x10.journal bal -V || 39.73 | 15.00 | 1.51 |
| -f examples/10000x1000x10.journal bal -V || 50.72 | 25.61 | 2.15 |
+------------------------------------------++-----------------+-------------------+--------------------------+
There's one new limitation, not yet resolved: -V once again can pick a
valuation date in the future, if no report end date is specified and
the journal has future-dated transactions. We prefer to avoid that,
but reports currently are pure and don't have access to today's date.
-V is still quite a bit slower than no -V, but not as much as before:
+===========================================================++=======+
| hledger.999.pre -f examples/10000x10000x10.journal bal || 5.20 |
| hledger.999.pre -f examples/10000x10000x10.journal bal -V || 57.20 |
| hledger.999 -f examples/10000x10000x10.journal bal || 5.34 |
| hledger.999 -f examples/10000x10000x10.journal bal -V || 17.50 |
+-----------------------------------------------------------++-------+
I needed to be more careful about ordering, as johannesgerer's original
code was, and the tests missed it. I think I have it now.
Found the PR whose code I have been reworking, it was #438.
Hopefully this is will do it. This restores the past behaviour:
- parsing prices in balance assertions/assignments
- ignoring them in assertions
- using them in assignments
- and printing them
and clarifies tests and docs.
Going with option 1b from the issue: calculated and asserted amounts
are compared exactly, disregarding display precision.
But now balance assertion failure messages show those exact amounts at
full precision, avoiding confusion.
Surprisingly, balance assertions were checking to maximum precision,
which meant it was possible, with a display-precision-limiting
commodity directive, to have a failing assertion with the error
message showing asserted and actual amounts that looked the same.
Now we round the calculated account balance (but not the asserted
balance) to display precision before comparing. This should ensure
assertions always behave as you would expect from visual inspection.
This should eventually include accountnames from transaction
modifiers (if `--auto` is enabled), or periodic transactions (if
`--forecast` is enabled).
- parse a period expression by first extracting words separated by
single spaces, then by "re-parsing" this text with 'periodexprp'
- this way, the period expression parsers do not need to know about
the single- or double-space rules
A different approach: instead of converting to unit prices and fiddling
with the display precision, just multiply the total prices by the same
multiplier (and keep them positive).
This seems a little more natural. I'm not sure if one of these will be
more robust than the other.
Divide/multiply amounts *and* their total price, if they have one.
Helpful for keeping transactions balanced when transaction modifiers are
multiplying amounts.
Transaction modifier multipliers have never multiplied total-priced amounts
correctly (and prior to hledger 1.10, this could generate unbalanced
transactions).
Now, the generated postings in this situation will have unit prices,
and an extra digit of display precision. This helps ensure that
the modified transaction will remain balanced. I'm not sure yet if
it's guaranteed.
Noticed by peti: showTransaction could sometimes hide the last posting's
amount even if one of the other posting amounts was already implcit,
producing invalid transaction output.
* journal: Get rid of `journalFinalise` and use granular functions
Complete the process started in 53b3e2bd. This gets rid of the
`journalFinalise` function and uses the smaller steps, in order to
have more granular control.
* journal: Change order of operations in finalization
We want to make sure that we add the filepath after the order is
reversed, so the added filepath is on the head and not the tail (as it
would be if it were reversed after it was added).
* journal: Refine granular finalization functions
This commit fixes two of the granular finalization functions:
1. Rename `journalSetTime` to `journalSetLastReadTime` and improve
documentation.
2. Remove `journalSetFilePath`. It's redundant with `journalAddFile`
currently in `Hledger.Read.Common`. The only difference between the
functions is where the file is added (we keep the one in which it
is added to the tail), so we change the position vis-a-vis
reversal.
`journalFinalise` is only used in the `parseAndFinaliseJournal`
functions, but it needs to be run differently at different stages when
transaction modifiers are applied. This change breaks it into smaller
functions, and uses those smaller parts in `parseAndFinaliseJournal`
as needed.
Previously we ran if `--auto` was set. But this adds a small
performance hit if `--auto` becomes default. Now we only run twice if
there are transactionModifiers AND `--auto` is set. So even if auto is
specified, there will be no penalty if there are no modifiers.
Currently, automated transactions are added before the journal is
finalized. This means that no inferred values will be picked up. We
change the procedure, if `auto_` is set, to
1. first run `journalFinalise` without assertion checking (assertions
might be wrong until automated transactions), but with reordering
2. Insert transaction modifiers
3. Run `journalFinalise` again, this time with assertion checking as
set in the options, and without reordering.
If `auto_` is not set, all works as before.
Closes: #893
Currently `journalFinalise` always reverses the order of
entries. However, if there are automated transactions, we might need
to run it twice. This adds a boolean flag to make reordering
optional. This will be used in the `parseAndFinaliseJournal`
functions.
These commands now detect the account types declared by account directives.
Whenever such declarations are not present, built-in regular expressions
are used, as before.
