dropped journalPrices
renamed Price to AmountPrice, AKA "transaction price"
renamed MarketPrice to PriceDirective.
added new MarketPrice (more pure form of PriceDirective without the amount style information)
Prices is now a more efficient data structure, but not used yet.
This feature turns out to be quite involved, as valuation interacts
with the many report variations. Various bugs/specs have been
fixed/clarified relating to register's running total, balance totals
etc. Eg register's total should now be the sum of the posting amount
values, not the values of the original sums. Current level of support
has been documented.
When valuing at transaction date, we once again do early valuation of
all posting amounts, to get more correct results. variants. This means
--value-at=t can be slower than other valuation modes when there are
many transactions and many prices. This could be revisited for
optimisation when things are more settled.
Instead of converting all journal amounts to value early on, we now
convert just the report amounts to value, before rendering.
This was basically how it originally worked (for the balance command),
but now it's built in to the four basic reports used by print,
register, balance and their variants - Entries, Postings, Balance,
MultiBalance - each of which now has its own xxValue helper.
This should mostly fix -V's performance when there are many
transactions and prices (the price lookups could still be optimised),
and allow more flexibility for report-specific value calculations.
+------------------------------------------++-----------------+-------------------+--------------------------+
| || hledger.999.pre | hledger.999.1sort | hledger.999.after-report |
+==========================================++=================+===================+==========================+
| -f examples/1000x1000x10.journal bal -V || 1.08 | 0.96 | 0.76 |
| -f examples/2000x1000x10.journal bal -V || 1.65 | 1.05 | 0.73 |
| -f examples/3000x1000x10.journal bal -V || 2.43 | 1.58 | 0.84 |
| -f examples/4000x1000x10.journal bal -V || 4.39 | 1.96 | 0.93 |
| -f examples/5000x1000x10.journal bal -V || 7.75 | 2.99 | 1.07 |
| -f examples/6000x1000x10.journal bal -V || 11.21 | 3.72 | 1.16 |
| -f examples/7000x1000x10.journal bal -V || 16.91 | 4.72 | 1.19 |
| -f examples/8000x1000x10.journal bal -V || 27.10 | 9.83 | 1.40 |
| -f examples/9000x1000x10.journal bal -V || 39.73 | 15.00 | 1.51 |
| -f examples/10000x1000x10.journal bal -V || 50.72 | 25.61 | 2.15 |
+------------------------------------------++-----------------+-------------------+--------------------------+
There's one new limitation, not yet resolved: -V once again can pick a
valuation date in the future, if no report end date is specified and
the journal has future-dated transactions. We prefer to avoid that,
but reports currently are pure and don't have access to today's date.
-V is still quite a bit slower than no -V, but not as much as before:
+===========================================================++=======+
| hledger.999.pre -f examples/10000x10000x10.journal bal || 5.20 |
| hledger.999.pre -f examples/10000x10000x10.journal bal -V || 57.20 |
| hledger.999 -f examples/10000x10000x10.journal bal || 5.34 |
| hledger.999 -f examples/10000x10000x10.journal bal -V || 17.50 |
+-----------------------------------------------------------++-------+
I needed to be more careful about ordering, as johannesgerer's original
code was, and the tests missed it. I think I have it now.
Found the PR whose code I have been reworking, it was #438.
Going with option 1b from the issue: calculated and asserted amounts
are compared exactly, disregarding display precision.
But now balance assertion failure messages show those exact amounts at
full precision, avoiding confusion.
Surprisingly, balance assertions were checking to maximum precision,
which meant it was possible, with a display-precision-limiting
commodity directive, to have a failing assertion with the error
message showing asserted and actual amounts that looked the same.
Now we round the calculated account balance (but not the asserted
balance) to display precision before comparing. This should ensure
assertions always behave as you would expect from visual inspection.
This should eventually include accountnames from transaction
modifiers (if `--auto` is enabled), or periodic transactions (if
`--forecast` is enabled).
A different approach: instead of converting to unit prices and fiddling
with the display precision, just multiply the total prices by the same
multiplier (and keep them positive).
This seems a little more natural. I'm not sure if one of these will be
more robust than the other.
Divide/multiply amounts *and* their total price, if they have one.
Helpful for keeping transactions balanced when transaction modifiers are
multiplying amounts.
Transaction modifier multipliers have never multiplied total-priced amounts
correctly (and prior to hledger 1.10, this could generate unbalanced
transactions).
Now, the generated postings in this situation will have unit prices,
and an extra digit of display precision. This helps ensure that
the modified transaction will remain balanced. I'm not sure yet if
it's guaranteed.
Noticed by peti: showTransaction could sometimes hide the last posting's
amount even if one of the other posting amounts was already implcit,
producing invalid transaction output.
* journal: Get rid of `journalFinalise` and use granular functions
Complete the process started in 53b3e2bd. This gets rid of the
`journalFinalise` function and uses the smaller steps, in order to
have more granular control.
* journal: Change order of operations in finalization
We want to make sure that we add the filepath after the order is
reversed, so the added filepath is on the head and not the tail (as it
would be if it were reversed after it was added).
* journal: Refine granular finalization functions
This commit fixes two of the granular finalization functions:
1. Rename `journalSetTime` to `journalSetLastReadTime` and improve
documentation.
2. Remove `journalSetFilePath`. It's redundant with `journalAddFile`
currently in `Hledger.Read.Common`. The only difference between the
functions is where the file is added (we keep the one in which it
is added to the tail), so we change the position vis-a-vis
reversal.
`journalFinalise` is only used in the `parseAndFinaliseJournal`
functions, but it needs to be run differently at different stages when
transaction modifiers are applied. This change breaks it into smaller
functions, and uses those smaller parts in `parseAndFinaliseJournal`
as needed.
Currently `journalFinalise` always reverses the order of
entries. However, if there are automated transactions, we might need
to run it twice. This adds a boolean flag to make reordering
optional. This will be used in the `parseAndFinaliseJournal`
functions.
These commands now detect the account types declared by account directives.
Whenever such declarations are not present, built-in regular expressions
are used, as before.
Previously you had to use one of the standard english account names
(assets, liabilities..) for top-level accounts, if you wanted to use
the bs/bse/cf/is commands.
Now, account directives can specify which of the big five categories
an account belongs to - asset, liability, equity, revenue or expense -
by writing one of the letters A, L, E, R or X two or more spaces after
the account name (where the numeric account code used to be).
This might change. Some thoughts influencing the current syntax:
- easy to type and read
- does not require multiple lines
- does not depend on any particular account numbering scheme
- allows more types later if needed
- still anglocentric, but only a little
- could be treated as syntactic sugar for account tags later
- seems to be compatible with (ignored by) current Ledger
The current design permits unlimited account type declarations anywhere
in the account tree. So you could declare a liability account somewhere
under assets, and maybe a revenue account under that, and another asset
account even further down. In such cases you start to see oddities like
accounts appearing in multiple places in a tree-mode report. In theory
the reports will still behave reasonably, but this has not been tested
too hard. In any case this is clearly too much freedom. I have left it
this way, for now, in case it helps with:
- modelling contra accounts ?
- multiple files. I suspect the extra expressiveness may come in handy
when combining multiple files with account type declarations,
rewriting account names, apply parent accounts etc.
If we only allowed type declarations on top-level accounts, or
only allowed a single account of each type, complications seem likely.
- In anticipation of megaparsec 7, which removes support for stacks of
include files (as far as I can tell)
- Intended for the 'StateT Journal' layer of the parser
- A stack of include files would be better in a 'ReaderT' layer, but I
don't want to add another layer to the parser
- Intended for detecting cycles of include files
- Potential issue: for proper error messages for include file cycles,
we must remember to provide the filepath of the root journal file via
the initial journal state passed to a 'JournalParser'; I imagine
that we may forget to do so because in all other cases it is okay
not to do so.
A bunch of account sorting changes that got intermingled.
First, account codes have been dropped. They can still be parsed and
will be ignored, for now. I don't know if anyone used them.
Instead, account display order is now controlled by the order of account
directives, if any. From the mail list:
I'd like to drop account codes, introduced in hledger 1.9 to control
the display order of accounts. In my experience,
- they are tedious to maintain
- they duplicate/compete with the natural tendency to arrange account
directives to match your mental chart of accounts
- they duplicate/compete with the tree structure created by account
names
and it gets worse if you think about using them more extensively,
eg to classify accounts by type.
Instead, I plan to just let the position (parse order) of account
directives determine the display order of those declared accounts.
Undeclared accounts will be displayed after declared accounts,
sorted alphabetically as usual.
Second, the various account sorting modes have been implemented more
widely and more correctly. All sorting modes (alphabetically, by account
declaration, by amount) should now work correctly in almost all commands
and modes (non-tabular and tabular balance reports, tree and flat modes,
the accounts command). Sorting bugs have been fixed, eg #875.
Only the budget report (balance --budget) does not yet support sorting.
Comprehensive functional tests for sorting in the accounts and balance
commands have been added. If you are confused by some sorting behaviour,
studying these tests is recommended, as sorting gets tricky.
Custom Show instances were obscuring important details in test failure
output again. The best policy seems to be: stick with default derived
Show instances as far as possible, but when necessary customize them
to conform to haskell syntax so pretty-show can do its thing (eg when
they contain Day values, cf https://github.com/haskell/time/issues/101).
This makes skipping/unskipping tests easier, and improves readability
a bit.
Note it's also possible to just write the test name with no preceding
function, when the type is constrained (see Journal.hs).
Amount's default show instance hid important details, making eg test
failures hard to understand. Showing full detail required increasing
the debug level which was inconvenient.
Now it has a single show instance which shows more information, is
fairly compact, and is pretty-printable with pretty-show.
Ellipses (..) in the output indicate where fields are
- not shown in full detail, and/or
- shown in pseudo syntax (double quoted) to work with pretty-show.
ghci> usd 1
OLD:
Amount {acommodity="$", aquantity=1.00, ..}
NEW:
Amount {acommodity = "$", aquantity = 1.00, aprice = NoPrice, astyle = AmountStyle "L False 2 Just '.' Nothing..", amultiplier = False}
MixedAmount's show instance is unchanged, but showMixedAmountDebug
is affected by this change:
ghci> putStrLn $ showMixedAmountDebug $ Mixed [usd 1]
OLD:
Mixed [Amount {acommodity="$", aquantity=1.00, aprice=, astyle=AmountStyle {ascommodityside = L, ascommodityspaced = False, asprecision = 2, asdecimalpoint = Just '.', asdigitgroups = Nothing}}]
NEW:
Mixed [Amount {acommodity="$", aquantity=1.00, aprice=, astyle=AmountStyle "L False 2 Just '.' Nothing.."}]
This removes transactionModifierToFunction's extra query parameter;
the rewrite command sets it in the TransactionModifier instead, which
I think is equivalent. I had to change one functional test, but it
seems correct now, so perhaps it wasn't working right before ?
For Data/Dates.hs in particular:
- Changed `SimpleTextParser` to `TextParser m` for all parsers
- Changed `string` to the case-insensitive `string'` to match the
behaviour of `T.toLower` found in `parsePeriodExpr`
- export `periodexprp` for "direct" use
base-compat-batteries provides the same API across more ghc versions
than base-compat does, at the cost of more dependencies. Eg it exports
Prelude.Compat ((<>)) with ghc 7.10/base 4.8, which we expect.
My belief is that several of our deps already require it so the added
cost is not too great. We should probably go back to base-compat when
possible though, eg when we stop supporting ghc 7.10.
The new version of our package set apparently contains both base-compat and
base-compat-batteries in its transitive closure. This breaks the doctest suite,
which just imports everything into scope when the tests are run, thereby making
module names like Prelude.Compat ambiguous.
It now works slightly differently. Eg:
- <unbudgeted>'s subaccounts are hidden by default
- --show-unbudgeted shows all unbudgeted accounts, including subaccounts of budgeted parents
- --show-unbudgeted doesn't affect the grouping under <unbudgeted>
IMHO it's a nice simplification and increase in consistency, while still meeting the original intent.
A commodity directive that doesn't specify the decimal point character
increases ambiguity and the chance of misparsing numbers, especially
as it overrides all style information inferred from the journal amounts.
In some cases it caused amounts with a decimal point to be parsed as if
with a digit group separator so 1.234 became 1234.
We could augment it with extra info from the journal amounts, when available,
but it would still be possible to be ambiguous, and that won't be obvious.
A commodity directive is what we recommend to nail down the style.
It seems the simple and really only way to do this reliably is to require
an explicit decimal point character. Most folks probably do this already.
Unfortunately, it makes another potential incompatiblity with ledger and
beancount journals. But the error message will be clear and easy to
work around.
Inferred amounts now have the appropriate standard amount style applied.
And when checking for balanced transactions, amount styles declared with
commodity directives are also used (previously only inferred amount styles were).
