Bracketed posting dates were fragile; they worked only if you wrote full
10-character dates. Also some semantics were a bit unclear. Now they
should be robust, and have been documented more clearly. This is a
legacy undocumented Ledger syntax, but it improves compatibility and
might be preferable to the more verbose "date:" tags if you write
posting dates often (as I do).
Internally, bracketed posting dates are no longer considered to be tags.
Journal comment, tag, and posting date parsers have been reworked, all
with doctests. Also the journal parser types generally have been
tightened up and clarified, making it much easier to know how to combine
and run them. There's now
-- | A parser of strings with generic user state, monad and return type.
type StringParser u m a = ParsecT String u m a
-- | A string parser with journal-parsing state.
type JournalParser m a = StringParser JournalContext m a
-- | A journal parser that runs in IO and can throw an error mid-parse.
type ErroringJournalParser a = JournalParser (ExceptT String IO) a
and corresponding convenience functions (and short aliases) for running them.
We now parse account directives, like Ledger's. We don't do anything
with them yet. The default parent account feature must now be spelled
"apply account"/"end apply account".
Amount display style canonicalisation code and terminology has been
clarified a bit. Individual amounts still have styles; from these we
derive the standard "commodity styles". In user docs, we might call
these "commodity formats" since a Ledger-compatible commodity directive
would use the "format" keyword.
Since market price amounts didn't contribute to the canonical commodity
styles, they were being reset to the null style. And this propagated to
the reported amounts when -V was in effect, causing much confusion.
Now, market prices contribute to canonicalisation and the expected
styles are preserved even with -V.
cf https://github.com/simonmichael/hledger/issues/131#issuecomment-133545140
print now always right-aligns the amounts in an entry, even when they
are wider than 12 characters.
If there is a price, it's considered part of the amount for
right-alignment. Maybe it would be nicer to put amounts and prices in
separate columns ? That will get a little complicated, needs more
discussion/design.
Also some cleanup of postingAsLines.
The print command wasn't lining up amounts with wide chars in account
names, fixed it properly this time. Transaction and Posting's Show instances
should also be wide-char-aware now.
Simple (non-multicolumn) balance reports containing wide characters
should now align correctly (in apps and fonts that show wide chars as
double width). Likewise, the print command.
Wide characters, eg chinese/japanese/korean characters, are typically
rendered wider than latin characters. In some applications (eg gnome
terminal or osx terminal) and fonts (eg monaco) they are exactly double
width. This is a start at making hledger aware of this. A register
report containing wide characters (in descriptions, account names, or
commodity symbols) should now align its columns correctly, when viewed
with a suitable font and application.
This adds a accountNameApplyAliasesMemo, which memoises the result of
applying a set of aliases (simple and regex) to an account name. In
theory this should reduce more repetitive work, but in practice it
doesn't seem to make a difference, so it's unused for now.
Roughly speaking, the time to apply regular expression account aliases
was O(aliases x transactions), and should now be O(aliases x accounts).
Also, the constant factor was reduced a lot by the recent commit
memoising toRegex. So now, regex aliases should be "free" like simple
aliases - use as many as you want, the slowdown shouldn't be noticeable.
hledger-lib-0.24's "track the commodity of zero amounts when
possible (useful eg for hledger-web's multi-commodity charts)" preserved
the commodity when normalising a zero mixed amount, but not the amount
style. This showed up as occasionally incorrect amount style (commodity
symbol placement, decimal point character, etc.) in balance reports with
certain journals, like this:
$ hledger bal
€3000.00 a <------ not using the canonical € style
4000,58€ 1
-1000,58€ D
-3000,00€ e
--------------------
0
I thought this would require a big rewrite of amount arithmetic, but it
seems that just being a little more careful is enough. When normalising
a mixed amount containing multiple zeros in the same commodity, we now
preserve the last zero with its amount style, instead of replacing them
all with a new one.
The register screen is now like the register view in hledger-web (and
other accounting systems), rather than hledger's register command.
This means:
- it shows transactions affecting a particular current account, rather
than postings matching a pattern.
- Each line represents a whole transaction.
- The account field shows the *other* account being transacted with.
When there is more than one, they are all listed, abbreviated and
marked with "(split)".
- The amount field shows the effect of the transaction on the current
account; positive for an inflow to this account, negative for an
outflow.
- The balance field should usually show the current account's historic
balance as of the transaction date, even when you change the report
start date. (Not working yet - currently it always shows the running
total).
- Transactions are listed most recent first, currently.
The balance command's --format option (in single-column mode) can now
adjust the rendering of multi-line strings, such as amounts with multiple
commodities. To control this, begin the format string with one of:
%_ - renders on multiple lines, bottom-aligned (the default)
%^ - renders on multiple lines, top-aligned
%, - render on one line, comma-separated
Also the final total (and the line above it) now adapt themselves to a
custom format.
The --format option's OutputFormat type was named confusingly like the
--output-format option. It has been renamed StringFormat to distinguish
it from StorageFormat (aka the data file format, referenced by
--output-format). Related code and types have been consolidated.
Also the (single-column) balance report's item rendering has had
some cleanup.
so we can benchmark things more easily with criterion.
As well as NFData, the Generic instance and a bunch more GHC extensions
seemed necessary. This is a little scary, impact unknown.
Simpler and clearer. We now have "transaction prices" (recorded as part
of transaction amounts) and "market prices" (recorded with P
directives). Both are matters of historical record, also this avoids
confusion with the balance command's "historical balances".
Initial support of market value reporting and currency conversion,
similar in spirit to Ledger's. The balance command now has a -V/--value
flag that converts all the reported amounts using their "default market
price". That is the latest market price (P directive, formerly called
"historical prices") found in the journal for their commodity that is on
or before the report end date.
Unlike Ledger, hledger's -V only uses the market prices recorded with P
directives, ignoring transaction prices recorded as part of posting
amounts (which -B/--cost uses). Using -B and -V together is allowed.
We don't do a good job of calculating good-looking unit prices when the
commodity display precisions are low. Eg when a journal doesn't use any
decimal places, any inferred unit prices are shown by the print command
also with no decimal places, which makes them look wrong.
Now inferred unit prices always have a minimum display precision of 2,
which helps a bit. Could do better.
When a transaction posts to two commodities without specifying the
conversion price, we generate a price which makes it balance
(cf http://hledger.org/manual.html#prices).
Until now, these generated prices were always shown with full precision
(all available decimal digits) so that a manual calculation with the
displayed numbers would agree.
If there's just one posting in the commodity being priced, we can use an
exact total price and the precision is no problem.
But if there are multiple postings in the commodity being priced, we
must show the averaged unit price. This can be an irrational number,
which with our current Decimal-based implementation would display an
excessive 255 decimal digits. So in this case we now set the price's
display precision to the sum of the (max) display precisions of the
commodities involved. An example:
hledgerdev -f- print
<<<
1/1
c C 10.00
c C 11.00
d D -320.00
>>>
2015/01/01
c C 10.00 @ D 15.2381
c C 11.00 @ D 15.2381
d D -320.00
>>>=0
There might still be cases where this will show more price decimal
places than necessary. For now, YAGNI.
A transaction/posting status of ! (pending) was effectively equivalent
to * (cleared). Now it's a separate state, not matched by --cleared.
The new Ledger-compatible --pending flag matches it, and so does
--uncleared. The equivalent search queries are now status:*, status:!
and status: (the old status:1 and status:0 spellings are deprecated).
Since we interpret --uncleared and status: as "any state except cleared",
it's not currently possible to match things which are neither cleared
nor pending.
The regex account aliases added in 0.24 trip up people switching between
hledger and Ledger. (Also they are currently slow).
This change makes the old non-regex aliases the default; they are
unsurprising, useful, and pretty close in functionality to Ledger's.
The new regex aliases are also available; they must be enclosed in
forward slashes. Ledger effectively ignores these, which is ok.
Also clarify docs, refactor, and use the same parser for alias
directives and alias options
NOTE: this is important to correctly build JournalContext
NOTE: currently a list reverse must done at the end,
maybe using a Data.Queue would be more efficient.
Eg recognise that 2014/11/30-2014/12/1 can be abbreviated to
2014/11/30d, similarly 2014/12/31-2015/1/1. Doesn't handle feb 29th
correctly, so eg 2000/2/28-2000/3/1 is wrongly abbreviated to
2000/2/28d.
alias match patterns (the part left of the =) are now case-insensitive
regular expressions matching anywhere in the account name. The
replacement string (the part right of the =) can replace multiple
matches within the account name. The replacement string does not yet
support any of the usual syntax like backreferences.
Previously, a depth:0 query produced an empty report (since there are no
level zero accounts). Now, it aggregates all data into one summary item
with account name "...".
This makes it easier to see the kind of data Gwern was looking for from
register-csv (net worth over time). Eg this shows one line per month
summarising the total of assets and liabilities:
hledger register-csv -- -MHE ^assets ^liabilities depth:0
Single and multi-column balance reports behave similarly.
hledger has represented quantities with floating point (Double) until
now. While this has been working fine in practice, the time has come to
upgrade our number representation to something more principled: Decimal,
for now. As a bonus, this brings a ~30% speed boost to most reports.
We'll keep the old representation(s) around for a while, selectable via
hledger-lib cabal flag, for research/testing/benchmarking purposes. To
build with the old Double representation: cabal install -fdouble
hledger-lib hledger hledger-web
In periodic multicolumn balance reports, column headings now show a more
compact description of common periods (years, half-years, quarters,
months, weeks) for better readability and screen space efficiency.
- show a background color for future and less-than-zero regions
- show points for transactions, not all line corners
- hovering over point shows balance, date, posted amount and transaction
- clicking a point scrolls towards that date
Now that balance assertions are checking only a single commodity, it can
be confusing. Eg say all your amounts are in dollars, an assertion like
"= 0" checked the dollar balance in hledger 0.23 but always succeeds in
hledger 0.24. When an assertion fails, we now report which commodity was
checked to help troubleshooting.
To avoid totally breaking the layout of multicolumn balance reports when
there are multi-commodity balances, these are now rendered on a single
line, comma separated. Simple balance reports still use the old vertical
format for now.
Amount display styles have been reworked a bit; they are now calculated
after journal parsing, not during it. This allows the fix for #196:
we now search through the amounts until a decimal point is detected,
instead of just looking at the first one; likewise for digit groups.
Digit groups are now implemented with a better type.
Digit group size detection has been improved a little:
1000,000 now gives group sizes [3,4,4,...], not [3,3,...], and
10,000 gives groups sizes [3,3,...] not [3,2,2,..].
(To get [3,2,2,...] you'd use eg 00,00,000.)
There are still some old (or new ?) issues; I don't think we handle
inconsistent decimal points & digit groups too well. But for now all
tests pass.
This change means you can make assertions on a multi-commodity account
balance (asserting one commodity at a time). On the flip side, you can
no longer assert the complete balance of an account (new unexpected
commodities will not be detected.) We might restore that ability later,
using the == syntax.
Can be helpful when reading Ledger files, where assertions may have
different semantics; or for getting some answers from your journal
to help you fix your assertions.
Could be called --no-assertions, but this might create surprise when it
has an effect contrary to --no-new-accounts.
I had to add another flag throughout the parsers & journal read
functions, ok for now.
- The CSV reader no longer writes a "(stdin).rules" file when reading
from stdin.
- Selection of reader(s) is now smarter when input is coming from stdin.
Previously, all readers were considered applicable for stdin. This
meant that when reading a CSV file from stdin, the journal and timelog
readers were always tried first, and if the CSV file was unparseable,
you'd see the first (journal) reader's error instead of the CSV
reader's. Now, the readers do some basic content sniffing when
reading stdin, so it generally tries only the one right reader and
we'll see the right errors.
- The read system now has more debug output.
Changes include:
- flat mode now shows exclusive (subaccount-excluding) balances.
This is a deviation from ledger, but seems simpler and clearer
for users and implementors across the various modes.
- in flat mode, --depth now aggregates deeper accounts at the
depth limit, rather than just excluding them from the report.
This is more useful.
- in flat mode, --empty no longer shows parent accounts with
no postings.
- more tests, more debug output, clearer code
Two new multi-column balance report modes show ending balance per
period: `--cumulative`, starting from 0, and `--historical`, starting
from the historical starting balance.
The balance command's specification has been clarified and consolidated
in the Balance.hs haddock. Reports.hs has also had haddock updates. The
old AccountsReport type is now BalanceReport, still used by
single-column balance report. The new MultiBalanceReport type is used by
the multi-column reports.
Amounts and journal values are often rendered too verbosely in debug
output, obscuring what's important. Here we try adjusting the level
of detail in the Show instance based on the global debug level.
Needs more work.
At long last. The main change is a new rules file format that aims to
be more powerful and more intuitive than v1 (hledger 0.19.x and older).
Existing rules files will need to be adapted manually to the new format.
This refactoring fixes an O(n^2) slowdown in the balance command with
large numbers of accounts. It's now speedy, and the implementation is
clearer. To facilitate this, the Account type now represents a tree of
accounts which can easily be traversed up or down (and/or flattened
into a list).
Benchmark on a 2010 macbook:
+-------------------------------------------++--------------+------------+--------+
| || before: | after: | |
| || hledger-0.18 | hledgeropt | ledger |
+===========================================++==============+============+========+
| -f data/100x100x10.journal balance || 0.21 | 0.07 | 0.09 |
| -f data/1000x1000x10.journal balance || 10.13 | 0.47 | 0.62 |
| -f data/1000x10000x10.journal balance || 40.67 | 0.67 | 1.01 |
| -f data/10000x1000x10.journal balance || 15.01 | 3.22 | 2.36 |
| -f data/10000x1000x10.journal balance aa || 4.77 | 4.40 | 2.33 |
+-------------------------------------------++--------------+------------+--------+
- use new query system for command line too, filterspec is no more
- move unit tests near the code they test, run them in bottom up order, add more
- more precise Show instances, used for debugging not ui
