Commit Graph

898 Commits

Author SHA1 Message Date
Stephen Morgan
7fe58f1346 lib: More efficiently check whether Amounts are or look zero.
Comparing two Quantity (either with (==) or compare) does a lot of
normalisation (calling roundMax) which is unnecessary if we're comparing
to zero. Do things more directly to save work.

For reg -f examples/10000x10000x10.journal, this results in
- A 12% reduction in heap allocations, from 70GB to 62GB
- A 14% reduction in (profiled) time, from 79s to 70s
Results for bal -f examples/10000x10000x10.journal are of the same order
of magnitude.
2021-03-23 11:26:30 -07:00
Stephen Morgan
522c8a6ad3 lib: In sorting account names, perform lookups on HashSets and HashMaps,
rather than lists. This is probably not an enormous performance sink in real
situations, but it takes a huge amount of time and memory in our
benchmarks (specifically 10000x10000x10.journal).

For bal -f examples/10000x10000x10.journal, this results in
- A 23% reduction in heap allocation, from 27GiB to 21GiB
- A 33% reduction in (profiled) time running, from 26.5s to 17.9s
2021-03-23 11:26:30 -07:00
Stephen Morgan
7488140608 lib: Do not call showAmount twice for every posting.
For print -f examples/10000x10000x10.journal, this results in
- A 7.7% reduction in heap allocations, from 7.6GB to 7.1GB.
2021-03-21 07:10:27 -07:00
Stephen Morgan
d54e276658 lib: Split showMixedAmountB into showMixedAmountB and showAmountsB, the
former being a simple wrapper around the latter.

This removes the need for the showNormalised option, as showMixedAmountB
will always showNormalised and showAmountsB will never do so.

We also strip prices from MixedAmount before displaying if not displaying prices.
2021-03-21 07:10:27 -07:00
Stephen Morgan
d6a4310d8f lib,cli,ui,bin: Eliminate all uses of Mixed outside of Hledger.Data.Amount.
Exceptions are for dealing with the pamount field, which is really just
dealing with an unnormalised list of amounts.

This creates an API for dealing with MixedAmount, so we never have to
access the internals outside of Hledger.Data.Amount.

Also remove a comment, since it looks like #1207 has been resolved.
2021-03-18 09:47:59 +11:00
Stephen Morgan
dabb3ef82e lib,cli,ui,bin: Create a new API for MixedAmount arithmetic. This should
supplant the old interface, which relied on the Num typeclass.

MixedAmount did not have a very good Num instance. The only functions
which were defined were fromInteger, (+), and negate. Furthermore, it
was not law-abiding, as 0 + a /= a in general. Replacements for used
functions are:
0 -> nullmixedamt / mempty
(+) -> maPlus / (<>)
(-) -> maMinus
negate -> maNegate
sum -> maSum
sumStrict -> maSum

Also creates some new constructors for MixedAmount:
mixedAmount :: Amount -> MixedAmount
maAddAmount :: MixedAmount -> Amount -> MixedAmount
maAddAmounts :: MixedAmount -> [Amount] -> MixedAmount

Add Semigroup and Monoid instances for MixedAmount.
Ideally we would remove the Num instance entirely.

The only change needed have nullmixedamt/mempty substitute for
0 without problems was to not squash prices in
mixedAmount(Looks|Is)Zero. This is correct behaviour in any case.
2021-03-18 09:47:21 +11:00
Simon Michael
2505c69f80 lib: transactionFile 2021-03-03 06:44:25 -08:00
Stephen Morgan
edaaef897b lib: Do not include price directives in journalDateSpan. Only include
price directives after the last transaction/posting date if using
--value=end.

Also enlarges the reportspan to encompass full intervals for budget
goals.
2021-03-03 05:26:55 -08:00
Stephen Morgan
f0655d1c7f lib: (amount|mixedAmount)(Looks|Is)Zero functions now check whether
both the quantity and the cost are zero. This is usually what you want,
but if you do only want to check whether the quantity is zero, you
can run mixedAmountStripPrices (or similar) before this.

(multiply|divide)(Mixed)?Amount now also multiply or divide the
TotalPrice if it is present, and the old
(multiply|divide)(Mixed)?AmountAndPrice functions are removed.
2021-02-23 14:54:39 -08:00
Stephen Morgan
9d527a9926 lib: normaliseHelper now uses a strict Map for combining amounts
internally, closing a big space leak.

This also now combines Amounts with TotalPrices in the same commodity
when normalising; amounts with TotalPrices were previously never
combined.
2021-02-23 14:54:39 -08:00
Stephen Morgan
ecca7f4e0c lib: Distinguish between an Amount having quantity (or rounded quantity 0), and having both quantity and totalprice 0 (or rounded to 0). 2021-02-23 14:54:39 -08:00
Stephen Morgan
81b778a389 lib: Make fields of Amount, AmountPrice, AmountStyle, and DigitGroupStyle strict. 2021-02-23 14:54:39 -08:00
Stephen Morgan
2ada289e28 lib: Include sign in TotalPrice in Amount, rather than relying on the sign of
aquantity.

Journal entries still require a positive @@ price, but now the sign is
set after parsing, rather than when converting in amountToCost.

The reason for this change is that, if we're going to perform arithmetic
on Amount with TotalCost, then the presence of aquantity=0 means that
amountToCost would render the total cost as 0, because signum 0 == 0.
This makes journal entries like the following impossible to balance:
2000-01-01
    a  0 @@ 10 A
    b  -10 A
2021-02-23 14:54:39 -08:00
Stephen Morgan
a65ef7cd19 lib: Consume list immediately in commodityStylesFromAmounts.
This reduced the maximum heap size per thread from ~850K to ~430K in a
real-world register test.
2021-02-23 14:54:39 -08:00
Simon Michael
5e7c4fc7bc add, print --match: prioritise infix matches
If the search description occurs in full within the other description,
that match gets a +0.5 score boost.
2021-02-20 14:08:47 -08:00
Simon Michael
44508f58c8 add, lib: debug output, refactor similar transactions lookup
add --debug=1 shows the top hits for similar past transactions.

added:
Hledger.Cli.Utils.journalSimilarTransaction
 provides --debug=1 output

changed:
Hledger.Cli.Commands.Add.transactionsSimilarTo -> Hledger.Data.Journal.journalTransactionsSimilarTo
 now takes an extra number-of-results argument
2021-02-20 13:41:37 -08:00
Simon Michael
b31e40ceca lib: avoid shady Not (Or []) in account type queries 2021-02-13 14:58:06 -08:00
Simon Michael
6c392e9f43 bs: don't let just a Cash declaration hide Asset accounts
Clarify the account type queries a bit, and don't let a
declaration of account as Cash cause the fallback regex
for Asset accounts to be ignored.
2021-02-13 14:54:04 -08:00
Simon Michael
b9d33a4294 bs: just declaring a Cash account no longer hides Asset accounts
Since Cash is also an Asset, declaring an account as Cash also disabled the fallback regexp for Asset accounts.
2021-02-13 08:09:45 -08:00
Simon Michael
50bfc26ca7 ;lib: commentAddTagNextLine was broken
It would not add the tag when a comment already existed.
This affected hledger-print-location.hs and probably
the generated-transaction: tag in periodic transactions.
2021-02-12 13:56:02 -08:00
Simon Michael
d865ec5d65 lib: refactor: more consistent amount precision helpers
Hledger.Data.Amount:
renamed:
setAmountPrecision -> amountSetPrecision
setFullPrecision -> amountSetFullPrecision
setMixedAmountPrecision -> mixedAmountSetPrecision
added:
mixedAmountSetFullPrecision
2021-02-05 16:09:49 -08:00
Simon Michael
d5ed8090fa ;lib: document AmountPrecision 2021-02-05 16:09:49 -08:00
Simon Michael
37e49ada70 cli: rename --infer-value to --infer-market-price
For clarity; infer-value was too vague. The old spelling remains
supported for compatibility, but is now deprecated.
When typing, --infer-market or even --infer (for now) is sufficient.
2021-02-05 15:16:39 -08:00
Simon Michael
2a4170dd3b ui: transaction: show all decimal places (like print, cf #931)
On the accounts screen and register screen we round amounts according
to commodity styles, but when you drill down to a transaction you
probably want to see the unrounded amounts.
2021-02-04 08:13:55 -08:00
Simon Michael
f52117120b lib: refactor: rename journal/transaction mapping helpers
Hledger.Data.Journal:
changed:
mapJournalTransactions -> journalMapTransactions
mapJournalPostings -> journalMapPostings
removed:
mapTransactionPostings

Hledger.Data.Transaction:
added:
transactionMapPostings
2021-02-04 08:13:55 -08:00
Simon Michael
6101368954 print: always show all decimal places (#931)
Ensures parseable and more sensible-looking output in more cases, and behaves more like Ledger's print.

There is still an issue with adding trailing zeroes, which would be nice to prevent.
2021-02-04 08:13:55 -08:00
Simon Michael
0274b30742 print: don't add decimal places in assertion/assignment amounts either (#1465) 2021-02-04 06:46:09 -08:00
Simon Michael
201f967411 print: show all decimal places in assertion/assignment amounts (#1465)
Overriding the commodity style, per https://hledger.org/hledger.html#commodity-display-style.
2021-02-03 18:11:44 -08:00
Stephen Morgan
c9eb7d1bcf lib,cli,ui: Separate costing from valuation; each can now be specified
independently.

You can now combine costing and valuation, for example "--cost
--value=then" will first convert to costs, and then value according to
the "--value=then" strategy. Any valuation strategy can be used with or
without costing.

If multiple valuation and costing strategies are specified on the
command line, then if any of them include costing
(-B/--cost/--value=cost) then amounts will be converted to cost, and for
valuation strategy the rightmost will be used.

--value=cost is deprecated, but still supported and is equivalent to
--cost/-B. --value=cost,COMM is no longer supported, but this behaviour can be
achieved with "--cost --value=then,COMM".
2021-01-28 13:57:48 -08:00
Stephen Morgan
130739e3ef lib,cli,ui: Introduce *ApplyCostValuation functions, which perform both
costing and valuation.

This currently is given a dummy NoCost argument and is equivalent to
"maybe id (*ApplyValuation ...)", but provides a constant interface so
that internal behaviour can be changed freely.
2021-01-28 13:57:48 -08:00
Stephen Morgan
9881ec9652 lib: Use mapMixedAmount more, decreasing reliance on internal representation of Mixed. 2021-01-28 13:57:48 -08:00
Simon Michael
422329fbfa lib: journalPayeesUsed, journalPayeesDeclaredOrUsed 2021-01-17 16:37:38 -08:00
Simon Michael
db9b034037 lib: Journal: avoid a build failure if monad-extras is installed 2021-01-17 16:37:38 -08:00
Stephen Morgan
217bfc5e74 lib: Rename alignCell to textCell, minor cleanups. 2021-01-15 12:56:48 -08:00
Simon Michael
4952ac22a1 ;lib: valuation: improve --debug=2 output 2021-01-07 08:59:04 -08:00
Stephen Morgan
83110e8820 lib,cli: Extend AtThen valuation to all report types.
Also adds a postingDate argument to amountApplyValuation, and re-orders
the ValuationType and (Transaction/Posting) arguments to
(transaction/posting)ApplyValuation, to be consistent with
amountApplyValuation.
2021-01-05 14:42:09 -08:00
Simon Michael
3d7d5c0db7 new price search that really finds the shortest path (#1443)
This one should also reliably prevent runaway searches in the event of more bugs, giving up after 1000 iterations.
2021-01-04 18:42:51 -08:00
Simon Michael
7c9303a15c lib: valuation: simplify price search code
This version tries counting recursions instead of path length, but I
think they are the same.
2021-01-03 09:49:00 -08:00
Simon Michael
73678393b1 lib: valuation: don't hang when finding prices (fixes #1439)
Searching for prices during valuation no longer now properly excludes
price loops, avoiding near infinite looping with certain
configurations of market prices. Also we now always use a direct price
when available, rather than searching unnecessarily.

Price searching progress info, useful for troubleshooting, is now
displayed with --debug=2.

There could still be some corner cases we don't handle correctly. We
now give up with an error message if the searched price chains get too
long (> 1000). More importantly, we should also give up if the search
iterates too many times, but this is not done yet.
2021-01-02 07:13:07 -08:00
Stephen Morgan
7d3cf1747a lib: Make consistent naming scheme for showMixedAmount* functions,
add conversion between old API and new API in the documentation.
2021-01-02 15:08:09 +11:00
Stephen Morgan
e63138ef7d lib,cli: Assorted fixes for older GHC. 2021-01-02 15:08:09 +11:00
Stephen Morgan
e4e533eb9f lib,cli,ui: Replace some uses of String with Text, get rid of some unpacks, clean up showMixed options. 2021-01-02 15:08:09 +11:00
Stephen Morgan
07a7c3d3a8 lib: Use Text and Text builder only in postingAsLines. 2021-01-02 15:08:09 +11:00
Stephen Morgan
b9c00dce61 lib,cli,ui: Implement all showMixed* functions in terms of DisplayAmountOpts and WideBuilder. 2021-01-02 15:08:09 +11:00
Stephen Morgan
0a686e220e lib: Use AmountDisplayOpts for showAmount*, reducing need for many different named functions. 2021-01-02 15:08:09 +11:00
Stephen Morgan
c86e8a9794 lib: Implement showAmountHelper using AmountBuilder. 2021-01-02 15:08:09 +11:00
Stephen Morgan
5dedec83da lib: showamountquantity shows directly, rather than parsing string output of show instance for Quantity. 2021-01-02 15:08:09 +11:00
Stephen Morgan
f998a791cf lib: Remove unused optional width argument for StringFormat. 2021-01-02 15:08:09 +11:00
Stephen Morgan
462a13cad7 lib,cli: Use Text Builder for Balance commands. 2021-01-02 15:08:09 +11:00
Stephen Morgan
e3ec01c3c6 lib,cli,ui: Use Text for showDate and related. 2021-01-02 15:08:09 +11:00
Stephen Morgan
74b296f865 lib,cli: Make showTransaction return Text rather than String. 2021-01-02 15:08:09 +11:00
Stephen Morgan
12a6435c51 lib: Add wrap convenience function. 2021-01-02 15:08:09 +11:00
Stephen Morgan
646ee0bce5 cli: Using Text Builder for posting reports. 2021-01-02 15:08:09 +11:00
Damien Cassou
b47d582583 ;check: fixes 2020-12-30 18:08:31 -08:00
Simon Michael
3c232fbd7d check: add "payees" check requiring payee declarations 2020-12-30 18:08:30 -08:00
Stephen Morgan
e406cebcce lib: Remove unused AtDefault constructor for ValuationType. 2020-12-19 11:59:44 -08:00
Stephen Morgan
cdec0f9382 lib: Remove special handling of now-inaccessible AtDefault valuation constructor.
This simplifies all the *ApplyValuation functions, as they no longer
need mreportdate or multiperiod arguments.
2020-12-19 11:59:44 -08:00
Stephen Morgan
439eddc8b9 lib: Include price directives when calculating journal end date. 2020-12-16 19:04:33 -08:00
Stephen Morgan
92645cca14 lib: Sorting accounts by amount should also consider account name after amounts. 2020-12-14 06:48:47 -08:00
Simon Michael
e33667f580 check: support quoted check name + args
A "hledger check" argument may now be a quoted string containing
the check name followed by space-separated arguments, for
checks which make use of those. This means the check command
can replicate "check-dates --unique" and (in principle)
"check-fancyassertions ASSERTIONS..". Eg:

hledger check "dates --unique"

I think it'll be better for checks to take no arguments or options,
so this is probably just a transitional feature for compatibility.
2020-11-29 20:48:31 -08:00
Simon Michael
8c23a16bf8 journal: strict mode also requires commodity declarations 2020-11-27 17:54:55 -08:00
Simon Michael
ea0d4901ab lib: journalApplyAliases performance tweak 2020-11-25 21:06:31 -08:00
Simon Michael
0dc8eca68a lib: remove partial code from journalApplyAliases etc. 2020-11-25 21:06:31 -08:00
Simon Michael
94b3f090be csv, timedot, timeclock: respect --alias options (fix #859)
Command-line account aliases now also affect transactions read
from these formats (not just journal format).

lib: journalApplyAliases, transactionApplyAliases, postingApplyAliases
helpers have been added.
2020-11-24 09:17:01 -08:00
Simon Michael
221a6d9001 lib: new more robust price lookup implementation, fixing #1402 2020-11-23 18:08:41 -08:00
Simon Michael
d60c1f764d lib: partial valuation fix, more debug output for #1402 2020-11-22 13:10:31 -08:00
Simon Michael
654ab4243a ; comments 2020-11-20 16:45:40 -08:00
Simon Michael
4a74680a22 ;lib: rename some budget report helpers
The new names are of course verbose, but more consistent and revealing.
2020-11-19 14:59:08 -08:00
Simon Michael
0ce9c352ec add: don't let number style disturb the default (Fixes #1378) 2020-11-14 11:08:01 -08:00
Simon Michael
1f94aa1628 ;lib: make file modification/reloading helpers more robust (#1390)
Work on hledger-web tests showed some bad behaviour, in particular
journalReloadIfNewer would always reload a journal read from a string
or stdout. This is now fixed, and an ugly read.show conversion has
been cleaned up.

Hledger.Cli.Utils API changes:

removed:
- journalSpecifiedFileIsNewer
- fileModificationTime

added:
- utcTimeToClockTime

changed:
- journalFileIsNewer now requires a file argument
2020-11-13 16:28:35 -08:00
Simon Michael
70f95e9f2c ;doc: journal: clarify that it's the first-seen digit group style 2020-11-10 09:29:04 -08:00
Simon Michael
f43c7e41d1 csv, journal: debug output
The prolific "assignment" output is now at level 9.
2020-11-10 08:39:32 -08:00
Simon Michael
a97daaf322 lib: replace pretty-show with pretty-simple
pretty-simple, already used in .ghci, will hopefully give nicer debug
output, including for values which don't have Read-able Show output.
This should mean that we can start removing custom string-like Show
instances that were a workaround for pretty-show.

We are using the latest version (4.0.0.0) to get compact output.
Here's some old pretty-show output:

 CsvRules
   { rdirectives = [ ( "skip" , "1" ) ]
   , rcsvfieldindexes = [ ( "date" , 1 ) , ( "amount" , 2 ) ]
   , rassignments = [ ( "amount" , "%2" ) , ( "date" , "%1" ) ]
   , rconditionalblocks = []
   }

And the new pretty-simple output:

 CsvRules
   { rdirectives=
     [ ( "skip", "1" ) ]
   , rcsvfieldindexes=
     [ ( "date", 1 ), ( "amount", 2 ) ]
   , rassignments=
     [ ( "amount", "%2" ), ( "date", "%1" ) ]
   , rconditionalblocks= []
   }

Non-compact pretty-simple output would be:

 CsvRules
     { rdirectives=
         [
             ( "skip"
             , "1B"
             )
         ]
     , rcsvfieldindexes=
         [
             ( "date"
             , 1
             )
         ,
             ( "amount"
             , 2
             )
         ]
     , rassignments=
         [
             ( "amount"
             , "%2"
             )
         ,
             ( "date"
             , "%1"
             )
         ]
     , rconditionalblocks=[]
     }

Also:

- Account's Show instance no longer converts : to _ in account names

- drop unused pretty-show dependency from hledger, hledger-ui packages

- regenerate hledger-lib with the older hpack that's shipped in stack
2020-11-10 08:06:11 -08:00
Simon Michael
66af4962a1 ;lib: drop a todo, seems ok 2020-11-09 17:14:38 -08:00
Simon Michael
bfb5c6ee2a lib: global commodity display styles can be set in InputOpts or Journal, overriding all others 2020-11-09 17:14:37 -08:00
Simon Michael
0eddbe7a4b ;lib: reader cleanups 2020-11-09 17:14:37 -08:00
Simon Michael
524e23bc37 csv: decimal-mark rule to help with number parsing
Journal keeps a new piece of parsing state, a decimal mark character,
which can optionally be set to force the number format expected by all
amount parsers.
2020-11-08 16:39:03 -08:00
Stephen Morgan
3abc9f5985 lib: When calculating elisions, don't skip over some amounts. 2020-11-04 14:25:21 +11:00
Stephen Morgan
dcb884c5ff lib: Align postings to unnormalised amount widths. 2020-11-04 14:25:20 +11:00
Stephen Morgan
7e47c11fda lib: Remove some uses of pad(Left|Right)Wide to ensure we're using the
width provided by showMixed.

Also refactor renderTable to be a bit clearer, and to avoid duplicate
calculations.
2020-11-04 14:25:20 +11:00
Stephen Morgan
b39de5989f lib: Refactor and improve comments for new mixed amount display functions. 2020-11-04 14:25:20 +11:00
Stephen Morgan
9de238757b lib,cli,ui: Introduce showMixed*Unnormalised, eliminate most direct calls of strWidth.
This introduces some new helper functions which are exactly the same
as what we had before, but do not call
normaliseMixedAmountSquashPricesForDisplay, so that we can use the new
functions for displaying Transaction and Posting. It also goes through
and gets rid of most uses of the old showMixed* functions which would
benefit from using the new interface.
2020-11-04 14:25:20 +11:00
Stephen Morgan
162a936360 lib: BudgetReport uses new renderTable inteface, now has more compact output.
This changes showMixedAmountElided so that the width to elide to is
given as an argument, rather than fixed at 22 characters. This
actually uses the new renderTable interface. Mostly this is just an
internal change, but since we have more information about the widths
of things, we can actually get rid of some superfluous spaces in the
budget report output, previously there to make sure it stayed aligned
with the largest reasonable contents.
2020-11-04 14:25:20 +11:00
Stephen Morgan
57d7b223a2 lib: Write new showMixedAmount* functions which keep track of length.
Previously showMixedAmountElided would show two amounts and then the
elision string if necessary. Now it will display as many Amounts as it
can subject to the condition that the amounts plus the elision string
fit within 22 characters.
2020-11-04 14:25:20 +11:00
Stephen Morgan
64e3e5c54d lib: Improve documentation for StringFormat, remove type alias for Valuation. 2020-10-23 14:14:11 -07:00
Stephen Morgan
103308e795 lib,cli: Store StringFormat in ReportOpts, rather than unparsed String.
StringFormat now also takes an optional overline width, which is
currently only used by defaultBalanceLineFormat.
2020-10-23 14:14:11 -07:00
Stephen Morgan
5a6d38fdf2 lib: For ymd date parsing, don't consume invalid date components. 2020-10-18 21:28:46 -07:00
Simon Michael
08ee311875 journal: fix error when P directive has a zero price (#1373) 2020-10-18 20:48:14 -07:00
Simon Michael
c5513ba722 ;tweak a few parse errors for clarity, consistency (#1329) 2020-09-19 08:10:39 -07:00
Simon Michael
a9fbaaf284 ;partial comment cleanups 2020-09-03 09:52:00 -07:00
Simon Michael
c2929939e4 make account type autodetection (& hledger-smooth) case insensitive again (#1341)
lib: added case-insensitive variants of the accountNameToRegex functions.
2020-09-03 09:52:00 -07:00
Stephen Morgan
07dd30c1e5 lib,cli,ui: Change to consistent naming scheme for Hledger.Utils.Regex. 2020-09-01 11:36:34 +10:00
Stephen Morgan
8dfffb1e61 lib,web: Replace regex functions with simple text replacement. 2020-08-31 20:45:28 +10:00
Stephen Morgan
2cd7877c46 lib: Remove unnecessary NFData instances. 2020-08-31 15:41:36 +10:00
Stephen Morgan
af31d6e140 lib,cli,ui: Remove redundant Typeable and Data instances.
Also add some explicit import lists.
2020-08-31 15:41:30 +10:00
Stephen Morgan
e5371d5a6a lib,cli,ui,web: Make Regexp a wrapper for Regex. 2020-08-31 12:04:45 +10:00
Stephen Morgan
7182fa4746 lib: Improve documentation for amountRoundedQuantity, fix a typo. 2020-08-30 23:01:04 +10:00
Stephen Morgan
f6fa76bba7 lib,cli: Get rid of magic values for asprecision, use a sum type instead. 2020-08-30 23:00:35 +10:00
Stephen Morgan
ba59fed6b2 lib: Replace more instances of fromIntegral with safer versions. 2020-08-30 22:20:58 +10:00
Stephen Morgan
ca2e55c954 lib: Replace some fromIntegral with toInteger. 2020-08-30 22:20:58 +10:00