Commit Graph

834 Commits

Author SHA1 Message Date
Simon Michael
de330bdfce imp: journal: check for adjacent conversion postings only in strict mode
This avoids breaking existing journals if, say, they happen to have
a single "equity:trading" posting somewhere.
2023-01-25 19:00:29 -10:00
Simon Michael
76f8eaf91a imp: journal: ignore redundant costs; --infer-costs more robustly
You can now write both @/@@ costs and corresponding equity conversion postings
in a transaction at any time, not just when using --infer-costs or --infer-equity.
hledger will recognise the redundancy and ignore it.

One thing has become more strict: hledger now requires conversion postings
to occur in adjacent pairs; an odd number of them is not allowed.
(Conversion postings are postings to accounts of type `V`/`Conversion`,
or named `equity:conversion`, `equity:trade`, `equity:trading`,
or subaccounts of these.)

And, --infer-costs now works in transactions with an implicit amount
(inferring costs from equity now happens after transaction balancing,
not before).
2023-01-25 19:00:29 -10:00
Chris Lemaire
43c9f018dc journal: Account directives reject account names with brackets
Previously, the accounts passed to account directives would be stripped
of their surrounding brackets, but the required behaviour is to have
account directives plain reject bracketed accounts. This change ensures
that accounts in account directives may not start with a bracket
character.
2023-01-24 13:40:01 -10:00
Simon Michael
9497472e64
Merge branch 'master' into 1984-source-name-and-position-for-forecast 2023-01-23 16:04:00 -10:00
Chris Lemaire
4ada2a3fbc journal: Fully unbracket AccountNames in account directives
Currently an account name like "a:(aa)" will not have (aa) unbracketed.
However, this seems reasonable since the full name is unbracketed and
thus will not be confused with virtual or virtual-balanced posting.
2023-01-23 12:43:13 -10:00
Chris Lemaire
9443fe0e0d journal: json: Add source positions to forecast transactions 2023-01-21 22:30:30 +01:00
Simon Michael
5b27554f0a dev: note why inferring costs from equity before balancing 2023-01-19 20:57:02 -10:00
Simon Michael
cf469c080d lib: journalInferCosts -> journalInferCostsFromEquity 2023-01-19 20:25:41 -10:00
Simon Michael
28eb8be4fa lib: BalancingOpts{infer_transaction_prices_ -> infer_balancing_costs_} 2023-01-19 20:21:31 -10:00
Simon Michael
dfea6eaee2 lib: Hledger.Data.Journal: journalAddPricesFromEquity -> journalInferCosts 2023-01-19 09:17:56 -10:00
Simon Michael
dfebf3174c imp: csv: check assigned account names are valid (parseable) (#1978) 2023-01-11 21:42:47 -10:00
Simon Michael
a9b63bb694 fix: csv: skip header lines before attempting to parse records (#1967) 2022-12-27 12:21:20 -10:00
Simon Michael
4b36d852b4 imp: journal: ignore ((valuation expressions)) after amounts (ledger compat) 2022-12-22 20:06:28 -10:00
Simon Michael
1ea2bcc83f imp: journal: also parse (lot notes) in amounts (ledger compat)
and rename lotpricep -> lotcostp
and instrument some amount parsers for debugging with megaparsec's dbg
2022-12-22 19:07:40 -10:00
Simon Michael
aa54c3273a dev: lib: clarify some amount parsers; describe Ledger lot notation
amountpwithmultiplier -> amountp'
amountpnolotpricesp   -> amountnobasisp
amountwithoutpricep   -> simpleamountp
priceamountp          -> costp
2022-12-22 09:15:43 -10:00
Simon Michael
b1e08ffe46 imp: journal: python directive also consumes empty lines (ledger compat) 2022-12-22 02:30:29 -10:00
Simon Michael
85e9fbcf7d imp: journal: allow an empty double-quoted commodity symbol
"" is equivalent to writing no symbol.
But it's now possible to declare market prices for the no-symbol
commodity, which occurs eg with timedot data.

P 2022-01-01 "" $100
2022-12-22 01:53:06 -10:00
Simon Michael
a070875fe6 imp: journal: ignore end apply year, end apply tag, drop "pop" (ledger compat) 2022-12-22 01:50:27 -10:00
Simon Michael
a5d6b42322 imp: journal: ignore end apply fixed, end apply tag directives (ledger compat) 2022-12-22 01:12:56 -10:00
Simon Michael
af5645ebb7 imp: journal: ignore python, eval directives (ledger compat) 2022-12-22 00:50:43 -10:00
Simon Michael
2e77586d1b imp: journal: ignore value directives (ledger compat) 2022-12-22 00:38:51 -10:00
Simon Michael
fb7134c9a0 imp: journal: ignore tag subdirectives (ledger compat) 2022-12-22 00:34:33 -10:00
Simon Michael
c58b9e6ac9 imp: journal: accept apply year as another synonym for Y (ledger compat) 2022-12-22 00:27:35 -10:00
Simon Michael
2357a7a750 dev: journal: ignore CLI flags in journal 2022-12-21 23:40:43 -10:00
Simon Michael
abd104ccee imp: journal: ignore more ledger directives (ledger compat)
apply fixed, apply tag, assert, bucket, A, capture, check, define, expr

And reorder the directivep parsers.
2022-12-21 23:04:58 -10:00
Simon Michael
e09ab0936a imp: journal: accept @ as a prefix of directives (ledger compat) 2022-12-21 22:56:32 -10:00
Simon Michael
95d7679f6d imp: journal: accept year as a synonym for Y (ledger compat) 2022-12-21 22:27:08 -10:00
Simon Michael
826a7cfb2a imp: journal: ignore payee subdirectives 2022-12-21 22:09:15 -10:00
Simon Michael
bf6a2a7ccf imp: journal: ignore commodity subdirectives other than format 2022-12-21 22:08:54 -10:00
Simon Michael
ace185f7d2 ;doc: update old manual links 2022-12-10 18:56:47 -10:00
Simon Michael
b50d60cfea ;doc: csv, timeclock, timedot: clarify comment lines (#1953) 2022-12-06 10:38:50 -10:00
Simon Michael
78771e900b dev: lib: support debug logging to file in a few more places for hledger-ui 2022-11-06 09:01:12 -10:00
Simon Michael
988c164ec8 imp: debug logging improvements; hledger-ui logs to hledger-ui.log only
Hledger.Utils.Debug's "trace or log" functions are now controlled as
follows: to enable logging, append ",logging" to the program name at
startup (using withProgName). This also works when running in GHCI.
And they log to PROGNAME.log, not debug.log.

All (hopefully) debug logging in the hledger packages is now "trace or
log" capable.

This means that hledger-ui should now log all debug output to
./hledger-ui.log, with none of it appearing on the console.
2022-11-03 16:07:54 -10:00
Simon Michael
7fd25809e8 dev: fix customErrorBundlePretty import warnings 2022-10-07 07:43:28 -10:00
Simon Michael
de5a97600c imp: lib: Hledger.Read.Common: rename/add amount parsing helpers
removed:
 amountp'
 mamountp'

added:
 parseamount
 parseamount'
 parsemixedamount
 parsemixedamount'
2022-10-07 06:34:34 -10:00
Simon Michael
01387548e7 feat: csv: intra-day-reversed compensates when days' txns are reversed
As in eg vanguard CSV.
2022-10-06 22:21:55 -10:00
Simon Michael
15b2e7d586 fix: csv: ignore extra whitespace in account rule when detecting virtual postings
Reported by CruxOfTheB in chat.
2022-10-03 07:50:23 -10:00
Simon Michael
3b24d9465b imp: csv: new timezone rule; convert zoned date-times to local dates (#1936)
Previously, CSV date-times with a different time zone from yours
(with or without explicit timezones in the CSV) could give off-by-one
dates, because the CSV timezone was ignored.

Now,

1. you can use the `timezone` rule to indicate which other
   timezone a CSV is implicitly using

2. CSV date-times with a timezone - whether declared by rule or
   parsed with %Z - are localised to the system time zone
   (or another set with the TZ environment variable).
2022-10-01 14:50:35 -10:00
Simon Michael
bfb632dd8e dev: lib: Hledger.Data.Amount: drop amount, use nullamt always 2022-08-23 12:16:15 +01:00
Simon Michael
c80c72d7cd dev: lib, cli, bin: enable/fix name shadowing warnings
And a few other cleanups.
2022-08-23 12:16:15 +01:00
Simon Michael
b32b130045 dev: lib: fix a failing doctest 2022-08-18 05:08:10 +01:00
Simon Michael
3003a18c94 dev: lib: hlint improvements 2022-08-16 09:13:30 +01:00
Simon Michael
a3096d0549 dev: journalFilePath, include: show (unknown) instead of nothing
Probably not user-visible, but might improve debug logging slightly.
2022-08-16 09:13:30 +01:00
Simon Michael
1f08a8a94e fix: fix multi-file account display order; improve file read logging (#1909) 2022-08-16 09:13:30 +01:00
Simon Michael
07e3dca735 imp: at --debug 5, log account declarations info while parsing (#1909) 2022-08-16 09:13:30 +01:00
Simon Michael
3d6e363461 lib: save account directive positions, for troubleshooting (#1909) 2022-08-16 09:13:30 +01:00
Simon Michael
b46aa1330f fix: don't reverse account display order with include files (#1909) 2022-08-05 13:44:56 +01:00
Simon Michael
09d66c5fc1 fix: account display order is now tracked across all files (fix #1909) 2022-08-04 19:05:52 +01:00
Simon Michael
f135001b87 ;dev: improve followingcommentp' doc 2022-07-28 09:14:13 +01:00
Simon Michael
147856e3bb imp: errors: timeclock, csv error improvements 2022-07-23 02:35:52 +01:00
Stephen Morgan
c54fb4d514 imp: cost: Allow matching equity conversion equity postings to
transaction prices.

When given --infer-costs, hledger will now separately infer transaction
prices for different prices. Given a pair of adjacent conversion
postings, hledger will check if there is a single posting with a
transaction price which matches both the amounts. If so, it associates
those conversion postings to that priced post.

If it can't find any transaction price postings which match, it will
find the first non-transaction price post which matches one of the two
amounts, and will add a transaction price to that, and associate them.
2022-07-15 12:32:38 +01:00
Stephen Morgan
07d1b01287 imp: cost: Allow inferring cost from conversion equity postings with --infer-costs.
When given --infer-equity, hledger will change transaction prices to balancing equity postings. This introduces the inverse operation, --infer-costs, which will match balancing equity postings and transform them into a transaction price, allowing --cost to work properly with them. This is only a partial inverse as it needs to use some heuristics to match the postings which will not work in complicated cases.

Specifically, when hledger finds exactly two conversion postings in a transaction (by default, subaccounts of equity:conversion or equity:trad(e|ing)), it will find the first posting in the transaction whose amount is negative one of the conversion posting amounts, and inserts the corresponding transaction price.
2022-07-15 12:32:38 +01:00
Simon Michael
7d8fbe2dc0 dev: errors: update doctests 2022-07-13 19:00:00 +01:00
Simon Michael
1c67d0860e imp: errors: more error prettification 2022-07-13 03:59:44 +01:00
Simon Michael
dd14f9c822 imp: lib: Hledger.Utils.Parse: export HledgerParseErrors 2022-06-29 09:58:56 +01:00
Stephen Morgan
9155d679fe fix!: Revert "fix!: utf-8: Use with-utf8 to ensure all files are read and written with utf8 encoding. (#1619)"
This reverts commit e233f001c5.

This would break at least some people's workflow. A lighter touch is
probably sufficient.
2022-06-01 09:35:18 +10:00
Simon Michael
a9779b2377 ref: move journal checking/pretty errors down further, to Hledger.Data
now at Hledger.Data.JournalChecks*, Hledger.Data.Errors
2022-05-21 18:29:13 -10:00
Simon Michael
307f723b0a lib: simpler, more consistent names for check functions
API changes:
journalCheckAccountsDeclared
journalCheckCommoditiesDeclared
journalCheckPayeesDeclared
->
journalCheckAccounts
journalCheckCommodities
journalCheckPayees
2022-05-21 18:29:13 -10:00
Simon Michael
65e913b7c5
Merge pull request #1834 from Xitian9/utf8
Use with-utf8 and don't use Data.Text.IO.
2022-05-21 17:50:59 -10:00
Stephen Morgan
e233f001c5 fix!: utf-8: Use with-utf8 to ensure all files are read and written with utf8 encoding. (#1619)
May also fix #1154, #1033, #708, #536, #73: testing is needed.

This aims to solve all problems where misconfigured locales lead to
parsers failing on utf8-encoded data. This should hopefully avoid
encoding issues, but since it fundamentally alters how encoding is dealt
with it may lead to unexpected outcomes. Widespread testing on a number
of different platforms would be useful.
2022-05-22 13:12:19 +10:00
Stephen Morgan
15a5d5d38b
Merge pull request #1814 from Xitian9/csverror
imp: csv: Give an error if unable to substitute csv templates. (#1803)
2022-05-22 11:35:39 +10:00
Simon Michael
db26456e1c imp: consolidate ordereddates check under Hledger.Read.Checks too
And remove the last vestiges of older more complex behaviour.
ordereddates now always checks all transactions in each file,
unaffected by a query. (But still affected by --date2).
2022-05-09 00:05:39 -10:00
Simon Michael
810a868c88 ref: consolidate most checks under Hledger.Read.Checks
and error message helpers in Hledger.Read.Error.
2022-05-09 00:04:33 -10:00
Simon Michael
a00dc04e0d imp: check: uniqueleafnames: use the standard error format (#1436)
Hledger.Read.Common:
export makePostingErrorExcerpt
2022-04-25 02:56:59 -10:00
Simon Michael
7e45ab338f imp: check: show column ranges properly (#1436) 2022-04-25 02:56:59 -10:00
Simon Michael
66d0beea5e imp: check: ordereddates: use the standard error format (#1436)
Hledger.Read.Common:
export makeTransactionErrorExcerpt
2022-04-25 02:56:59 -10:00
Simon Michael
792434ca7d imp: check: payees: use the standard error format (#1436) 2022-04-25 02:56:59 -10:00
Simon Michael
c7e8f58c33 imp: check: mark all error columns when that's preferable (#1436)
Undeclared commodity errors now mark the whole amount and assertion
region, since locating the exact position of commodity symbols is
difficult.
2022-04-25 02:56:59 -10:00
Simon Michael
8f4405e628 dev: check: refactor error messages (#1436) 2022-04-25 02:56:59 -10:00
Simon Michael
6ec0e05c42 imp: check: commodities: use the standard error format (#1436) 2022-04-25 02:56:59 -10:00
Simon Michael
4b3644d780 imp: check: accounts: print a standardised error message like megaparsec's (#1436)
added:
formatExcerptLikeMegaparsec
2022-04-25 02:56:59 -10:00
Simon Michael
5ab7b9f643 ref: HledgerParseErrors type alias 2022-04-25 02:56:59 -10:00
Simon Michael
2f28e1b0a7 ref: rename CustomErr -> HledgerParseErrorData
Verbose, but use every chance to clarify the complicated parse error
situation.
2022-04-25 02:56:59 -10:00
Simon Michael
53332ee6a5 stack: re-enable hledger-web with ghc 9.2 2022-04-15 15:07:17 -10:00
Stephen Morgan
c48d98c515 imp: csv: Substitute empty string if csv template fails. (#1803) 2022-03-29 18:03:33 +11:00
Stephen Morgan
8968733630 cln!: Clean up journal parsing.
parseAndFinaliseJournal' has been removed. In the unlikely event you
needed it in your code, you can replace it with:
parseAndFinaliseJournal' parser iopts fp t =>
initialiseAndParseJournal parser iopts fp t
>>= liftEither . journalApplyAliases (aliasesFromOpts iopts)
>>= journalFinalise iopts fp t

Some parsers have been generalised from JournalParser to TextParser.
2022-03-26 15:35:19 -10:00
Simon Michael
a2f6b30fbd cln: silence some warnings/lints 2022-03-26 08:34:21 -10:00
Stephen Morgan
603b2e9f09 ref: Use ExceptT String IO a instead of IO (Either String a).
This increases composability and avoids some ugly case handling. We
re-export runExceptT in Hledger.Read.

The final return types of the following functions has been changed from
IO (Either String a) to ExceptT String IO a. If this causes a problem,
you can get the old behaviour by calling runExceptT on the output:
readJournal, readJournalFiles, readJournalFile

Or, you can use the easy functions readJournal', readJournalFiles', and
readJournalFile', which assume default options and return in the IO
monad.
2022-03-25 14:23:27 -10:00
Simon Michael
b584774a03 ci: fix a typo I fat-fingered 2022-03-18 11:03:30 -10:00
Simon Michael
2b2d643333 test: errors: make tests runnable 2022-03-18 06:59:33 -10:00
Stephen Morgan
e6bf04fce3 fix: aliases: Make sure we can escape forward slashes in account
aliases, but otherwise the regular expression handler handles escapes.
2022-03-16 19:01:03 -10:00
Stephen Morgan
a65e64115a fix: alias: Allow escaped characters in regular expression aliases. (#982) 2022-03-10 17:00:03 -10:00
Stephen Morgan
19ed6d3f00 cln!: sourcepos: Use megaparsec-supplied sourcePosPretty.
Change showSourcePos to sourcePosPretty, and showSourcePosPair to
sourcePosPairPretty.
2022-03-10 16:56:28 -10:00
Stephen Morgan
2f47ae05c6 fix: csv: Allow unicode in field references for csv. (#1809) 2022-02-06 14:16:17 -10:00
Stephen Morgan
73925ae965 fix: types: Ensure auto postings can match against and be matched by type: queries.
This requires checking parent accounts for any new accounts introduced by auto postings which do not exist in the original journal.

Also refactor journalFinalise to only call journalPostingsAddAccountTags once, and use fewer intermediate variables.
2022-02-03 15:32:00 -10:00
Stephen Morgan
a16c88b1b1 fix: Make sure cash and conversion account types are correctly
determined when there are no explicit declarations.
2022-02-03 15:32:00 -10:00
Simon Michael
45127dc5f5 feat: new type: query for easy matching by account type
type:TYPES, where TYPES is any of the (case insensitive) letters
ALERXCV, matches accounts by their declared or inferred type.
(See https://hledger.org/hledger.html#account-types.)
This should work with most commands, eg:

    hledger bal type:al
    hledger reg type:x

API changes:

Journal has a new jaccounttypes map.
The journalAccountType lookup function makes it easy to check an account's type.
The journalTags and journalInheritedTags functions look up an account's tags.
Functions like journalFilterPostings and journalFilterTransactions,
and new matching functions matchesAccountExtra, matchesPostingExtra
and matchesTransactionExtra, use these to allow more powerful matching
that is aware of account types and tags.
2022-01-31 16:44:13 -10:00
Simon Michael
885a4fcfec ref: journalFinalise: clarify sequence, top to bottom 2022-01-31 16:44:13 -10:00
Simon Michael
56be63e6f1 feat: queries can now match account tags (#1817)
Accounts, postings, and transactions can now all be filtered by the
tags in an account's declaration. In particular it's now possible to
more reliably select accounts by type, using their type: tag rather
than their name:

    account myasset       ; type:Asset
    account myliability   ; type:Liability

    $ hledger accounts tag:type=^a
    myasset

Accounts inherit tags from their parents.

API changes:
A finalised Journal has a new jdeclaredaccounttags field
for easy lookup of account tags.
Query.matchesTaggedAccount is a tag-aware version of matchesAccount.
2022-01-30 09:47:52 -10:00
Simon Michael
4ed80d4069 ;doc: update journalFinalise haddock 2022-01-30 09:47:52 -10:00
Stephen Morgan
df4949ad1a imp!: journal: Remove deprecated account type code syntax from account
directives.

Previously, you declare an account type with the following format:
account assets  A
This has been deprecated since 1.13, and should now be declared with one
of:
account assets  ; type:A
account assets  ; type:asset
2022-01-28 12:38:38 -10:00
Stephen Morgan
55a022a941 fix: strict: Make sure forecast transactions and auto postings are
checked with --strict. (#1810)
2022-01-26 07:57:13 -10:00
Stephen Morgan
57d055b643 fix: cost: Add inferred equity postings during journal finalisation, so
they can be matched by queries.
2022-01-16 18:37:18 -10:00
Stephen Morgan
5006e26995 imp: check: Do not check commodities for zero amounts with empty
commodity. (#1767)
2022-01-16 08:11:02 -10:00
Stephen Morgan
8eedbbbe87 imp: cost: Generate totally balanced conversion postings for amounts with costs.
Introduce --infer-equity option which will generate conversion postings.
--cost will override --infer-equity.

This means there will no longer be unbalanced transactions, but will be
offsetting conversion postings to balance things out. For example.

2000-01-01
  a   1 AAA @@ 2 BBB
  b  -2 BBB

When converting to cost, this is treated the same as before.
When used with --infer-equity, this is now treated as:

2000-01-01
  a                               1 AAA
  equity:conversion:AAA-BBB:AAA  -1 AAA
  equity:conversion:AAA-BBB:BBB   2 BBB
  b                              -2 BBB

There is a new account type, Conversion/V, which is a subtype of Equity/E.
The first account declared with this type, if any, is used as the base account
for inferred equity postings in conversion transactions, overriding the default
"equity:conversion".

API changes:

Costing has been changed to ConversionOp with three options:
NoConversionOp, ToCost, and InferEquity.
The first correspond to the previous NoCost and Cost options, while the
third corresponds to the --infer-equity flag.  This converts transactions with costs
(one or more transaction prices) to transactions with equity:conversion postings.
It is in ConversionOp because converting to cost with -B/--cost and inferring conversion
equity postings with --infer-equity are mutually exclusive.

Correspondingly, the cost_ record of ReportOpts has been changed to
conversionop_.

This also removes show_costs_ option in ReportOpts, as its functionality
has been replaced by the richer cost_ option.
2021-12-21 10:50:13 -10:00
Simon Michael
0d83bdf6d7 cln: csv: small rename 2021-12-08 16:57:53 -10:00
Simon Michael
1fee70b51f cln: hlint cleanups 2021-11-21 19:33:21 -10:00
Lawrence
fb817ffa55 imp: journal: support decimal-mark directive in journal files
To match csv-rule and override amount-style as described in
https://github.com/simonmichael/hledger/issues/1670
2021-11-18 22:17:07 -10:00
Stephen Morgan
e35d0b7865 fix: csv: Successfully parse empty csv file. (#1183) 2021-11-18 20:50:02 -10:00
Stephen Morgan
87a7a586d4 fix: csv: Handle multiple zero amounts in postings in csv files. (#1733) 2021-11-18 20:48:55 -10:00