This enables a "relaxed" workflow where you delay balance assertions
checking until strict mode is turned on: always run hledger -I, and
add -s when you're ready.
When the error message repeated the invalid date at the end,
it was possible to misinterpret that as a suggested fix (reported in chat).
Instead, date errors (most of them) now rely on the highlighted data
excerpt above. This is also preferable since it shows the original
date as written, not a reconstruction with a possibly different format.
Should this be the policy for all error messages going forward ?
It would be easier.
Can we assume the data excerpt is always visible along with the error message ?
It isn't shown by flycheck-hledger in emacs, eg.
Adjust getEffectiveAssignment to compute an intermediary form of the
active assignments (with an additional Either wrapper to distinguish
top-level and conditional assignments) and move the remaining work to
its only caller, hledgerField.
Rework hledgerFieldValue. Instead of calling hledgerField, call
getEffectiveAssignment and--in the conditional block case--construct
a CsvRules scoped just to the active ConditionalBlock before calling
renderTemplate.
Adjust regexMatchValue to use rconditionalblocks to access conditional
blocks from the CsvRules, rather than rblocksassigning, since we haven't
narrowed the scope of that field.
The result is match group references are only expanded for match groups
that occur within the in-scope ConditionalBlock. Fixes: #2158.
Signed-off-by: Jonathan Dowland <jon@dow.land>
hledgerField is an alias to the function getEffectiveAssignment: both
names are used in various parts of RulesReader.
Treat hledgerField as the canonical name, and getEffectiveAssignment
as an implementation detail of hledgerField.
Replace all uses of getEffectiveAssignment with hledgerField (except the
one in hledgerField.)
Signed-off-by: Jonathan Dowland <jon@dow.land>
Note the headErr/tailErr calls will print stack traces if they fail
(small ones: five lines, one of which is the useful location info),
which may or may not be best UX.
Based on feedback in chat, I added support for several more kinds of
Unicode space character for separating digit groups, both when reading
and when displaying numbers. These are the spaces currently supported,
which are just my best guess at the ones that might show up in CSV files
now and then:
space,
no-break space,
en space,
em space,
punctuation space,
thin space,
narrow no-break space,
medium mathematical space
When reading a symbolically-linked journal file,
relative paths in include directives are now evaluated
relative to the directory of the real linked file,
not the directory of the symlink.
This also seems to fix an obscure case where stats did not report
absolute included file paths in certain circumstances (stdin, maybe no
terminal..)
When processing costs and equity postings in transactions during
journal finalisation, we now pass just the conversion account name(s)
rather than the entire map of account types. This slowdown was severe
for some users/data/machines.
In field assignment values we now parse %FIELD references, \MATCHGROUP references
and "\n" newline markers more carefully, so all can coexist.
Parsing these values might be slower than before, but hopefully not noticeably so.
Strict checks now run only once, at end of the high level read operation,
and not for each individual file; this fixes some spurious --strict failures,
like account declarations not affecting a sibling file as they should.
And .latest file writing now happens as the last step, after passing
strict checks. This is mainly for the import command, but it also
means that hledger print --new now does not update .latest files
if strict checks are failing.
The file reading API has been improved and documented in more detail.
This and the preceding commits were "work in progress" that got out of control.
There's more to do, but this one brings these precision-related improvements
(at least):
When "infinite decimals" arise, they are now generally shown with
8 decimal digits rather than 255.
print and prices no longer add trailing decimal zeros unnecessarily.
Some code has been refactored or given more debug output.
All tests have been updated to match the recent changes.
Replace occurrences of '\N' (where N is a positive number) in field
templates with the corresponding regular expression match group, if it
exists.
E.g. Warp the date to the first of the month for the second posting
if %date (....-..)-..
comment2 date:\1-01
E.g. Strip a prefix from an imported account name
if %account1 liabilities:jon:(.*)
account1 \1
Fixes#2009.
Signed-off-by: Jonathan Dowland <jon@dow.land>
All commands that suport csv output now also support tsv output. The
data is identical, but the fields are separated by tab characters and
there is no quoting or escaping. Tab, carriage return, and newline
characters in data are converted to spaces (this should rarely if ever
happen in practice).
Changes to enable more control of "rounding" behaviour
(ie, choosing display precisions for amounts).
This reverts 1.31's change of asprecision, making it a non-Maybe
again, and adds a new asrounding field providing more control over how
a target display precision is applied to existing amounts (two options
for now, more later). Functionality is in an interim state (reports do
no rounding).
A quantity with a unit like "15m" was being parsed internally as
0.249999999... rather than 0.25 (and since hledger 1.21, printed that
way also). Now we round such quantities to two places during parsing,
to get exact quarter-hour amounts.
This simplifies the code for styling amounts with or without precision.
But it complicates the semantics (Nothing is useful only when setting style).
Not sure if it's the best way.
CSV rules files can now be read directly, eg you have the option of
writing `hledger -f foo.csv.rules CMD`. By default this will read data
from foo.csv in the same directory. But you can also specify a
different data file with a new `source FILE` rule. This has some
convenience features:
- If the data file does not exist, it is treated as empty, not an
error.
- If FILE is a relative path, it is relative to the rules file's
directory. If it is just a file name with no path, it is relative
to ~/Downloads/.
- If FILE is a glob pattern, the most recently modified matched file
is used.
This helps remove some of the busywork of managing CSV downloads.
Most of your financial institutions's default CSV filenames are
different and can be recognised by a glob pattern. So you can put a
rule like `source Checking1*.csv` in foo-checking.csv.rules,
periodically download CSV from Foo's website accepting your browser's
defaults, and then run `hledger import checking.csv.rules` to import
any new transactions. The next time, if you have done no cleanup, your
browser will probably save it as something like Checking1-2.csv, and
hledger will still see that because of the * wild card. You can choose
whether to delete CSVs after import, or keep them for a while as
temporary backups, or archive them somewhere.
Inner empty lines were not being skipped automatically, contrary to
docs. Now all empty lines are skipped automatically, and the `skip`
rule is needed only for non-empty lines, as intended.
This may be a breaking change: it's possible that the `skip` count
might need to be adjusted in some CSV rules files.
Breaking change: previously timeclock descriptions could contain
semicolons. Now a semicolon in the description will end it and
start a comment (which may contain tags).
I found at least one user for whom this would be a breaking change
(they generate forecast txns, and have auto posting rules, but don't
want the latter applied to the former). I guess it's better to keep
things as they were for now: if you need auto postings on your
forecast txns you must use two flags, --forecast --auto.
Boolean queries are now prefixed with an 'expr:' prefix, making them
completely separable from old queries and making the addition of them a
little more migration proof.
The tests are updated accordingly, changes made to the tests previously
are removed and extra cautious documentation is also removed.
You can now write both @/@@ costs and corresponding equity conversion postings
in a transaction at any time, not just when using --infer-costs or --infer-equity.
hledger will recognise the redundancy and ignore it.
One thing has become more strict: hledger now requires conversion postings
to occur in adjacent pairs; an odd number of them is not allowed.
(Conversion postings are postings to accounts of type `V`/`Conversion`,
or named `equity:conversion`, `equity:trade`, `equity:trading`,
or subaccounts of these.)
And, --infer-costs now works in transactions with an implicit amount
(inferring costs from equity now happens after transaction balancing,
not before).
Previously, the accounts passed to account directives would be stripped
of their surrounding brackets, but the required behaviour is to have
account directives plain reject bracketed accounts. This change ensures
that accounts in account directives may not start with a bracket
character.
Currently an account name like "a:(aa)" will not have (aa) unbracketed.
However, this seems reasonable since the full name is unbracketed and
thus will not be confused with virtual or virtual-balanced posting.
"" is equivalent to writing no symbol.
But it's now possible to declare market prices for the no-symbol
commodity, which occurs eg with timedot data.
P 2022-01-01 "" $100
Hledger.Utils.Debug's "trace or log" functions are now controlled as
follows: to enable logging, append ",logging" to the program name at
startup (using withProgName). This also works when running in GHCI.
And they log to PROGNAME.log, not debug.log.
All (hopefully) debug logging in the hledger packages is now "trace or
log" capable.
This means that hledger-ui should now log all debug output to
./hledger-ui.log, with none of it appearing on the console.
Previously, CSV date-times with a different time zone from yours
(with or without explicit timezones in the CSV) could give off-by-one
dates, because the CSV timezone was ignored.
Now,
1. you can use the `timezone` rule to indicate which other
timezone a CSV is implicitly using
2. CSV date-times with a timezone - whether declared by rule or
parsed with %Z - are localised to the system time zone
(or another set with the TZ environment variable).
transaction prices.
When given --infer-costs, hledger will now separately infer transaction
prices for different prices. Given a pair of adjacent conversion
postings, hledger will check if there is a single posting with a
transaction price which matches both the amounts. If so, it associates
those conversion postings to that priced post.
If it can't find any transaction price postings which match, it will
find the first non-transaction price post which matches one of the two
amounts, and will add a transaction price to that, and associate them.
When given --infer-equity, hledger will change transaction prices to balancing equity postings. This introduces the inverse operation, --infer-costs, which will match balancing equity postings and transform them into a transaction price, allowing --cost to work properly with them. This is only a partial inverse as it needs to use some heuristics to match the postings which will not work in complicated cases.
Specifically, when hledger finds exactly two conversion postings in a transaction (by default, subaccounts of equity:conversion or equity:trad(e|ing)), it will find the first posting in the transaction whose amount is negative one of the conversion posting amounts, and inserts the corresponding transaction price.
May also fix#1154, #1033, #708, #536, #73: testing is needed.
This aims to solve all problems where misconfigured locales lead to
parsers failing on utf8-encoded data. This should hopefully avoid
encoding issues, but since it fundamentally alters how encoding is dealt
with it may lead to unexpected outcomes. Widespread testing on a number
of different platforms would be useful.
And remove the last vestiges of older more complex behaviour.
ordereddates now always checks all transactions in each file,
unaffected by a query. (But still affected by --date2).
parseAndFinaliseJournal' has been removed. In the unlikely event you
needed it in your code, you can replace it with:
parseAndFinaliseJournal' parser iopts fp t =>
initialiseAndParseJournal parser iopts fp t
>>= liftEither . journalApplyAliases (aliasesFromOpts iopts)
>>= journalFinalise iopts fp t
Some parsers have been generalised from JournalParser to TextParser.
This increases composability and avoids some ugly case handling. We
re-export runExceptT in Hledger.Read.
The final return types of the following functions has been changed from
IO (Either String a) to ExceptT String IO a. If this causes a problem,
you can get the old behaviour by calling runExceptT on the output:
readJournal, readJournalFiles, readJournalFile
Or, you can use the easy functions readJournal', readJournalFiles', and
readJournalFile', which assume default options and return in the IO
monad.
