Since 1.50.3, canonicalizePath was being called wastefully when
processing journals with many nested include files and/or many matches
for include glob paths. On a slow filesystem, with unusually
many includes, this might have been quite noticeable.
Now we canonicalise each file path just once as it is encountered,
avoiding the wasted IO work.
showing the problem include directive (previously the line number was
off by one). Likewise for other IO errors like when resolving ~ and
a home directory can't be found.
1.50* attempted to work around Glob's implicit searching of non-top-level dot dirs.
This was overzealous; it meant that journal's include completely
excluded paths involving a glob and a dot dir or dot file anywhere in the path.
Now, the pre-1.50 behaviour is restored:
`*` and `**` won't implicitly match dot files or top-level dot directories.
They will implicitly search non-top-level dot directories, as before (#Glob#49).
Before 1.50, journal's include directive's handling of glob patterns (*, **, ?, etc.)
had these limitations:
- ** always searched intermediate dot directories
- ** matched only directories, not files
In 1.50-1.50.3, it had different limitations, some unintended:
- it ignored all dot files, dot dirs, and symbolic links to dot dirs,
even when explicitly mentioned in the pattern (unless using --old-glob)
- it showed symbolic links dereferenced, eg in `hledger files` output
Now it has fewer limitations, mainly this:
- it ignores all dot files and dot dirs, even when explicitly mentioned (unless using --old-glob)
Ie it no longer ignores symbolic links to dot dirs, and it no longer shows symbolic links dereferenced.
Also: including the current file is now always harmless, whether using a glob pattern or not.
Internally, file paths in the "include file stack" (jincludefilestack) are now just absolute,
but not canonicalised; showing symbolic links un-dereferenced in output and error messages seems
generally more useful. This might affect output elsewhere also.
(Those paths are still canonicalised on the fly when checking for include cycles,
not so efficiently: each time an include directive is parsed, all the current parent files
and all the new glob-matched include files will be re-canonicalised.
Hopefully this is unnoticeable.)
Since 1.50, sourceFilePath, which does IO operations, was being called for every item in the journal.
On my machine this was causing a ~40% slowdown,
but probably it could be more depending on storage system.
Now it's once again called only once per include directive.
Speed seems slightly better now than 1.43 for some reason
(eg: 13k txns/s -> 8k txns/s -> 14k txns/s).
The default timeclock parser (ie when not using --old-timeclock) has
the following changes, related to issues such as
[#2141], [#2365], [#2400], [#2417]:
- semicolon now always starts a comment; timeclock account names can't include semicolons
(though journal account names still can)
- clock-in and clock-out entries now have different syntax
- clock-ins now require an account name
- clock-outs now can have a comment and tags
- the doc has been rewritten, and now mentions the --old-timeclock flag
- lib: accountnamep and modifiedaccountnamep now take a flag to allow semicolons or not
Errors in the main file are being reported a few lines too high,
due to the setOffset in includedirectivep.
It seems reverting this should have restored the original bug with
wrong line number in certain include error messages, but I can't find
that right now.
** now ignores anything under dotted directories, ie directories whose
name begins with a dot. Eg .git/, foo/.secret/, etc.
Switched from Glob to filepattern lib.
Previously, hledger could read CSV files containing non-ascii
characters only if they are UTF8-encoded. Now there is a new CSV
rule, encoding ENCODING, which allows reading CSV files with other
encodings.
This adds a dependency on the encoding library, which supports fewer
encodings than text-icu but does not require a third-party C library.
To avoid build issues on various platforms, we require version 0.10+.
This adds some use of the ImplicitParams language extension, required
by encoding's API, but only in a small code region.
This also changes the type of Reader's rReadFn; it now takes
a `Handle` rather than a `Text`, allowing more flexibility.
When reading a symbolically-linked journal file,
relative paths in include directives are now evaluated
relative to the directory of the real linked file,
not the directory of the symlink.
This also seems to fix an obscure case where stats did not report
absolute included file paths in certain circumstances (stdin, maybe no
terminal..)
CSV rules files can now be read directly, eg you have the option of
writing `hledger -f foo.csv.rules CMD`. By default this will read data
from foo.csv in the same directory. But you can also specify a
different data file with a new `source FILE` rule. This has some
convenience features:
- If the data file does not exist, it is treated as empty, not an
error.
- If FILE is a relative path, it is relative to the rules file's
directory. If it is just a file name with no path, it is relative
to ~/Downloads/.
- If FILE is a glob pattern, the most recently modified matched file
is used.
This helps remove some of the busywork of managing CSV downloads.
Most of your financial institutions's default CSV filenames are
different and can be recognised by a glob pattern. So you can put a
rule like `source Checking1*.csv` in foo-checking.csv.rules,
periodically download CSV from Foo's website accepting your browser's
defaults, and then run `hledger import checking.csv.rules` to import
any new transactions. The next time, if you have done no cleanup, your
browser will probably save it as something like Checking1-2.csv, and
hledger will still see that because of the * wild card. You can choose
whether to delete CSVs after import, or keep them for a while as
temporary backups, or archive them somewhere.
