From b14052946500437eed812b78bb3cf00d7d423b25 Mon Sep 17 00:00:00 2001 From: Simon Michael Date: Wed, 25 Jan 2023 11:59:12 -1000 Subject: [PATCH] ;doc: update manuals --- hledger-ui/hledger-ui.1 | 10 + hledger-ui/hledger-ui.info | 64 +- hledger-ui/hledger-ui.txt | 8 + hledger-web/hledger-web.1 | 5 +- hledger-web/hledger-web.info | 42 +- hledger-web/hledger-web.txt | 6 +- hledger/hledger.1 | 2646 +++++++++++++------------ hledger/hledger.info | 3560 +++++++++++++++++----------------- hledger/hledger.txt | 3250 ++++++++++++++++--------------- 9 files changed, 4943 insertions(+), 4648 deletions(-) diff --git a/hledger-ui/hledger-ui.1 b/hledger-ui/hledger-ui.1 index d14ab29ac..31ed13eb5 100644 --- a/hledger-ui/hledger-ui.1 +++ b/hledger-ui/hledger-ui.1 @@ -279,6 +279,16 @@ move automatically to track the current date. To set a non-standard period, you can use \f[V]/\f[R] and a \f[V]date:\f[R] query. .PP +(Mac users: SHIFT-DOWN/UP keys do not work by default in Terminal, as of +MacOS Monterey. +You can configure them as follows: open Terminal, press CMD-comma to +open preferences, click Profiles, select your current terminal profile +on the left, click Keyboard on the right, click + and add this for +Shift-Down: \f[V]\[rs]033[1;2B\f[R], click + and add this for Shift-Up: +\f[V]\[rs]033[1;2A\f[R]. +Press the Escape key to enter the \f[V]\[rs]033\f[R] part, you can\[aq]t +type it directly.) +.PP \f[V]/\f[R] lets you set a general filter query limiting the data shown, using the same query terms as in hledger and hledger-web. While editing the query, you can use CTRL-a/e/d/k, BS, cursor keys; diff --git a/hledger-ui/hledger-ui.info b/hledger-ui/hledger-ui.info index 2ee608292..12301c5c8 100644 --- a/hledger-ui/hledger-ui.info +++ b/hledger-ui/hledger-ui.info @@ -298,6 +298,14 @@ report period to today. With the ‘-w/--watch’ option, when viewing a period will move automatically to track the current date. To set a non-standard period, you can use ‘/’ and a ‘date:’ query. + (Mac users: SHIFT-DOWN/UP keys do not work by default in Terminal, as +of MacOS Monterey. You can configure them as follows: open Terminal, +press CMD-comma to open preferences, click Profiles, select your current +terminal profile on the left, click Keyboard on the right, click + and +add this for Shift-Down: ‘\033[1;2B’, click + and add this for Shift-Up: +‘\033[1;2A’. Press the Escape key to enter the ‘\033’ part, you can’t +type it directly.) + ‘/’ lets you set a general filter query limiting the data shown, using the same query terms as in hledger and hledger-web. While editing the query, you can use CTRL-a/e/d/k, BS, cursor keys; press ‘ENTER’ to @@ -704,34 +712,34 @@ Node: MOUSE7336 Ref: #mouse7431 Node: KEYS7674 Ref: #keys7767 -Node: SCREENS12061 -Ref: #screens12159 -Node: Menu12867 -Ref: #menu12959 -Node: All accounts13036 -Ref: #all-accounts13175 -Node: Balance sheet accounts13434 -Ref: #balance-sheet-accounts13614 -Node: Income statement accounts13806 -Ref: #income-statement-accounts13988 -Node: Register16482 -Ref: #register16619 -Node: Transaction18647 -Ref: #transaction18770 -Node: Error19654 -Ref: #error19748 -Node: TIPS19992 -Ref: #tips20091 -Node: Watch mode20133 -Ref: #watch-mode20240 -Node: Debug output21728 -Ref: #debug-output21839 -Node: ENVIRONMENT22059 -Ref: #environment22170 -Node: FILES23593 -Ref: #files23692 -Node: BUGS23952 -Ref: #bugs24029 +Node: SCREENS12526 +Ref: #screens12624 +Node: Menu13332 +Ref: #menu13424 +Node: All accounts13501 +Ref: #all-accounts13640 +Node: Balance sheet accounts13899 +Ref: #balance-sheet-accounts14079 +Node: Income statement accounts14271 +Ref: #income-statement-accounts14453 +Node: Register16947 +Ref: #register17084 +Node: Transaction19112 +Ref: #transaction19235 +Node: Error20119 +Ref: #error20213 +Node: TIPS20457 +Ref: #tips20556 +Node: Watch mode20598 +Ref: #watch-mode20705 +Node: Debug output22193 +Ref: #debug-output22304 +Node: ENVIRONMENT22524 +Ref: #environment22635 +Node: FILES24058 +Ref: #files24157 +Node: BUGS24417 +Ref: #bugs24494  End Tag Table diff --git a/hledger-ui/hledger-ui.txt b/hledger-ui/hledger-ui.txt index f2d0781ac..e96ca0d4d 100644 --- a/hledger-ui/hledger-ui.txt +++ b/hledger-ui/hledger-ui.txt @@ -258,6 +258,14 @@ KEYS ically to track the current date. To set a non-standard period, you can use / and a date: query. + (Mac users: SHIFT-DOWN/UP keys do not work by default in Terminal, as + of MacOS Monterey. You can configure them as follows: open Terminal, + press CMD-comma to open preferences, click Profiles, select your cur- + rent terminal profile on the left, click Keyboard on the right, click + + and add this for Shift-Down: \033[1;2B, click + and add this for Shift- + Up: \033[1;2A. Press the Escape key to enter the \033 part, you can't + type it directly.) + / lets you set a general filter query limiting the data shown, using the same query terms as in hledger and hledger-web. While editing the query, you can use CTRL-a/e/d/k, BS, cursor keys; press ENTER to set diff --git a/hledger-web/hledger-web.1 b/hledger-web/hledger-web.1 index 743030835..8846b0e24 100644 --- a/hledger-web/hledger-web.1 +++ b/hledger-web/hledger-web.1 @@ -100,8 +100,9 @@ socket. .TP \f[V]--base-url=URL\f[R] set the base url (default: http://IPADDR:PORT). -You would change this when sharing over the network, or integrating -within a larger website. +Note: affects url generation but not route parsing. +Can be useful if running behind a reverse web proxy that does path +rewriting. .TP \f[V]--file-url=URL\f[R] set the static files url (default: BASEURL/static). diff --git a/hledger-web/hledger-web.info b/hledger-web/hledger-web.info index 1f76e74c2..4e0001332 100644 --- a/hledger-web/hledger-web.info +++ b/hledger-web/hledger-web.info @@ -105,9 +105,9 @@ before options, as shown in the synopsis above. operating system can provide this type of socket. ‘--base-url=URL’ - set the base url (default: http://IPADDR:PORT). You would change - this when sharing over the network, or integrating within a larger - website. + set the base url (default: http://IPADDR:PORT). Note: affects url + generation but not route parsing. Can be useful if running behind + a reverse web proxy that does path rewriting. ‘--file-url=URL’ set the static files url (default: BASEURL/static). hledger-web @@ -666,24 +666,24 @@ Tag Table: Node: Top225 Node: OPTIONS2727 Ref: #options2832 -Node: PERMISSIONS10588 -Ref: #permissions10727 -Node: EDITING UPLOADING DOWNLOADING11985 -Ref: #editing-uploading-downloading12166 -Node: RELOADING13008 -Ref: #reloading13142 -Node: JSON API13575 -Ref: #json-api13690 -Node: DEBUG OUTPUT19222 -Ref: #debug-output19347 -Node: Debug output19374 -Ref: #debug-output-119475 -Node: ENVIRONMENT19900 -Ref: #environment20020 -Node: FILES21369 -Ref: #files21469 -Node: BUGS21729 -Ref: #bugs21807 +Node: PERMISSIONS10626 +Ref: #permissions10765 +Node: EDITING UPLOADING DOWNLOADING12023 +Ref: #editing-uploading-downloading12204 +Node: RELOADING13046 +Ref: #reloading13180 +Node: JSON API13613 +Ref: #json-api13728 +Node: DEBUG OUTPUT19260 +Ref: #debug-output19385 +Node: Debug output19412 +Ref: #debug-output-119513 +Node: ENVIRONMENT19938 +Ref: #environment20058 +Node: FILES21407 +Ref: #files21507 +Node: BUGS21767 +Ref: #bugs21845  End Tag Table diff --git a/hledger-web/hledger-web.txt b/hledger-web/hledger-web.txt index bb5339324..5ff0a3817 100644 --- a/hledger-web/hledger-web.txt +++ b/hledger-web/hledger-web.txt @@ -82,9 +82,9 @@ OPTIONS operating system can provide this type of socket. --base-url=URL - set the base url (default: http://IPADDR:PORT). You would - change this when sharing over the network, or integrating within - a larger website. + set the base url (default: http://IPADDR:PORT). Note: affects + url generation but not route parsing. Can be useful if running + behind a reverse web proxy that does path rewriting. --file-url=URL set the static files url (default: BASEURL/static). hledger-web diff --git a/hledger/hledger.1 b/hledger/hledger.1 index 34d4ded9a..c6b3eec53 100644 --- a/hledger/hledger.1 +++ b/hledger/hledger.1 @@ -36,7 +36,7 @@ productively, but when you have a question about functionality, this doc should answer it. It is detailed, so do skip ahead or skim when needed. You can read it on hledger.org, or as an info manual or man page on your -system (each has benefits). +system. You can also get it from hledger itself with .PD 0 .P @@ -1647,18 +1647,62 @@ But if you accidentally leave only one space (or tab) before the amount, the amount will be considered part of the account name. .SS Account names .PP -Account names typically have several parts separated by a full colon, -from which hledger derives a hierarchical chart of accounts. -They can be anything you like, but in finance there are traditionally -five top-level accounts: \f[V]assets\f[R], \f[V]liabilities\f[R], -\f[V]revenue\f[R], \f[V]expenses\f[R], and \f[V]equity\f[R]. +Accounts are the main way of categorising things in hledger. +As in Double Entry Bookkeeping, they can represent real world accounts +(such as a bank account), or more abstract categories such as \[dq]money +borrowed from Frank\[dq] or \[dq]money spent on electricity\[dq]. .PP -Account names may contain single spaces, eg: -\f[V]assets:accounts receivable\f[R]. -Because of this, they must always be followed by \f[B]two or more -spaces\f[R] (or newline). +You can use any account names you like, but we usually start with the +traditional accounting categories, which in english are +\f[V]assets\f[R], \f[V]liabilities\f[R], \f[V]equity\f[R], +\f[V]revenues\f[R], \f[V]expenses\f[R]. +(You might see these referred to as A, L, E, R, X for short.) .PP -Account names can be aliased. +For more precise reporting, we usually divide the top level accounts +into more detailed subaccounts, by writing a full colon between account +name parts. +For example, from the account names \f[V]assets:bank:checking\f[R] and +\f[V]expenses:food\f[R], hledger will infer this hierarchy of five +accounts: +.IP +.nf +\f[C] +assets +assets:bank +assets:bank:checking +expenses +expenses:food +\f[R] +.fi +.PP +Shown as an outline, the hierarchical tree structure is more clear: +.IP +.nf +\f[C] +assets + bank + checking +expenses + food +\f[R] +.fi +.PP +hledger reports can summarise the account tree to any depth, so you can +go as deep as you like with subcategories, but keeping your account +names relatively simple may be best when starting out. +.PP +Account names may be capitalised or not; they may contain letters, +numbers, symbols, or single spaces. +Note, when an account name and an amount are written on the same line, +they must be separated by \f[B]two or more spaces\f[R] (or tabs). +.PP +Parentheses or brackets enclosing the full account name indicate virtual +postings, described below. +Parentheses or brackets internal to the account name have no special +meaning. +.PP +Account names can be altered temporarily or permanently by account +aliases. .SS Amounts .PP After the account name, there is usually an amount. @@ -2597,6 +2641,17 @@ hledger-style account name, eg: account assets:bank:checking \f[R] .fi +.PP +Note, however, that accounts declared in account directives are not +allowed to have surrounding brackets and parentheses, unlike accounts +used in postings. +So the following journal will not parse: +.IP +.nf +\f[C] +account (assets:bank:checking) +\f[R] +.fi .SS Account comments .PP Text following \f[B]two or more spaces\f[R] and \f[V];\f[R] at the end @@ -3827,6 +3882,7 @@ value EXPR .PP See also https://hledger.org/ledger.html for a detailed hledger/Ledger syntax comparison. +.PP .SH CSV .PP hledger can read CSV files (Character Separated Value - usually comma, @@ -3835,10 +3891,9 @@ each record into a transaction. .PP (To learn about \f[I]writing\f[R] CSV, see CSV output.) .PP -Note, for best error messages when reading CSV/TSV/SSV files, make sure -they have a corresponding \f[V].csv\f[R], \f[V].tsv\f[R] or -\f[V].ssv\f[R] file extension or use a hledger file prefix (see File -Extension below). +For best error messages when reading CSV/TSV/SSV files, make sure they +have a corresponding \f[V].csv\f[R], \f[V].tsv\f[R] or \f[V].ssv\f[R] +file extension or use a hledger file prefix (see File Extension below). .PP Each CSV file must be described by a corresponding \f[I]rules file\f[R]. .PD 0 @@ -3857,17 +3912,6 @@ option. If no rules file is found, hledger will create a sample rules file, which you\[aq]ll need to adjust. .PP -There\[aq]s an introductory Importing CSV data tutorial on hledger.org. -.SS Examples -.PP -Here are some sample hledger CSV rules files. -See also the full collection at: -.PD 0 -.P -.PD -https://github.com/simonmichael/hledger/tree/master/examples/csv -.SS Basic -.PP At minimum, the rules file must identify the date and amount fields, and often it also specifies the date format and how many header lines there are. @@ -3884,7 +3928,7 @@ Date, Description, Id, Amount \f[C] # basic.csv.rules skip 1 -fields date, description, _, amount +fields date, description, , amount date-format %d/%m/%Y \f[R] .fi @@ -3898,7 +3942,1265 @@ $ hledger print -f basic.csv \f[R] .fi .PP -Default account names are chosen, since we didn\[aq]t set them. +There\[aq]s an introductory Importing CSV data tutorial on hledger.org, +and more CSV rules examples below, and a larger collection at +https://github.com/simonmichael/hledger/tree/master/examples/csv. +.SS CSV rules cheatsheet +.PP +The following kinds of rule can appear in the rules file, in any order. +(Blank lines and lines beginning with \f[V]#\f[R] or \f[V];\f[R] or +\f[V]*\f[R] are ignored.) +.PP +.TS +tab(@); +lw(23.7n) lw(46.3n). +T{ +\f[B]\f[VB]separator\f[B]\f[R] +T}@T{ +declare the field separator, instead of relying on file extension +T} +T{ +\f[B]\f[VB]skip\f[B]\f[R] +T}@T{ +skip one or more header lines at start of file +T} +T{ +\f[B]\f[VB]date-format\f[B]\f[R] +T}@T{ +declare how to parse CSV dates/date-times +T} +T{ +\f[B]\f[VB]timezone\f[B]\f[R] +T}@T{ +declare the time zone of ambiguous CSV date-times +T} +T{ +\f[B]\f[VB]newest-first\f[B]\f[R] +T}@T{ +improve txn order when: there are multiple records, newest first, all +with the same date +T} +T{ +\f[B]\f[VB]intra-day-reversed\f[B]\f[R] +T}@T{ +improve txn order when: same-day txns are in opposite order to the +overall file +T} +T{ +\f[B]\f[VB]decimal-mark\f[B]\f[R] +T}@T{ +declare the decimal mark used in CSV amounts, when ambiguous +T} +T{ +\f[B]\f[VB]fields\f[B] list\f[R] +T}@T{ +name CSV fields for easy reference, and optionally assign their values +to hledger fields +T} +T{ +\f[B]Field assignment\f[R] +T}@T{ +assign a CSV value or interpolated text value to a hledger field +T} +T{ +\f[B]\f[VB]if\f[B] block\f[R] +T}@T{ +conditionally assign values to hledger fields, or \f[V]skip\f[R] a +record or \f[V]end\f[R] (skip rest of file) +T} +T{ +\f[B]\f[VB]if\f[B] table\f[R] +T}@T{ +conditionally assign values to hledger fields, using compact syntax +T} +T{ +\f[B]\f[VB]balance-type\f[B]\f[R] +T}@T{ +select which type of balance assertions/assignments to generate +T} +T{ +\f[B]\f[VB]include\f[B]\f[R] +T}@T{ +inline another CSV rules file +T} +.TE +.PP +Working with CSV tips can be found below, including How CSV rules are +evaluated. +.SS \f[V]separator\f[R] +.PP +You can use the \f[V]separator\f[R] rule to read other kinds of +character-separated data. +The argument is any single separator character, or the words +\f[V]tab\f[R] or \f[V]space\f[R] (case insensitive). +Eg, for comma-separated values (CSV): +.IP +.nf +\f[C] +separator , +\f[R] +.fi +.PP +or for semicolon-separated values (SSV): +.IP +.nf +\f[C] +separator ; +\f[R] +.fi +.PP +or for tab-separated values (TSV): +.IP +.nf +\f[C] +separator TAB +\f[R] +.fi +.PP +If the input file has a \f[V].csv\f[R], \f[V].ssv\f[R] or \f[V].tsv\f[R] +file extension (or a \f[V]csv:\f[R], \f[V]ssv:\f[R], \f[V]tsv:\f[R] +prefix), the appropriate separator will be inferred automatically, and +you won\[aq]t need this rule. +.SS \f[V]skip\f[R] +.IP +.nf +\f[C] +skip N +\f[R] +.fi +.PP +The word \f[V]skip\f[R] followed by a number (or no number, meaning 1) +tells hledger to ignore this many non-empty lines at the start of the +input data. +(Empty/blank lines are skipped automatically, so you don\[aq]t need to +count those.) +You\[aq]ll need this whenever your CSV data contains header lines. +Header lines skipped in this way are ignored, and not parsed as CSV. +.PP +\f[V]skip\f[R] can also be used inside if blocks (described below), to +skip individual data records. +Note records skipped in this way are still required to be valid CSV, +even though otherwise ignored. +.SS \f[V]date-format\f[R] +.IP +.nf +\f[C] +date-format DATEFMT +\f[R] +.fi +.PP +This is a helper for the \f[V]date\f[R] (and \f[V]date2\f[R]) fields. +If your CSV dates are not formatted like \f[V]YYYY-MM-DD\f[R], +\f[V]YYYY/MM/DD\f[R] or \f[V]YYYY.MM.DD\f[R], you\[aq]ll need to add a +date-format rule describing them with a strptime-style date parsing +pattern - see +https://hackage.haskell.org/package/time/docs/Data-Time-Format.html#v:formatTime. +The pattern must parse the CSV date value completely. +Some examples: +.IP +.nf +\f[C] +# MM/DD/YY +date-format %m/%d/%y +\f[R] +.fi +.IP +.nf +\f[C] +# D/M/YYYY +# The - makes leading zeros optional. +date-format %-d/%-m/%Y +\f[R] +.fi +.IP +.nf +\f[C] +# YYYY-Mmm-DD +date-format %Y-%h-%d +\f[R] +.fi +.IP +.nf +\f[C] +# M/D/YYYY HH:MM AM some other junk +# Note the time and junk must be fully parsed, though only the date is used. +date-format %-m/%-d/%Y %l:%M %p some other junk +\f[R] +.fi +.SS \f[V]timezone\f[R] +.IP +.nf +\f[C] +timezone TIMEZONE +\f[R] +.fi +.PP +When CSV contains date-times that are implicitly in some time zone other +than yours, but containing no explicit time zone information, you can +use this rule to declare the CSV\[aq]s native time zone, which helps +prevent off-by-one dates. +.PP +When the CSV date-times do contain time zone information, you don\[aq]t +need this rule; instead, use \f[V]%Z\f[R] in \f[V]date-format\f[R] (or +\f[V]%z\f[R], \f[V]%EZ\f[R], \f[V]%Ez\f[R]; see the formatTime link +above). +.PP +In either of these cases, hledger will do a time-zone-aware conversion, +localising the CSV date-times to your current system time zone. +If you prefer to localise to some other time zone, eg for +reproducibility, you can (on unix at least) set the output timezone with +the TZ environment variable, eg: +.IP +.nf +\f[C] +$ TZ=-1000 hledger print -f foo.csv # or TZ=-1000 hledger import foo.csv +\f[R] +.fi +.PP +\f[V]timezone\f[R] currently does not understand timezone names, except +\[dq]UTC\[dq], \[dq]GMT\[dq], \[dq]EST\[dq], \[dq]EDT\[dq], +\[dq]CST\[dq], \[dq]CDT\[dq], \[dq]MST\[dq], \[dq]MDT\[dq], +\[dq]PST\[dq], or \[dq]PDT\[dq]. +For others, use numeric format: +HHMM or -HHMM. +.SS \f[V]newest-first\f[R] +.PP +hledger tries to ensure that the generated transactions will be ordered +chronologically, including intra-day transactions. +Usually it can auto-detect how the CSV records are ordered. +But if it encounters CSV where all records are on the same date, it +assumes that the records are oldest first. +If in fact the CSV\[aq]s records are normally newest first, like: +.IP +.nf +\f[C] +2022-10-01, txn 3... +2022-10-01, txn 2... +2022-10-01, txn 1... +\f[R] +.fi +.PP +you can add the \f[V]newest-first\f[R] rule to help hledger generate the +transactions in correct order. +.IP +.nf +\f[C] +# same-day CSV records are newest first +newest-first +\f[R] +.fi +.SS \f[V]intra-day-reversed\f[R] +.PP +CSV records for each day are sometimes ordered in reverse compared to +the overall date order. +Eg, here dates are newest first, but the transactions on each date are +oldest first: +.IP +.nf +\f[C] +2022-10-02, txn 3... +2022-10-02, txn 4... +2022-10-01, txn 1... +2022-10-01, txn 2... +\f[R] +.fi +.PP +In this situation, add the \f[V]intra-day-reversed\f[R] rule, and +hledger will compensate, improving the order of transactions. +.IP +.nf +\f[C] +# transactions within each day are reversed with respect to the overall date order +intra-day-reversed +\f[R] +.fi +.SS \f[V]decimal-mark\f[R] +.IP +.nf +\f[C] +decimal-mark . +\f[R] +.fi +.PP +or: +.IP +.nf +\f[C] +decimal-mark , +\f[R] +.fi +.PP +hledger automatically accepts either period or comma as a decimal mark +when parsing numbers (cf Amounts). +However if any numbers in the CSV contain digit group marks, such as +thousand-separating commas, you should declare the decimal mark +explicitly with this rule, to avoid misparsed numbers. +.SS \f[V]fields\f[R] list +.IP +.nf +\f[C] +fields FIELDNAME1, FIELDNAME2, ... +\f[R] +.fi +.PP +A fields list (the word \f[V]fields\f[R] followed by comma-separated +field names) is optional, but convenient. +It does two things: +.IP "1." 3 +It names the CSV field in each column. +This can be convenient if you are referencing them in other rules, so +you can say \f[V]%SomeField\f[R] instead of remembering \f[V]%13\f[R]. +.IP "2." 3 +Whenever you use one of the special hledger field names (described +below), it assigns the CSV value in this position to that hledger field. +This is the quickest way to populate hledger\[aq]s fields and build a +transaction. +.PP +Here\[aq]s an example that says \[dq]use the 1st, 2nd and 4th fields as +the transaction\[aq]s date, description and amount; name the last two +fields for later reference; and ignore the others\[dq]: +.IP +.nf +\f[C] +fields date, description, , amount, , , somefield, anotherfield +\f[R] +.fi +.PP +In a fields list, the separator is always comma; it is unrelated to the +CSV file\[aq]s separator. +Also: +.IP \[bu] 2 +There must be least two items in the list (at least one comma). +.IP \[bu] 2 +Field names may not contain spaces. +Spaces before/after field names are optional. +.IP \[bu] 2 +Field names may contain \f[V]_\f[R] (underscore) or \f[V]-\f[R] +(hyphen). +.IP \[bu] 2 +Fields you don\[aq]t care about can be given a dummy name or an empty +name. +.PP +If the CSV contains column headings, it\[aq]s convenient to use these +for your field names, suitably modified (eg lower-cased with spaces +replaced by underscores). +.PP +Sometimes you may want to alter a CSV field name to avoid assigning to a +hledger field with the same name. +Eg you could call the CSV\[aq]s \[dq]balance\[dq] field +\f[V]balance_\f[R] to avoid directly setting hledger\[aq]s +\f[V]balance\f[R] field (and generating a balance assertion). +.SS Field assignment +.IP +.nf +\f[C] +HLEDGERFIELD FIELDVALUE +\f[R] +.fi +.PP +Field assignments are the more flexible way to assign CSV values to +hledger fields. +They can be used instead of or in addition to a fields list (see above). +.PP +To assign a value to a hledger field, write the field name (any of the +standard hledger field/pseudo-field names, defined below), a space, +followed by a text value on the same line. +This text value may interpolate CSV fields, referenced by their 1-based +position in the CSV record (\f[V]%N\f[R]), or by the name they were +given in the fields list (\f[V]%CSVFIELD\f[R]). +.PP +Some examples: +.IP +.nf +\f[C] +# set the amount to the 4th CSV field, with \[dq] USD\[dq] appended +amount %4 USD + +# combine three fields to make a comment, containing note: and date: tags +comment note: %somefield - %anotherfield, date: %1 +\f[R] +.fi +.PP +Tips: +.IP \[bu] 2 +Interpolation strips outer whitespace (so a CSV value like +\f[V]\[dq] 1 \[dq]\f[R] becomes \f[V]1\f[R] when interpolated) (#1051). +.IP \[bu] 2 +Interpolations always refer to a CSV field - you can\[aq]t interpolate a +hledger field. +(See Referencing other fields below). +.SS Field names +.PP +Note the two kinds of field names mentioned here, and used only in +hledger CSV rules files: +.IP "1." 3 +\f[B]CSV field names\f[R] (\f[V]CSVFIELD\f[R] in these docs): you can +optionally name the CSV columns for easy reference (since hledger +doesn\[aq]t yet automatically recognise column headings in a CSV file), +by writing arbitrary names in a \f[V]fields\f[R] list, eg: +.RS 4 +.IP +.nf +\f[C] +fields When, What, Some_Id, Net, Total, Foo, Bar +\f[R] +.fi +.RE +.IP "2." 3 +Special \f[B]hledger field names\f[R] (\f[V]HLEDGERFIELD\f[R] in these +docs): you must set at least some of these to generate the hledger +transaction from a CSV record, by writing them as the left hand side of +a field assignment, eg: +.RS 4 +.IP +.nf +\f[C] +date %When +code %Some_Id +description %What +comment %Foo %Bar +amount1 $ %Total +\f[R] +.fi +.PP +or directly in a \f[V]fields\f[R] list: +.IP +.nf +\f[C] +fields date, description, code, , amount1, Foo, Bar +currency $ +comment %Foo %Bar +\f[R] +.fi +.RE +.PP +Here are all the special hledger field names available, and what happens +when you assign values to them: +.SS date field +.PP +Assigning to \f[V]date\f[R] sets the transaction date. +.SS date2 field +.PP +\f[V]date2\f[R] sets the transaction\[aq]s secondary date, if any. +.SS status field +.PP +\f[V]status\f[R] sets the transaction\[aq]s status, if any. +.SS code field +.PP +\f[V]code\f[R] sets the transaction\[aq]s code, if any. +.SS description field +.PP +\f[V]description\f[R] sets the transaction\[aq]s description, if any. +.SS comment field +.PP +\f[V]comment\f[R] sets the transaction\[aq]s comment, if any. +.PP +\f[V]commentN\f[R], where N is a number, sets the Nth posting\[aq]s +comment. +.PP +You can assign multi-line comments by writing literal \f[V]\[rs]n\f[R] +in the code. +A comment starting with \f[V]\[rs]n\f[R] will begin on a new line. +.PP +Comments can contain tags, as usual. +.SS account field +.PP +Assigning to \f[V]accountN\f[R], where N is 1 to 99, sets the account +name of the Nth posting, and causes that posting to be generated. +.PP +Most often there are two postings, so you\[aq]ll want to set +\f[V]account1\f[R] and \f[V]account2\f[R]. +Typically \f[V]account1\f[R] is associated with the CSV file, and is set +once with a top-level assignment, while \f[V]account2\f[R] is set based +on each transaction\[aq]s description, in conditional rules. +.PP +If a posting\[aq]s account name is left unset but its amount is set (see +below), a default account name will be chosen (like +\[dq]expenses:unknown\[dq] or \[dq]income:unknown\[dq]). +.SS amount field +.PP +There are several \[dq]amount\[dq] field name variants, useful for +different situations: +.IP \[bu] 2 +\f[V]amountN\f[R] sets the amount of the Nth posting, and causes that +posting to be generated. +By assigning to \f[V]amount1\f[R], \f[V]amount2\f[R], ... +etc. +you can generate up to 99 postings. +Posting numbers don\[aq]t have to be consecutive; in certain situations +using a high number might be helpful to influence the layout of +postings. +.IP \[bu] 2 +\f[V]amountN-in\f[R] and \f[V]amountN-out\f[R] should be used instead, +as a pair, when and only when the amount must be obtained from two CSV +fields. +Eg when the CSV has separate Debit and Credit fields instead of a single +Amount field. +Note: +.RS 2 +.IP \[bu] 2 +Don\[aq]t think \[dq]-in is for the first posting and -out is for the +second posting\[dq] - that\[aq]s not correct. +Think: \[dq]\f[V]amountN-in\f[R] and \f[V]amountN-out\f[R] together +detect the amount for posting N, by inspecting two CSV fields at +once.\[dq] +.IP \[bu] 2 +hledger assumes both CSV fields are unsigned, and will automatically +negate the -out value. +.IP \[bu] 2 +It also expects that at least one of the values is empty or zero, so it +knows which one to ignore. +If that\[aq]s not the case you\[aq]ll need an if rule (see Setting +amounts below). +.RE +.IP \[bu] 2 +\f[V]amount\f[R], with no posting number (and similarly, +\f[V]amount-in\f[R] and \f[V]amount-out\f[R] with no number) are an +older syntax. +We keep them for backwards compatibility, and because they have special +behaviour that is sometimes convenient: +.RS 2 +.IP \[bu] 2 +They set the amount of posting 1 and (negated) the amount of posting 2. +.IP \[bu] 2 +Posting 2\[aq]s amount will be converted to cost if it has a cost price. +.IP \[bu] 2 +Any of the newer rules for posting 1 or 2 (like \f[V]amount1\f[R], or +\f[V]amount2-in\f[R] and \f[V]amount2-out\f[R]) will take precedence. +This allows incrementally migrating old rules files to the new syntax. +.RE +.PP +There\[aq]s more to say about amount-setting that doesn\[aq]t fit here; +please see also \[dq]Setting amounts\[dq] below. +.SS currency field +.PP +\f[V]currency\f[R] sets a currency symbol, to be prepended to all +postings\[aq] amounts. +You can use this if the CSV amounts do not have a currency symbol, eg if +it is in a separate column. +.PP +\f[V]currencyN\f[R] prepends a currency symbol to just the Nth +posting\[aq]s amount. +.SS balance field +.PP +\f[V]balanceN\f[R] sets a balance assertion amount (or if the posting +amount is left empty, a balance assignment) on posting N. +.PP +\f[V]balance\f[R] is a compatibility spelling for hledger <1.17; it is +equivalent to \f[V]balance1\f[R]. +.PP +You can adjust the type of assertion/assignment with the +\f[V]balance-type\f[R] rule (see below). +.PP +See Tips below for more about setting amounts and currency. +.SS \f[V]if\f[R] block +.PP +Rules can be applied conditionally, depending on patterns in the CSV +data. +This allows flexibility; in particular, it is how you can categorise +transactions, selecting an appropriate account name based on their +description (for example). +There are two ways to write conditional rules: \[dq]if blocks\[dq], +described here, and \[dq]if tables\[dq], described below. +.PP +An if block is the word \f[V]if\f[R] and one or more \[dq]matcher\[dq] +expressions (can be a word or phrase), one per line, starting either on +the same or next line; followed by one or more indented rules. +Eg, +.IP +.nf +\f[C] +if MATCHER + RULE +\f[R] +.fi +.PP +or +.IP +.nf +\f[C] +if +MATCHER +MATCHER +MATCHER + RULE + RULE +\f[R] +.fi +.PP +If any of the matchers succeeds, all of the indented rules will be +applied. +They are usually field assignments, but the following special rules may +also be used within an if block: +.IP \[bu] 2 +\f[V]skip\f[R] - skips the matched CSV record (generating no transaction +from it) +.IP \[bu] 2 +\f[V]end\f[R] - skips the rest of the current CSV file. +.PP +Some examples: +.IP +.nf +\f[C] +# if the record contains \[dq]groceries\[dq], set account2 to \[dq]expenses:groceries\[dq] +if groceries + account2 expenses:groceries +\f[R] +.fi +.IP +.nf +\f[C] +# if the record contains any of these phrases, set account2 and a transaction comment as shown +if +monthly service fee +atm transaction fee +banking thru software + account2 expenses:business:banking + comment XXX deductible ? check it +\f[R] +.fi +.IP +.nf +\f[C] +# if an empty record is seen (assuming five fields), ignore the rest of the CSV file +if ,,,, + end +\f[R] +.fi +.SS Matchers +.PP +There are two kinds: +.IP "1." 3 +A record matcher is a word or single-line text fragment or regular +expression (\f[V]REGEX\f[R]), which hledger will try to match +case-insensitively anywhere within the CSV record. +.PD 0 +.P +.PD +Eg: \f[V]whole foods\f[R] +.IP "2." 3 +A field matcher is preceded with a percent sign and CSV field name +(\f[V]%CSVFIELD REGEX\f[R]). +hledger will try to match these just within the named CSV field. +.PD 0 +.P +.PD +Eg: \f[V]%date 2023\f[R] +.PP +The regular expression is (as usual in hledger) a POSIX extended regular +expression, that also supports GNU word boundaries (\f[V]\[rs]b\f[R], +\f[V]\[rs]B\f[R], \f[V]\[rs]<\f[R], \f[V]\[rs]>\f[R]), and nothing else. +If you have trouble, see \[dq]Regular expressions\[dq] in the hledger +manual (https://hledger.org/hledger.html#regular-expressions). +.PP +With record matchers, it\[aq]s important to know that the record matched +is not the original CSV record, but a modified one: separators will be +converted to commas, and enclosing double quotes (but not enclosing +whitespace) are removed. +So for example, when reading an SSV file, if the original record was: +.IP +.nf +\f[C] +2020-01-01; \[dq]Acme, Inc.\[dq]; 1,000 +\f[R] +.fi +.PP +the regex would see, and try to match, this modified record text: +.IP +.nf +\f[C] +2020-01-01,Acme, Inc., 1,000 +\f[R] +.fi +.PP +When an if block has multiple matchers, they are combined as follows: +.IP \[bu] 2 +By default they are OR\[aq]d (any one of them can match) +.IP \[bu] 2 +When a matcher is preceded by ampersand (\f[V]&\f[R]) it will be +AND\[aq]ed with the previous matcher (both of them must match). +.PP +There\[aq]s not yet an easy syntax to negate a matcher. +.SS \f[V]if\f[R] table +.PP +\[dq]if tables\[dq] are an alternative to if blocks; they can express +many matchers and field assignments in a more compact tabular format, +like this: +.IP +.nf +\f[C] +if,HLEDGERFIELD1,HLEDGERFIELD2,... +MATCHERA,VALUE1,VALUE2,... +MATCHERB,VALUE1,VALUE2,... +MATCHERC,VALUE1,VALUE2,... + +\f[R] +.fi +.PP +The first character after \f[V]if\f[R] is taken to be the separator for +the rest of the table. +It should be a non-alphanumeric character like \f[V],\f[R] or +\f[V]|\f[R] that does not appear anywhere else in the table. +(Note: it is unrelated to the CSV file\[aq]s separator.) +Whitespace can be used in the matcher lines for readability, but not in +the if line currently. +The table must be terminated by an empty line (or end of file). +Each line must contain the same number of separators; empty values are +allowed. +.PP +The above means: try all of the matchers; whenever a matcher succeeds, +assign all of the values on that line to the corresponding hledger +fields; later lines can overrider earlier ones. +It is equivalent to this sequence of if blocks: +.IP +.nf +\f[C] +if MATCHERA + HLEDGERFIELD1 VALUE1 + HLEDGERFIELD2 VALUE2 + ... + +if MATCHERB + HLEDGERFIELD1 VALUE1 + HLEDGERFIELD2 VALUE2 + ... + +if MATCHERC + HLEDGERFIELD1 VALUE1 + HLEDGERFIELD2 VALUE2 + ... +\f[R] +.fi +.PP +Example: +.IP +.nf +\f[C] +if,account2,comment +atm transaction fee,expenses:business:banking,deductible? check it +%description groceries,expenses:groceries, +2020/01/12.*Plumbing LLC,expenses:house:upkeep,emergency plumbing call-out +\f[R] +.fi +.SS \f[V]balance-type\f[R] +.PP +Balance assertions generated by assigning to balanceN are of the simple +\f[V]=\f[R] type by default, which is a single-commodity, +subaccount-excluding assertion. +You may find the subaccount-including variants more useful, eg if you +have created some virtual subaccounts of checking to help with +budgeting. +You can select a different type of assertion with the +\f[V]balance-type\f[R] rule: +.IP +.nf +\f[C] +# balance assertions will consider all commodities and all subaccounts +balance-type ==* +\f[R] +.fi +.PP +Here are the balance assertion types for quick reference: +.IP +.nf +\f[C] += single commodity, exclude subaccounts +=* single commodity, include subaccounts +== multi commodity, exclude subaccounts +==* multi commodity, include subaccounts +\f[R] +.fi +.SS \f[V]include\f[R] +.IP +.nf +\f[C] +include RULESFILE +\f[R] +.fi +.PP +This includes the contents of another CSV rules file at this point. +\f[V]RULESFILE\f[R] is an absolute file path or a path relative to the +current file\[aq]s directory. +This can be useful for sharing common rules between several rules files, +eg: +.IP +.nf +\f[C] +# someaccount.csv.rules + +## someaccount-specific rules +fields date,description,amount +account1 assets:someaccount +account2 expenses:misc + +## common rules +include categorisation.rules +\f[R] +.fi +.SS Working with CSV +.PP +Some tips: +.SS Rapid feedback +.PP +It\[aq]s a good idea to get rapid feedback while +creating/troubleshooting CSV rules. +Here\[aq]s a good way, using entr from eradman.com/entrproject: +.IP +.nf +\f[C] +$ ls foo.csv* | entr bash -c \[aq]echo ----; hledger -f foo.csv print desc:SOMEDESC\[aq] +\f[R] +.fi +.PP +A desc: query (eg) is used to select just one, or a few, transactions of +interest. +\[dq]bash -c\[dq] is used to run multiple commands, so we can echo a +separator each time the command re-runs, making it easier to read the +output. +.SS Valid CSV +.PP +Note that hledger will only accept valid CSV conforming to RFC 4180, and +equivalent SSV and TSV formats (like RFC 4180 but with semicolon or tab +as separators). +This means, eg: +.IP \[bu] 2 +Values may be enclosed in double quotes, or not. +Enclosing in single quotes is not allowed. +(Eg \f[V]\[aq]A\[aq],\[aq]B\[aq]\f[R] is rejected.) +.IP \[bu] 2 +When values are enclosed in double quotes, spaces outside the quotes are +not allowed. +(Eg \f[V]\[dq]A\[dq], \[dq]B\[dq]\f[R] is rejected.) +.IP \[bu] 2 +When values are not enclosed in quotes, they may not contain double +quotes. +(Eg \f[V]A\[dq]A, B\f[R] is rejected.) +.PP +If your CSV/SSV/TSV is not valid in this sense, you\[aq]ll need to +transform it before reading with hledger. +Try using sed, or a more permissive CSV parser like python\[aq]s csv +lib. +.SS File Extension +.PP +To help hledger choose the CSV file reader and show the right error +messages (and choose the right field separator character by default), +it\[aq]s best if CSV/SSV/TSV files are named with a \f[V].csv\f[R], +\f[V].ssv\f[R] or \f[V].tsv\f[R] filename extension. +(More about this at Data formats.) +.PP +When reading files with the \[dq]wrong\[dq] extension, you can ensure +the CSV reader (and the default field separator) by prefixing the file +path with \f[V]csv:\f[R], \f[V]ssv:\f[R] or \f[V]tsv:\f[R]: Eg: +.IP +.nf +\f[C] +$ hledger -f ssv:foo.dat print +\f[R] +.fi +.PP +You can also override the default field separator with a separator rule +if needed. +.SS Reading CSV from standard input +.PP +You\[aq]ll need the file format prefix when reading CSV from stdin also, +since hledger assumes journal format by default. +Eg: +.IP +.nf +\f[C] +$ cat foo.dat | hledger -f ssv:- print +\f[R] +.fi +.SS Reading multiple CSV files +.PP +If you use multiple \f[V]-f\f[R] options to read multiple CSV files at +once, hledger will look for a correspondingly-named rules file for each +CSV file. +But if you use the \f[V]--rules-file\f[R] option, that rules file will +be used for all the CSV files. +.SS Valid transactions +.PP +After reading a CSV file, hledger post-processes and validates the +generated journal entries as it would for a journal file - balancing +them, applying balance assignments, and canonicalising amount styles. +Any errors at this stage will be reported in the usual way, displaying +the problem entry. +.PP +There is one exception: balance assertions, if you have generated them, +will not be checked, since normally these will work only when the CSV +data is part of the main journal. +If you do need to check balance assertions generated from CSV right +away, pipe into another hledger: +.IP +.nf +\f[C] +$ hledger -f file.csv print | hledger -f- print +\f[R] +.fi +.SS Deduplicating, importing +.PP +When you download a CSV file periodically, eg to get your latest bank +transactions, the new file may overlap with the old one, containing some +of the same records. +.PP +The import command will (a) detect the new transactions, and (b) append +just those transactions to your main journal. +It is idempotent, so you don\[aq]t have to remember how many times you +ran it or with which version of the CSV. +(It keeps state in a hidden \f[V].latest.FILE.csv\f[R] file.) +This is the easiest way to import CSV data. +Eg: +.IP +.nf +\f[C] +# download the latest CSV files, then run this command. +# Note, no -f flags needed here. +$ hledger import *.csv [--dry] +\f[R] +.fi +.PP +This method works for most CSV files. +(Where records have a stable chronological order, and new records appear +only at the new end.) +.PP +A number of other tools and workflows, hledger-specific and otherwise, +exist for converting, deduplicating, classifying and managing CSV data. +See: +.IP \[bu] 2 +https://hledger.org/cookbook.html#setups-and-workflows +.IP \[bu] 2 +https://plaintextaccounting.org -> data import/conversion +.SS Setting amounts +.PP +Continuing from amount field above, here are more tips on handling +various amount-setting situations: +.IP "1." 3 +\f[B]If the amount is in a single CSV field:\f[R] +.PD 0 +.P +.PD +.RS 4 +.IP "a." 3 +\f[B]If its sign indicates direction of flow:\f[R] +.PD 0 +.P +.PD +Assign it to \f[V]amountN\f[R], to set the Nth posting\[aq]s amount. +N is usually 1 or 2 but can go up to 99. +.IP "b." 3 +\f[B]If another field indicates direction of flow:\f[R] +.PD 0 +.P +.PD +Use one or more conditional rules to set the appropriate amount sign. +Eg: +.IP +.nf +\f[C] +# assume a withdrawal unless Type contains \[dq]deposit\[dq]: +amount1 -%Amount +if %Type deposit + amount1 %Amount +\f[R] +.fi +.RE +.IP "2." 3 +\f[B]If the amount is in one of two CSV fields (eg Debit and +Credit):\f[R] +.PD 0 +.P +.PD +.RS 4 +.IP "a." 3 +\f[B]If both fields are unsigned:\f[R] +.PD 0 +.P +.PD +Assign the fields to \f[V]amountN-in\f[R] and \f[V]amountN-out\f[R]. +This sets posting N\[aq]s amount to whichever of these has a non-zero +value. +If it\[aq]s the -out value, the amount will be negated. +.IP "b." 3 +\f[B]If either field is signed:\f[R] +.PD 0 +.P +.PD +Use a conditional rule to flip the sign when needed. +Eg below, the -out value already has a minus sign so we undo +hledger\[aq]s automatic negating by negating once more (but only if the +field is non-empty, so that we don\[aq]t leave a minus sign by itself): +.IP +.nf +\f[C] +fields date, description, amount1-in, amount1-out +if %amount1-out [1-9] + amount1-out -%amount1-out +\f[R] +.fi +.IP "c." 3 +\f[B]If both fields can contain a non-zero value (or both can be +empty):\f[R] +.PD 0 +.P +.PD +The -in/-out rules normally choose the value which is +non-zero/non-empty. +Some value pairs can be ambiguous, such as \f[V]1\f[R] and +\f[V]none\f[R]. +For such cases, use conditional rules to help select the amount. +Eg, to handle the above you could select the value containing non-zero +digits: +.IP +.nf +\f[C] +fields date, description, in, out +if %in [1-9] + amount1 %in +if %out [1-9] + amount1 %out +\f[R] +.fi +.RE +.IP "3." 3 +\f[B]If you want posting 2\[aq]s amount converted to cost:\f[R] +.PD 0 +.P +.PD +Use the unnumbered \f[V]amount\f[R] (or \f[V]amount-in\f[R] and +\f[V]amount-out\f[R]) syntax. +.IP "4." 3 +\f[B]If the CSV has only balance amounts, not transaction amounts:\f[R] +.PD 0 +.P +.PD +Assign to \f[V]balanceN\f[R], to set a balance assignment on the Nth +posting, causing the posting\[aq]s amount to be calculated +automatically. +\f[V]balance\f[R] with no number is equivalent to \f[V]balance1\f[R]. +In this situation hledger is more likely to guess the wrong default +account name, so you may need to set that explicitly. +.SS Amount signs +.PP +There is some special handling for amount signs, to simplify parsing and +sign-flipping: +.IP \[bu] 2 +\f[B]If an amount value begins with a plus sign:\f[R] +.PD 0 +.P +.PD +that will be removed: \f[V]+AMT\f[R] becomes \f[V]AMT\f[R] +.IP \[bu] 2 +\f[B]If an amount value is parenthesised:\f[R] +.PD 0 +.P +.PD +it will be de-parenthesised and sign-flipped: \f[V](AMT)\f[R] becomes +\f[V]-AMT\f[R] +.IP \[bu] 2 +\f[B]If an amount value has two minus signs (or two sets of parentheses, +or a minus sign and parentheses):\f[R] +.PD 0 +.P +.PD +they cancel out and will be removed: \f[V]--AMT\f[R] or \f[V]-(AMT)\f[R] +becomes \f[V]AMT\f[R] +.IP \[bu] 2 +\f[B]If an amount value contains just a sign (or just a set of +parentheses):\f[R] +.PD 0 +.P +.PD +that is removed, making it an empty value. +\f[V]\[dq]+\[dq]\f[R] or \f[V]\[dq]-\[dq]\f[R] or \f[V]\[dq]()\[dq]\f[R] +becomes \f[V]\[dq]\[dq]\f[R]. +.SS Setting currency/commodity +.PP +If the currency/commodity symbol is included in the CSV\[aq]s amount +field(s): +.IP +.nf +\f[C] +2020-01-01,foo,$123.00 +\f[R] +.fi +.PP +you don\[aq]t have to do anything special for the commodity symbol, it +will be assigned as part of the amount. +Eg: +.IP +.nf +\f[C] +fields date,description,amount +\f[R] +.fi +.IP +.nf +\f[C] +2020-01-01 foo + expenses:unknown $123.00 + income:unknown $-123.00 +\f[R] +.fi +.PP +If the currency is provided as a separate CSV field: +.IP +.nf +\f[C] +2020-01-01,foo,USD,123.00 +\f[R] +.fi +.PP +You can assign that to the \f[V]currency\f[R] pseudo-field, which has +the special effect of prepending itself to every amount in the +transaction (on the left, with no separating space): +.IP +.nf +\f[C] +fields date,description,currency,amount +\f[R] +.fi +.IP +.nf +\f[C] +2020-01-01 foo + expenses:unknown USD123.00 + income:unknown USD-123.00 +\f[R] +.fi +.PP +Or, you can use a field assignment to construct the amount yourself, +with more control. +Eg to put the symbol on the right, and separated by a space: +.IP +.nf +\f[C] +fields date,description,cur,amt +amount %amt %cur +\f[R] +.fi +.IP +.nf +\f[C] +2020-01-01 foo + expenses:unknown 123.00 USD + income:unknown -123.00 USD +\f[R] +.fi +.PP +Note we used a temporary field name (\f[V]cur\f[R]) that is not +\f[V]currency\f[R] - that would trigger the prepending effect, which we +don\[aq]t want here. +.SS Amount decimal places +.PP +Like amounts in a journal file, the amounts generated by CSV rules like +\f[V]amount1\f[R] influence commodity display styles, such as the number +of decimal places displayed in reports. +.PP +The original amounts as written in the CSV file do not affect display +style (because we don\[aq]t yet reliably know their commodity). +.SS Referencing other fields +.PP +In field assignments, you can interpolate only CSV fields, not hledger +fields. +In the example below, there\[aq]s both a CSV field and a hledger field +named amount1, but %amount1 always means the CSV field, not the hledger +field: +.IP +.nf +\f[C] +# Name the third CSV field \[dq]amount1\[dq] +fields date,description,amount1 + +# Set hledger\[aq]s amount1 to the CSV amount1 field followed by USD +amount1 %amount1 USD + +# Set comment to the CSV amount1 (not the amount1 assigned above) +comment %amount1 +\f[R] +.fi +.PP +Here, since there\[aq]s no CSV amount1 field, %amount1 will produce a +literal \[dq]amount1\[dq]: +.IP +.nf +\f[C] +fields date,description,csvamount +amount1 %csvamount USD +# Can\[aq]t interpolate amount1 here +comment %amount1 +\f[R] +.fi +.PP +When there are multiple field assignments to the same hledger field, +only the last one takes effect. +Here, comment\[aq]s value will be be B, or C if \[dq]something\[dq] is +matched, but never A: +.IP +.nf +\f[C] +comment A +comment B +if something + comment C +\f[R] +.fi +.SS How CSV rules are evaluated +.PP +Here\[aq]s how to think of CSV rules being evaluated (if you really need +to). +First, +.IP \[bu] 2 +\f[V]include\f[R] - all includes are inlined, from top to bottom, depth +first. +(At each include point the file is inlined and scanned for further +includes, recursively, before proceeding.) +.PP +Then \[dq]global\[dq] rules are evaluated, top to bottom. +If a rule is repeated, the last one wins: +.IP \[bu] 2 +\f[V]skip\f[R] (at top level) +.IP \[bu] 2 +\f[V]date-format\f[R] +.IP \[bu] 2 +\f[V]newest-first\f[R] +.IP \[bu] 2 +\f[V]fields\f[R] - names the CSV fields, optionally sets up initial +assignments to hledger fields +.PP +Then for each CSV record in turn: +.IP \[bu] 2 +test all \f[V]if\f[R] blocks. +If any of them contain a \f[V]end\f[R] rule, skip all remaining CSV +records. +Otherwise if any of them contain a \f[V]skip\f[R] rule, skip that many +CSV records. +If there are multiple matched \f[V]skip\f[R] rules, the first one wins. +.IP \[bu] 2 +collect all field assignments at top level and in matched \f[V]if\f[R] +blocks. +When there are multiple assignments for a field, keep only the last one. +.IP \[bu] 2 +compute a value for each hledger field - either the one that was +assigned to it (and interpolate the %CSVFIELD references), or a default +.IP \[bu] 2 +generate a hledger transaction (journal entry) from these values. +.PP +This is all part of the CSV reader, one of several readers hledger can +use to parse input files. +When all files have been read successfully, the transactions are passed +as input to whichever hledger command the user specified. +.PP +.SS Well factored rules +.PP +Some things than can help reduce duplication and complexity in rules +files: +.IP \[bu] 2 +Extracting common rules usable with multiple CSV files into a +\f[V]common.rules\f[R], and adding \f[V]include common.rules\f[R] to +each CSV\[aq]s rules file. +.IP \[bu] 2 +Splitting if blocks into smaller if blocks, extracting the frequently +used parts. +.SS CSV rules examples .SS Bank of Ireland .PP Here\[aq]s a CSV with two amount fields (Debit and Credit), and a @@ -3959,6 +5261,41 @@ $ hledger -f bankofireland-checking.csv print The balance assertions don\[aq]t raise an error above, because we\[aq]re reading directly from CSV, but they will be checked if these entries are imported into a journal file. +.SS Coinbase +.PP +A simple example with some CSV from Coinbase. +The spot price is recorded using cost notation. +The legacy \f[V]amount\f[R] field name conveniently sets amount 2 +(posting 2\[aq]s amount) to the total cost. +.IP +.nf +\f[C] +# Timestamp,Transaction Type,Asset,Quantity Transacted,Spot Price Currency,Spot Price at Transaction,Subtotal,Total (inclusive of fees and/or spread),Fees and/or Spread,Notes +# 2021-12-30T06:57:59Z,Receive,USDC,100,GBP,0.740000,\[dq]\[dq],\[dq]\[dq],\[dq]\[dq],\[dq]Received 100.00 USDC from an external account\[dq] +\f[R] +.fi +.IP +.nf +\f[C] +# coinbase.csv.rules +skip 1 +fields Timestamp,Transaction_Type,Asset,Quantity_Transacted,Spot_Price_Currency,Spot_Price_at_Transaction,Subtotal,Total,Fees_Spread,Notes +date %Timestamp +date-format %Y-%m-%dT%T%Z +description %Notes +account1 assets:coinbase:cc +amount %Quantity_Transacted %Asset \[at] %Spot_Price_at_Transaction %Spot_Price_Currency +\f[R] +.fi +.IP +.nf +\f[C] +$ hledger print -f coinbase.csv +2021-12-30 Received 100.00 USDC from an external account + assets:coinbase:cc 100 USDC \[at] 0.740000 GBP + income:unknown -74.000000 GBP +\f[R] +.fi .SS Amazon .PP Here we convert amazon.com order history, and use an if block to @@ -4190,1196 +5527,6 @@ $ hledger -f paypal-custom.csv print expenses:banking:paypal $0.59 ; business: \f[R] .fi -.PP -.SS CSV rules -.PP -The following kinds of rule can appear in the rules file, in any order. -Blank lines and lines beginning with \f[V]#\f[R] or \f[V];\f[R] or -\f[V]*\f[R] are ignored. -.PP -.TS -tab(@); -lw(24.9n) lw(45.1n). -T{ -\f[B]\f[VB]separator\f[B]\f[R] -T}@T{ -a custom field separator -T} -T{ -\f[B]\f[VB]skip\f[B]\f[R] -T}@T{ -skip one or more header lines or matched CSV records -T} -T{ -\f[B]\f[VB]date-format\f[B]\f[R] -T}@T{ -how to parse dates in CSV records -T} -T{ -\f[B]\f[VB]timezone\f[B]\f[R] -T}@T{ -declare the time zone of ambiguous CSV date-times -T} -T{ -\f[B]\f[VB]decimal-mark\f[B]\f[R] -T}@T{ -the decimal mark used in CSV amounts, if ambiguous -T} -T{ -\f[B]\f[VB]newest-first\f[B]\f[R] -T}@T{ -improve txn order when there are multiple records, newest first, all -with the same date -T} -T{ -\f[B]\f[VB]intra-day-reversed\f[B]\f[R] -T}@T{ -improve txn order when each day\[aq]s txns are reverse of the overall -date order -T} -T{ -\f[B]\f[VB]balance-type\f[B]\f[R] -T}@T{ -choose which type of balance assignments to use -T} -T{ -\f[B]\f[VB]fields\f[B]\f[R] -T}@T{ -name CSV fields, assign them to hledger fields -T} -T{ -\f[B]Field assignment\f[R] -T}@T{ -assign a value to one hledger field, with interpolation -T} -T{ -\f[B]Field names\f[R] -T}@T{ -hledger field names, used in the fields list and field assignments -T} -T{ -\f[B]\f[VB]if\f[B]\f[R] -T}@T{ -apply some rules to CSV records matched by patterns -T} -T{ -\f[B]\f[VB]if\f[B] table\f[R] -T}@T{ -apply some rules to CSV records matched by patterns, alternate syntax -T} -T{ -\f[B]\f[VB]end\f[B]\f[R] -T}@T{ -skip the remaining CSV records -T} -T{ -\f[B]\f[VB]include\f[B]\f[R] -T}@T{ -inline another CSV rules file -T} -.TE -.SS \f[V]separator\f[R] -.PP -You can use the \f[V]separator\f[R] rule to read other kinds of -character-separated data. -The argument is any single separator character, or the words -\f[V]tab\f[R] or \f[V]space\f[R] (case insensitive). -Eg, for comma-separated values (CSV): -.IP -.nf -\f[C] -separator , -\f[R] -.fi -.PP -or for semicolon-separated values (SSV): -.IP -.nf -\f[C] -separator ; -\f[R] -.fi -.PP -or for tab-separated values (TSV): -.IP -.nf -\f[C] -separator TAB -\f[R] -.fi -.PP -If the input file has a \f[V].csv\f[R], \f[V].ssv\f[R] or \f[V].tsv\f[R] -file extension (or a \f[V]csv:\f[R], \f[V]ssv:\f[R], \f[V]tsv:\f[R] -prefix), the appropriate separator will be inferred automatically, and -you won\[aq]t need this rule. -.SS \f[V]skip\f[R] -.IP -.nf -\f[C] -skip N -\f[R] -.fi -.PP -The word \f[V]skip\f[R] followed by a number (or no number, meaning 1) -tells hledger to ignore this many non-empty lines at the start of the -input data. -(Empty/blank lines are skipped automatically, so you don\[aq]t need to -count those.) -You\[aq]ll need this whenever your CSV data contains header lines. -Header lines skipped in this way are ignored, and not parsed as CSV. -.PP -\f[V]skip\f[R] can also be used inside if blocks (described below), to -skip individual data records. -Note records skipped in this way are still required to be valid CSV, -even though otherwise ignored. -.SS \f[V]date-format\f[R] -.IP -.nf -\f[C] -date-format DATEFMT -\f[R] -.fi -.PP -This is a helper for the \f[V]date\f[R] (and \f[V]date2\f[R]) fields. -If your CSV dates are not formatted like \f[V]YYYY-MM-DD\f[R], -\f[V]YYYY/MM/DD\f[R] or \f[V]YYYY.MM.DD\f[R], you\[aq]ll need to add a -date-format rule describing them with a strptime-style date parsing -pattern - see -https://hackage.haskell.org/package/time/docs/Data-Time-Format.html#v:formatTime. -The pattern must parse the CSV date value completely. -Some examples: -.IP -.nf -\f[C] -# MM/DD/YY -date-format %m/%d/%y -\f[R] -.fi -.IP -.nf -\f[C] -# D/M/YYYY -# The - makes leading zeros optional. -date-format %-d/%-m/%Y -\f[R] -.fi -.IP -.nf -\f[C] -# YYYY-Mmm-DD -date-format %Y-%h-%d -\f[R] -.fi -.IP -.nf -\f[C] -# M/D/YYYY HH:MM AM some other junk -# Note the time and junk must be fully parsed, though only the date is used. -date-format %-m/%-d/%Y %l:%M %p some other junk -\f[R] -.fi -.SS \f[V]timezone\f[R] -.IP -.nf -\f[C] -timezone TIMEZONE -\f[R] -.fi -.PP -When CSV contains date-times that are implicitly in some time zone other -than yours, but containing no explicit time zone information, you can -use this rule to declare the CSV\[aq]s native time zone, which helps -prevent off-by-one dates. -.PP -When the CSV date-times do contain time zone information, you don\[aq]t -need this rule; instead, use \f[V]%Z\f[R] in \f[V]date-format\f[R] (or -\f[V]%z\f[R], \f[V]%EZ\f[R], \f[V]%Ez\f[R]; see the formatTime link -above). -.PP -In either of these cases, hledger will do a time-zone-aware conversion, -localising the CSV date-times to your current system time zone. -If you prefer to localise to some other time zone, eg for -reproducibility, you can (on unix at least) set the output timezone with -the TZ environment variable, eg: -.IP -.nf -\f[C] -$ TZ=-1000 hledger print -f foo.csv # or TZ=-1000 hledger import foo.csv -\f[R] -.fi -.PP -\f[V]timezone\f[R] currently does not understand timezone names, except -\[dq]UTC\[dq], \[dq]GMT\[dq], \[dq]EST\[dq], \[dq]EDT\[dq], -\[dq]CST\[dq], \[dq]CDT\[dq], \[dq]MST\[dq], \[dq]MDT\[dq], -\[dq]PST\[dq], or \[dq]PDT\[dq]. -For others, use numeric format: +HHMM or -HHMM. -.SS \f[V]decimal-mark\f[R] -.IP -.nf -\f[C] -decimal-mark . -\f[R] -.fi -.PP -or: -.IP -.nf -\f[C] -decimal-mark , -\f[R] -.fi -.PP -hledger automatically accepts either period or comma as a decimal mark -when parsing numbers (cf Amounts). -However if any numbers in the CSV contain digit group marks, such as -thousand-separating commas, you should declare the decimal mark -explicitly with this rule, to avoid misparsed numbers. -.SS \f[V]newest-first\f[R] -.PP -hledger tries to ensure that the generated transactions will be ordered -chronologically, including intra-day transactions. -Usually it can auto-detect how the CSV records are ordered. -But if it encounters CSV where all records are on the same date, it -assumes that the records are oldest first. -If in fact the CSV\[aq]s records are normally newest first, like: -.IP -.nf -\f[C] -2022-10-01, txn 3... -2022-10-01, txn 2... -2022-10-01, txn 1... -\f[R] -.fi -.PP -you can add the \f[V]newest-first\f[R] rule to help hledger generate the -transactions in correct order. -.IP -.nf -\f[C] -# same-day CSV records are newest first -newest-first -\f[R] -.fi -.SS \f[V]intra-day-reversed\f[R] -.PP -CSV records for each day are sometimes ordered in reverse compared to -the overall date order. -Eg, here dates are newest first, but the transactions on each date are -oldest first: -.IP -.nf -\f[C] -2022-10-02, txn 3... -2022-10-02, txn 4... -2022-10-01, txn 1... -2022-10-01, txn 2... -\f[R] -.fi -.PP -In this situation, add the \f[V]intra-day-reversed\f[R] rule, and -hledger will compensate, improving the order of transactions. -.IP -.nf -\f[C] -# transactions within each day are reversed with respect to the overall date order -intra-day-reversed -\f[R] -.fi -.SS \f[V]fields\f[R] -.IP -.nf -\f[C] -fields FIELDNAME1, FIELDNAME2, ... -\f[R] -.fi -.PP -A fields list (the word \[dq]fields\[dq] followed by comma-separated -field names) is the quick way to assign CSV field values to hledger -fields. -(The other way is field assignments, see below.) -A fields list does does two things: -.IP "1." 3 -It names the CSV fields. -This is optional, but can be convenient later for interpolating them. -.IP "2." 3 -Whenever you use a standard hledger field name (defined below), the CSV -value is assigned to that part of the hledger transaction. -.PP -Here\[aq]s an example that says \[dq]use the 1st, 2nd and 4th fields as -the transaction\[aq]s date, description and amount; name the last two -fields for later reference; and ignore the others\[dq]: -.IP -.nf -\f[C] -fields date, description, , amount, , , somefield, anotherfield -\f[R] -.fi -.PP -Tips: -.IP \[bu] 2 -The fields list always use commas, even if your CSV data uses another -separator character. -.IP \[bu] 2 -Currently there must be least two items in the list (at least one -comma). -.IP \[bu] 2 -Field names may not contain spaces. -Spaces before/after field names are optional. -.IP \[bu] 2 -Field names may contain \f[V]_\f[R] (underscore) or \f[V]-\f[R] -(hyphen). -.IP \[bu] 2 -If the CSV contains column headings, it\[aq]s a good idea to use these, -suitably modified, as the basis for your field names (eg lower-cased, -with underscores instead of spaces). -.IP \[bu] 2 -If some heading names match standard hledger fields, but you don\[aq]t -want to set the hledger fields directly, alter those names, eg by -appending an underscore. -.IP \[bu] 2 -Fields you don\[aq]t care about can be given a dummy name (eg: -\f[V]_\f[R] ), or no name. -.SS Field assignment -.IP -.nf -\f[C] -HLEDGERFIELDNAME FIELDVALUE -\f[R] -.fi -.PP -Field assignments are the more flexible way to assign CSV values to -hledger fields. -They can be used instead of or in addition to a fields list (see above). -.PP -To assign a value to a hledger field, write the field name (any of the -standard hledger field/pseudo-field names, defined below), a space, -followed by a text value on the same line. -This text value may interpolate CSV fields, referenced by their 1-based -position in the CSV record (\f[V]%N\f[R]), or by the name they were -given in the fields list (\f[V]%CSVFIELDNAME\f[R]). -.PP -Some examples: -.IP -.nf -\f[C] -# set the amount to the 4th CSV field, with \[dq] USD\[dq] appended -amount %4 USD - -# combine three fields to make a comment, containing note: and date: tags -comment note: %somefield - %anotherfield, date: %1 -\f[R] -.fi -.PP -Tips: -.IP \[bu] 2 -Interpolation strips outer whitespace (so a CSV value like -\f[V]\[dq] 1 \[dq]\f[R] becomes \f[V]1\f[R] when interpolated) (#1051). -.IP \[bu] 2 -Interpolations always refer to a CSV field - you can\[aq]t interpolate a -hledger field. -(See Referencing other fields below). -.SS Field names -.PP -Here are the standard hledger field (and pseudo-field) names, which you -can use in a fields list or in field assignments. -For more about the transaction parts they refer to, see Transactions. -.SS date field -.PP -Assigning to \f[V]date\f[R] sets the transaction date. -.SS date2 field -.PP -\f[V]date2\f[R] sets the transaction\[aq]s secondary date, if any. -.SS status field -.PP -\f[V]status\f[R] sets the transaction\[aq]s status, if any. -.SS code field -.PP -\f[V]code\f[R] sets the transaction\[aq]s code, if any. -.SS description field -.PP -\f[V]description\f[R] sets the transaction\[aq]s description, if any. -.SS comment field -.PP -\f[V]comment\f[R] sets the transaction\[aq]s comment, if any. -.PP -\f[V]commentN\f[R], where N is a number, sets the Nth posting\[aq]s -comment. -.PP -You can assign multi-line comments by writing literal \f[V]\[rs]n\f[R] -in the code. -A comment starting with \f[V]\[rs]n\f[R] will begin on a new line. -.PP -Comments can contain tags, as usual. -.SS account field -.PP -Assigning to \f[V]accountN\f[R], where N is 1 to 99, sets the account -name of the Nth posting, and causes that posting to be generated. -.PP -Most often there are two postings, so you\[aq]ll want to set -\f[V]account1\f[R] and \f[V]account2\f[R]. -Typically \f[V]account1\f[R] is associated with the CSV file, and is set -once with a top-level assignment, while \f[V]account2\f[R] is set based -on each transaction\[aq]s description, and in conditional blocks. -.PP -If a posting\[aq]s account name is left unset but its amount is set (see -below), a default account name will be chosen (like -\[dq]expenses:unknown\[dq] or \[dq]income:unknown\[dq]). -.SS amount field -.PP -\f[V]amountN\f[R] sets the amount of the Nth posting, and causes that -posting to be generated. -By assigning to \f[V]amount1\f[R], \f[V]amount2\f[R], ... -etc. -you can generate up to 99 postings. -.PP -\f[V]amountN-in\f[R] and \f[V]amountN-out\f[R] can be used instead, if -the CSV uses separate fields for debits and credits (inflows and -outflows). -hledger assumes both of these CSV fields are unsigned, and will -automatically negate the \[dq]-out\[dq] value. -It also requires that at least one of them is either empty or zero. -See \[dq]Setting amounts\[dq] below for more on this topic. -.PP -\f[V]amount\f[R], or \f[V]amount-in\f[R] and \f[V]amount-out\f[R] are a -legacy mode, to keep pre-hledger-1.17 CSV rules files working (and for -occasional convenience). -They are suitable only for two-posting transactions; they set both -posting 1\[aq]s and posting 2\[aq]s amount. -Posting 2\[aq]s amount will be negated, and also converted to cost if -there\[aq]s a cost price. -.PP -Note: it might sound as if amount-in is for one posting and amount-out -for the other posting, but no; use the -in and -out rules together for -the same posting, producing one amount from two CSV fields. -.PP -If you have an existing rules file using the unnumbered form, you might -want to use the numbered form in certain conditional blocks, without -having to update and retest all the old rules. -To facilitate this, posting 1 ignores -\f[V]amount\f[R]/\f[V]amount-in\f[R]/\f[V]amount-out\f[R] if any of -\f[V]amount1\f[R]/\f[V]amount1-in\f[R]/\f[V]amount1-out\f[R] are -assigned, and posting 2 ignores them if any of -\f[V]amount2\f[R]/\f[V]amount2-in\f[R]/\f[V]amount2-out\f[R] are -assigned, avoiding conflicts. -.SS currency field -.PP -\f[V]currency\f[R] sets a currency symbol, to be prepended to all -postings\[aq] amounts. -You can use this if the CSV amounts do not have a currency symbol, eg if -it is in a separate column. -.PP -\f[V]currencyN\f[R] prepends a currency symbol to just the Nth -posting\[aq]s amount. -.SS balance field -.PP -\f[V]balanceN\f[R] sets a balance assertion amount (or if the posting -amount is left empty, a balance assignment) on posting N. -.PP -\f[V]balance\f[R] is a compatibility spelling for hledger <1.17; it is -equivalent to \f[V]balance1\f[R]. -.PP -You can adjust the type of assertion/assignment with the -\f[V]balance-type\f[R] rule (see below). -.PP -See Tips below for more about setting amounts and currency. -.SS \f[V]if\f[R] -.IP -.nf -\f[C] -if MATCHER - RULE - -if -MATCHER -MATCHER -MATCHER - RULE - RULE -\f[R] -.fi -.PP -Conditional blocks (\[dq]if blocks\[dq]) are a block of rules that are -applied only to CSV records which match certain patterns. -They are often used for customising account names based on transaction -descriptions. -.SS Matching the whole record -.PP -Each MATCHER can be a record matcher, which looks like this: -.IP -.nf -\f[C] -REGEX -\f[R] -.fi -.PP -REGEX is a case-insensitive regular expression that tries to match -anywhere within the CSV record. -It is a POSIX ERE (extended regular expression) that also supports GNU -word boundaries (\f[V]\[rs]b\f[R], \f[V]\[rs]B\f[R], \f[V]\[rs]<\f[R], -\f[V]\[rs]>\f[R]), and nothing else. -If you have trouble, be sure to check our doc: -https://hledger.org/hledger.html#regular-expressions -.PP -Important note: the record that is matched is not the original record, -but a synthetic one, with any enclosing double quotes (but not enclosing -whitespace) removed, and always comma-separated (which means that a -field containing a comma will appear like two fields). -Eg, if the original record is -\f[V]2020-01-01; \[dq]Acme, Inc.\[dq]; 1,000\f[R], the REGEX will -actually see \f[V]2020-01-01,Acme, Inc., 1,000\f[R]). -.SS Matching individual fields -.PP -Or, MATCHER can be a field matcher, like this: -.IP -.nf -\f[C] -%CSVFIELD REGEX -\f[R] -.fi -.PP -which matches just the content of a particular CSV field. -CSVFIELD is a percent sign followed by the field\[aq]s name or column -number, like \f[V]%date\f[R] or \f[V]%1\f[R]. -.SS Combining matchers -.PP -A single matcher can be written on the same line as the \[dq]if\[dq]; or -multiple matchers can be written on the following lines, non-indented. -Multiple matchers are OR\[aq]d (any one of them can match), unless one -begins with an \f[V]&\f[R] symbol, in which case it is AND\[aq]ed with -the previous matcher. -.IP -.nf -\f[C] -if -MATCHER -& MATCHER - RULE -\f[R] -.fi -.SS Rules applied on successful match -.PP -After the patterns there should be one or more rules to apply, all -indented by at least one space. -Three kinds of rule are allowed in conditional blocks: -.IP \[bu] 2 -field assignments (to set a hledger field) -.IP \[bu] 2 -skip (to skip the matched CSV record) -.IP \[bu] 2 -end (to skip all remaining CSV records). -.PP -Examples: -.IP -.nf -\f[C] -# if the CSV record contains \[dq]groceries\[dq], set account2 to \[dq]expenses:groceries\[dq] -if groceries - account2 expenses:groceries -\f[R] -.fi -.IP -.nf -\f[C] -# if the CSV record contains any of these patterns, set account2 and comment as shown -if -monthly service fee -atm transaction fee -banking thru software - account2 expenses:business:banking - comment XXX deductible ? check it -\f[R] -.fi -.SS \f[V]if\f[R] table -.IP -.nf -\f[C] -if,CSVFIELDNAME1,CSVFIELDNAME2,...,CSVFIELDNAMEn -MATCHER1,VALUE11,VALUE12,...,VALUE1n -MATCHER2,VALUE21,VALUE22,...,VALUE2n -MATCHER3,VALUE31,VALUE32,...,VALUE3n - -\f[R] -.fi -.PP -Conditional tables (\[dq]if tables\[dq]) are a different syntax to -specify field assignments that will be applied only to CSV records which -match certain patterns. -.PP -MATCHER could be either field or record matcher, as described above. -When MATCHER matches, values from that row would be assigned to the CSV -fields named on the \f[V]if\f[R] line, in the same order. -.PP -Therefore \f[V]if\f[R] table is exactly equivalent to a sequence of of -\f[V]if\f[R] blocks: -.IP -.nf -\f[C] -if MATCHER1 - CSVFIELDNAME1 VALUE11 - CSVFIELDNAME2 VALUE12 - ... - CSVFIELDNAMEn VALUE1n - -if MATCHER2 - CSVFIELDNAME1 VALUE21 - CSVFIELDNAME2 VALUE22 - ... - CSVFIELDNAMEn VALUE2n - -if MATCHER3 - CSVFIELDNAME1 VALUE31 - CSVFIELDNAME2 VALUE32 - ... - CSVFIELDNAMEn VALUE3n -\f[R] -.fi -.PP -Each line starting with MATCHER should contain enough (possibly empty) -values for all the listed fields. -.PP -Rules would be checked and applied in the order they are listed in the -table and, like with \f[V]if\f[R] blocks, later rules (in the same or -another table) or \f[V]if\f[R] blocks could override the effect of any -rule. -.PP -Instead of \[aq],\[aq] you can use a variety of other non-alphanumeric -characters as a separator. -First character after \f[V]if\f[R] is taken to be the separator for the -rest of the table. -It is the responsibility of the user to ensure that separator does not -occur inside MATCHERs and values - there is no way to escape separator. -.PP -Example: -.IP -.nf -\f[C] -if,account2,comment -atm transaction fee,expenses:business:banking,deductible? check it -%description groceries,expenses:groceries, -2020/01/12.*Plumbing LLC,expenses:house:upkeep,emergency plumbing call-out -\f[R] -.fi -.SS \f[V]end\f[R] -.PP -This rule can be used inside if blocks (only), to make hledger stop -reading this CSV file and move on to the next input file, or to command -execution. -Eg: -.IP -.nf -\f[C] -# ignore everything following the first empty record -if ,,,, - end -\f[R] -.fi -.SS \f[V]include\f[R] -.IP -.nf -\f[C] -include RULESFILE -\f[R] -.fi -.PP -This includes the contents of another CSV rules file at this point. -\f[V]RULESFILE\f[R] is an absolute file path or a path relative to the -current file\[aq]s directory. -This can be useful for sharing common rules between several rules files, -eg: -.IP -.nf -\f[C] -# someaccount.csv.rules - -## someaccount-specific rules -fields date,description,amount -account1 assets:someaccount -account2 expenses:misc - -## common rules -include categorisation.rules -\f[R] -.fi -.SS \f[V]balance-type\f[R] -.PP -Balance assertions generated by assigning to balanceN are of the simple -\f[V]=\f[R] type by default, which is a single-commodity, -subaccount-excluding assertion. -You may find the subaccount-including variants more useful, eg if you -have created some virtual subaccounts of checking to help with -budgeting. -You can select a different type of assertion with the -\f[V]balance-type\f[R] rule: -.IP -.nf -\f[C] -# balance assertions will consider all commodities and all subaccounts -balance-type ==* -\f[R] -.fi -.PP -Here are the balance assertion types for quick reference: -.IP -.nf -\f[C] -= single commodity, exclude subaccounts -=* single commodity, include subaccounts -== multi commodity, exclude subaccounts -==* multi commodity, include subaccounts -\f[R] -.fi -.SS Tips -.SS Rapid feedback -.PP -It\[aq]s a good idea to get rapid feedback while -creating/troubleshooting CSV rules. -Here\[aq]s a good way, using entr from eradman.com/entrproject: -.IP -.nf -\f[C] -$ ls foo.csv* | entr bash -c \[aq]echo ----; hledger -f foo.csv print desc:SOMEDESC\[aq] -\f[R] -.fi -.PP -A desc: query (eg) is used to select just one, or a few, transactions of -interest. -\[dq]bash -c\[dq] is used to run multiple commands, so we can echo a -separator each time the command re-runs, making it easier to read the -output. -.SS Valid CSV -.PP -Note that hledger will only accept valid CSV conforming to RFC 4180, and -equivalent SSV and TSV formats (like RFC 4180 but with semicolon or tab -as separators). -This means, eg: -.IP \[bu] 2 -Values may be enclosed in double quotes, or not. -Enclosing in single quotes is not allowed. -(Eg \f[V]\[aq]A\[aq],\[aq]B\[aq]\f[R] is rejected.) -.IP \[bu] 2 -When values are enclosed in double quotes, spaces outside the quotes are -not allowed. -(Eg \f[V]\[dq]A\[dq], \[dq]B\[dq]\f[R] is rejected.) -.IP \[bu] 2 -When values are not enclosed in quotes, they may not contain double -quotes. -(Eg \f[V]A\[dq]A, B\f[R] is rejected.) -.PP -If your CSV/SSV/TSV is not valid in this sense, you\[aq]ll need to -transform it before reading with hledger. -Try using sed, or a more permissive CSV parser like python\[aq]s csv -lib. -.SS File Extension -.PP -To help hledger identify the format and show the right error messages, -CSV/SSV/TSV files should normally be named with a \f[V].csv\f[R], -\f[V].ssv\f[R] or \f[V].tsv\f[R] filename extension. -Or, the file path should be prefixed with \f[V]csv:\f[R], \f[V]ssv:\f[R] -or \f[V]tsv:\f[R]. -Eg: -.IP -.nf -\f[C] -$ hledger -f foo.ssv print -\f[R] -.fi -.PP -or: -.IP -.nf -\f[C] -$ cat foo | hledger -f ssv:- foo -\f[R] -.fi -.PP -You can override the file extension with a separator rule if needed. -See also: Input files in the hledger manual. -.SS Reading multiple CSV files -.PP -If you use multiple \f[V]-f\f[R] options to read multiple CSV files at -once, hledger will look for a correspondingly-named rules file for each -CSV file. -But if you use the \f[V]--rules-file\f[R] option, that rules file will -be used for all the CSV files. -.SS Valid transactions -.PP -After reading a CSV file, hledger post-processes and validates the -generated journal entries as it would for a journal file - balancing -them, applying balance assignments, and canonicalising amount styles. -Any errors at this stage will be reported in the usual way, displaying -the problem entry. -.PP -There is one exception: balance assertions, if you have generated them, -will not be checked, since normally these will work only when the CSV -data is part of the main journal. -If you do need to check balance assertions generated from CSV right -away, pipe into another hledger: -.IP -.nf -\f[C] -$ hledger -f file.csv print | hledger -f- print -\f[R] -.fi -.SS Deduplicating, importing -.PP -When you download a CSV file periodically, eg to get your latest bank -transactions, the new file may overlap with the old one, containing some -of the same records. -.PP -The import command will (a) detect the new transactions, and (b) append -just those transactions to your main journal. -It is idempotent, so you don\[aq]t have to remember how many times you -ran it or with which version of the CSV. -(It keeps state in a hidden \f[V].latest.FILE.csv\f[R] file.) -This is the easiest way to import CSV data. -Eg: -.IP -.nf -\f[C] -# download the latest CSV files, then run this command. -# Note, no -f flags needed here. -$ hledger import *.csv [--dry] -\f[R] -.fi -.PP -This method works for most CSV files. -(Where records have a stable chronological order, and new records appear -only at the new end.) -.PP -A number of other tools and workflows, hledger-specific and otherwise, -exist for converting, deduplicating, classifying and managing CSV data. -See: -.IP \[bu] 2 -https://hledger.org/cookbook.html#setups-and-workflows -.IP \[bu] 2 -https://plaintextaccounting.org -> data import/conversion -.SS Setting amounts -.PP -Some tips on using the amount-setting rules discussed above. -.PP -Here are the ways to set a posting\[aq]s amount: -.IP "1." 3 -\f[B]If the CSV has a single amount field:\f[R] -.PD 0 -.P -.PD -Assign (via a fields list or a field assignment) to \f[V]amountN\f[R]. -This sets the Nth posting\[aq]s amount. -N is usually 1 or 2 but can go up to 99. -.IP "2." 3 -\f[B]If the CSV has separate amount fields for debit & credit (in & -out):\f[R] -.PD 0 -.P -.PD -.RS 4 -.IP "a." 3 -\f[B]If both fields are unsigned:\f[R] -.PD 0 -.P -.PD -Assign to \f[V]amountN-in\f[R] and \f[V]amountN-out\f[R]. -This sets posting N\[aq]s amount to whichever of these has a non-zero -value, and negates the \[dq]-out\[dq] value. -.IP "b." 3 -\f[B]If either field is signed (can contain a minus sign):\f[R] -.PD 0 -.P -.PD -Use a conditional rule to flip the sign (of non-empty values). -Since hledger always negates amountN-out, if it was already negative, we -must undo that by negating once more (but only if the field is -non-empty): -.IP -.nf -\f[C] -fields date, description, amount1-in, amount1-out -if %amount1-out [1-9] - amount1-out -%amount1-out -\f[R] -.fi -.IP "c." 3 -\f[B]If both fields, or neither field, can contain a non-zero -value:\f[R] -.PD 0 -.P -.PD -hledger normally expects exactly one of the fields to have a non-zero -value. -Eg, the \f[V]amountN-in\f[R]/\f[V]amountN-out\f[R] rules would reject -value pairs like these: -.IP -.nf -\f[C] -\[dq]\[dq], \[dq]\[dq] -\[dq]0\[dq], \[dq]0\[dq] -\[dq]1\[dq], \[dq]none\[dq] -\f[R] -.fi -.PP -So, use smarter conditional rules to set the amount from the appropriate -field. -Eg, these rules would make it use only the value containing non-zero -digits, handling the above: -.IP -.nf -\f[C] -fields date, description, in, out -if %in [1-9] - amount1 %in -if %out [1-9] - amount1 %out -\f[R] -.fi -.RE -.IP "3." 3 -\f[B]If you want posting 2\[aq]s amount converted to cost:\f[R] -.PD 0 -.P -.PD -Assign to \f[V]amount\f[R] (or to \f[V]amount-in\f[R] and -\f[V]amount-out\f[R]). -(This is the legacy numberless syntax, which sets amount1 and amount2 -and converts amount2 to cost.) -.IP "4." 3 -\f[B]If the CSV has the balance instead of the transaction amount:\f[R] -.PD 0 -.P -.PD -Assign to \f[V]balanceN\f[R], which sets posting N\[aq]s amount -indirectly via a balance assignment. -(Old syntax: \f[V]balance\f[R], equivalent to \f[V]balance1\f[R].) -.RS 4 -.IP \[bu] 2 -\f[B]If hledger guesses the wrong default account name:\f[R] -.PD 0 -.P -.PD -When setting the amount via balance assertion, hledger may guess the -wrong default account name. -So, set the account name explicitly, eg: -.RS 2 -.IP -.nf -\f[C] -fields date, description, balance1 -account1 assets:checking -\f[R] -.fi -.RE -.RE -.SS Amount signs -.PP -There is some special handling for amount signs, to simplify parsing and -sign-flipping: -.IP \[bu] 2 -\f[B]If an amount value begins with a plus sign:\f[R] -.PD 0 -.P -.PD -that will be removed: \f[V]+AMT\f[R] becomes \f[V]AMT\f[R] -.IP \[bu] 2 -\f[B]If an amount value is parenthesised:\f[R] -.PD 0 -.P -.PD -it will be de-parenthesised and sign-flipped: \f[V](AMT)\f[R] becomes -\f[V]-AMT\f[R] -.IP \[bu] 2 -\f[B]If an amount value has two minus signs (or two sets of parentheses, -or a minus sign and parentheses):\f[R] -.PD 0 -.P -.PD -they cancel out and will be removed: \f[V]--AMT\f[R] or \f[V]-(AMT)\f[R] -becomes \f[V]AMT\f[R] -.IP \[bu] 2 -\f[B]If an amount value contains just a sign (or just a set of -parentheses):\f[R] -.PD 0 -.P -.PD -that is removed, making it an empty value. -\f[V]\[dq]+\[dq]\f[R] or \f[V]\[dq]-\[dq]\f[R] or \f[V]\[dq]()\[dq]\f[R] -becomes \f[V]\[dq]\[dq]\f[R]. -.SS Setting currency/commodity -.PP -If the currency/commodity symbol is included in the CSV\[aq]s amount -field(s): -.IP -.nf -\f[C] -2020-01-01,foo,$123.00 -\f[R] -.fi -.PP -you don\[aq]t have to do anything special for the commodity symbol, it -will be assigned as part of the amount. -Eg: -.IP -.nf -\f[C] -fields date,description,amount -\f[R] -.fi -.IP -.nf -\f[C] -2020-01-01 foo - expenses:unknown $123.00 - income:unknown $-123.00 -\f[R] -.fi -.PP -If the currency is provided as a separate CSV field: -.IP -.nf -\f[C] -2020-01-01,foo,USD,123.00 -\f[R] -.fi -.PP -You can assign that to the \f[V]currency\f[R] pseudo-field, which has -the special effect of prepending itself to every amount in the -transaction (on the left, with no separating space): -.IP -.nf -\f[C] -fields date,description,currency,amount -\f[R] -.fi -.IP -.nf -\f[C] -2020-01-01 foo - expenses:unknown USD123.00 - income:unknown USD-123.00 -\f[R] -.fi -.PP -Or, you can use a field assignment to construct the amount yourself, -with more control. -Eg to put the symbol on the right, and separated by a space: -.IP -.nf -\f[C] -fields date,description,cur,amt -amount %amt %cur -\f[R] -.fi -.IP -.nf -\f[C] -2020-01-01 foo - expenses:unknown 123.00 USD - income:unknown -123.00 USD -\f[R] -.fi -.PP -Note we used a temporary field name (\f[V]cur\f[R]) that is not -\f[V]currency\f[R] - that would trigger the prepending effect, which we -don\[aq]t want here. -.SS Amount decimal places -.PP -Like amounts in a journal file, the amounts generated by CSV rules like -\f[V]amount1\f[R] influence commodity display styles, such as the number -of decimal places displayed in reports. -.PP -The original amounts as written in the CSV file do not affect display -style (because we don\[aq]t yet reliably know their commodity). -.SS Referencing other fields -.PP -In field assignments, you can interpolate only CSV fields, not hledger -fields. -In the example below, there\[aq]s both a CSV field and a hledger field -named amount1, but %amount1 always means the CSV field, not the hledger -field: -.IP -.nf -\f[C] -# Name the third CSV field \[dq]amount1\[dq] -fields date,description,amount1 - -# Set hledger\[aq]s amount1 to the CSV amount1 field followed by USD -amount1 %amount1 USD - -# Set comment to the CSV amount1 (not the amount1 assigned above) -comment %amount1 -\f[R] -.fi -.PP -Here, since there\[aq]s no CSV amount1 field, %amount1 will produce a -literal \[dq]amount1\[dq]: -.IP -.nf -\f[C] -fields date,description,csvamount -amount1 %csvamount USD -# Can\[aq]t interpolate amount1 here -comment %amount1 -\f[R] -.fi -.PP -When there are multiple field assignments to the same hledger field, -only the last one takes effect. -Here, comment\[aq]s value will be be B, or C if \[dq]something\[dq] is -matched, but never A: -.IP -.nf -\f[C] -comment A -comment B -if something - comment C -\f[R] -.fi -.SS How CSV rules are evaluated -.PP -Here\[aq]s how to think of CSV rules being evaluated (if you really need -to). -First, -.IP \[bu] 2 -\f[V]include\f[R] - all includes are inlined, from top to bottom, depth -first. -(At each include point the file is inlined and scanned for further -includes, recursively, before proceeding.) -.PP -Then \[dq]global\[dq] rules are evaluated, top to bottom. -If a rule is repeated, the last one wins: -.IP \[bu] 2 -\f[V]skip\f[R] (at top level) -.IP \[bu] 2 -\f[V]date-format\f[R] -.IP \[bu] 2 -\f[V]newest-first\f[R] -.IP \[bu] 2 -\f[V]fields\f[R] - names the CSV fields, optionally sets up initial -assignments to hledger fields -.PP -Then for each CSV record in turn: -.IP \[bu] 2 -test all \f[V]if\f[R] blocks. -If any of them contain a \f[V]end\f[R] rule, skip all remaining CSV -records. -Otherwise if any of them contain a \f[V]skip\f[R] rule, skip that many -CSV records. -If there are multiple matched \f[V]skip\f[R] rules, the first one wins. -.IP \[bu] 2 -collect all field assignments at top level and in matched \f[V]if\f[R] -blocks. -When there are multiple assignments for a field, keep only the last one. -.IP \[bu] 2 -compute a value for each hledger field - either the one that was -assigned to it (and interpolate the %CSVFIELDNAME references), or a -default -.IP \[bu] 2 -generate a synthetic hledger transaction from these values. -.PP -This is all part of the CSV reader, one of several readers hledger can -use to parse input files. -When all files have been read successfully, the transactions are passed -as input to whichever hledger command the user specified. -.PP .SH Timeclock .PP The time logging format of timeclock.el, as read by hledger. @@ -6902,6 +7049,8 @@ be named \f[V]equity:conversion\f[R], \f[V]equity:trade\f[R], .IP "4." 3 the equity postings\[aq] amounts must exactly match the non-equity postings\[aq] amounts +.IP "5." 3 +all of the amounts must be explicit, with none missing .PP Multiple such exchanges can coexist within a single transaction, should you need that. @@ -7948,13 +8097,9 @@ prices - show market price records .IP \[bu] 2 \f[B]print\f[R] - show transactions (journal entries) .IP \[bu] 2 -print-unique - show only transactions with unique descriptions -.IP \[bu] 2 \f[B]register (reg)\f[R] - show postings in one or more accounts & running total .IP \[bu] 2 -register-match - show a recent posting that best matches a description -.IP \[bu] 2 stats - show journal statistics .IP \[bu] 2 tags - show tag names @@ -8185,9 +8330,14 @@ for reviewing detailed revenues/expenses. You can write either the full account name, or a case-insensitive regular expression which will select the alphabetically first matched account. -(Eg if you have \f[V]assets:aaa:checking\f[R] and -\f[V]assets:bbb:checking\f[R] accounts, \f[V]hledger areg checking\f[R] -would select \f[V]assets:aaa:checking\f[R].) +.PP +When there are multiple matches, the alphabetically-first choice can be +surprising; eg if you have \f[V]assets:per:checking 1\f[R] and +\f[V]assets:biz:checking 2\f[R] accounts, +\f[V]hledger areg checking\f[R] would select +\f[V]assets:biz:checking 2\f[R]. +It\[aq]s just a convenience to save typing, so if in doubt, write the +full account name, or a distinctive substring that matches uniquely. .PP Transactions involving subaccounts of this account will also be shown. \f[V]aregister\f[R] ignores depth limits, so its final total will always @@ -10556,8 +10706,7 @@ cost using that price. This can be used for troubleshooting. .PP With \f[V]-m DESC\f[R]/\f[V]--match=DESC\f[R], print does a fuzzy search -for the one transaction whose description is most similar to DESC, also -preferring recent tranactions. +for one recent transaction whose description is most similar to DESC. DESC should contain at least two characters. If there is no similar-enough match, no transaction will be shown and the program exit code will be non-zero. @@ -10606,29 +10755,6 @@ The numeric amount is repeated in either the \[dq]credit\[dq] or \[dq]debit\[dq] column, for convenience. (Those names are not accurate in the accounting sense; it just puts negative amounts under credit and zero or greater amounts under debit.) -.SS print-unique -.PP -print-unique -.PD 0 -.P -.PD -Print transactions which do not reuse an already-seen description. -.PP -Example: -.IP -.nf -\f[C] -$ cat unique.journal -1/1 test - (acct:one) 1 -2/2 test - (acct:two) 2 -$ LEDGER_FILE=unique.journal hledger print-unique -(-f option not supported) -2015/01/01 test - (acct:one) 1 -\f[R] -.fi .SS register .PP register, reg @@ -10658,7 +10784,7 @@ $ hledger register checking \f[R] .fi .PP -With --date2, it shows and sorts by secondary date instead. +With \f[V]--date2\f[R], it shows and sorts by secondary date instead. .PP For performance reasons, column widths are chosen based on the first 1000 lines; this means unusually wide values in later lines can cause @@ -10757,6 +10883,12 @@ will be adjusted outward if necessary to contain a whole number of intervals. This ensures that the first and last intervals are full length and comparable to the others in the report. +.PP +With \f[V]-m DESC\f[R]/\f[V]--match=DESC\f[R], register does a fuzzy +search for one recent posting whose description is most similar to DESC. +DESC should contain at least two characters. +If there is no similar-enough match, no posting will be shown and the +program exit code will be non-zero. .SS Custom register output .PP register uses the full terminal width by default, except on windows. @@ -10794,18 +10926,6 @@ $ hledger reg -w $COLUMNS,40 # use terminal width, & description width 40 This command also supports the output destination and output format options The output formats supported are \f[V]txt\f[R], \f[V]csv\f[R], and (experimental) \f[V]json\f[R]. -.SS register-match -.PP -register-match -.PD 0 -.P -.PD -Print the one posting whose transaction description is closest to DESC, -in the style of the register command. -If there are multiple equally good matches, it shows the most recent. -Query options (options, not arguments) can be used to restrict the -search space. -Helps ledger-autosync detect already-seen transactions when importing. .SS rewrite .PP rewrite diff --git a/hledger/hledger.info b/hledger/hledger.info index bd7bfd04c..9c307ed56 100644 --- a/hledger/hledger.info +++ b/hledger/hledger.info @@ -30,8 +30,8 @@ bookkeeping/accounting as well! You don’t need to know everything in here to use hledger productively, but when you have a question about functionality, this doc should answer it. It is detailed, so do skip ahead or skim when needed. You can read it on hledger.org, or as an -info manual or man page on your system (each has benefits). You can -also get it from hledger itself with +info manual or man page on your system. You can also get it from +hledger itself with ‘hledger --man’, ‘hledger --info’ or ‘hledger help [TOPIC]’. The main function of the hledger CLI is to read plain text files @@ -1489,17 +1489,50 @@ File: hledger.info, Node: Account names, Next: Amounts, Prev: Postings, Up: 10.11 Account names =================== -Account names typically have several parts separated by a full colon, -from which hledger derives a hierarchical chart of accounts. They can -be anything you like, but in finance there are traditionally five -top-level accounts: ‘assets’, ‘liabilities’, ‘revenue’, ‘expenses’, and -‘equity’. +Accounts are the main way of categorising things in hledger. As in +Double Entry Bookkeeping, they can represent real world accounts (such +as a bank account), or more abstract categories such as "money borrowed +from Frank" or "money spent on electricity". - Account names may contain single spaces, eg: ‘assets:accounts -receivable’. Because of this, they must always be followed by *two or -more spaces* (or newline). + You can use any account names you like, but we usually start with the +traditional accounting categories, which in english are ‘assets’, +‘liabilities’, ‘equity’, ‘revenues’, ‘expenses’. (You might see these +referred to as A, L, E, R, X for short.) - Account names can be aliased. + For more precise reporting, we usually divide the top level accounts +into more detailed subaccounts, by writing a full colon between account +name parts. For example, from the account names ‘assets:bank:checking’ +and ‘expenses:food’, hledger will infer this hierarchy of five accounts: + +assets +assets:bank +assets:bank:checking +expenses +expenses:food + + Shown as an outline, the hierarchical tree structure is more clear: + +assets + bank + checking +expenses + food + + hledger reports can summarise the account tree to any depth, so you +can go as deep as you like with subcategories, but keeping your account +names relatively simple may be best when starting out. + + Account names may be capitalised or not; they may contain letters, +numbers, symbols, or single spaces. Note, when an account name and an +amount are written on the same line, they must be separated by *two or +more spaces* (or tabs). + + Parentheses or brackets enclosing the full account name indicate +virtual postings, described below. Parentheses or brackets internal to +the account name have no special meaning. + + Account names can be altered temporarily or permanently by account +aliases.  File: hledger.info, Node: Amounts, Next: Costs, Prev: Account names, Up: Journal @@ -2270,6 +2303,12 @@ account name, eg: account assets:bank:checking + Note, however, that accounts declared in account directives are not +allowed to have surrounding brackets and parentheses, unlike accounts +used in postings. So the following journal will not parse: + +account (assets:bank:checking) + * Menu: * Account comments:: @@ -3476,9 +3515,9 @@ each record into a transaction. (To learn about _writing_ CSV, see CSV output.) - Note, for best error messages when reading CSV/TSV/SSV files, make -sure they have a corresponding ‘.csv’, ‘.tsv’ or ‘.ssv’ file extension -or use a hledger file prefix (see File Extension below). + For best error messages when reading CSV/TSV/SSV files, make sure +they have a corresponding ‘.csv’, ‘.tsv’ or ‘.ssv’ file extension or use +a hledger file prefix (see File Extension below). Each CSV file must be described by a corresponding _rules file_. This contains rules describing the CSV data (header line, fields layout, @@ -3492,31 +3531,738 @@ specify a different rules file with the ‘--rules-file’ option. If no rules file is found, hledger will create a sample rules file, which you’ll need to adjust. - There’s an introductory Importing CSV data tutorial on hledger.org. + At minimum, the rules file must identify the date and amount fields, +and often it also specifies the date format and how many header lines +there are. Here’s a simple CSV file and a rules file for it: + +Date, Description, Id, Amount +12/11/2019, Foo, 123, 10.23 + +# basic.csv.rules +skip 1 +fields date, description, , amount +date-format %d/%m/%Y + +$ hledger print -f basic.csv +2019-11-12 Foo + expenses:unknown 10.23 + income:unknown -10.23 + + There’s an introductory Importing CSV data tutorial on hledger.org, +and more CSV rules examples below, and a larger collection at +https://github.com/simonmichael/hledger/tree/master/examples/csv. * Menu: -* Examples:: -* CSV rules:: +* CSV rules cheatsheet:: * separator:: * skip:: * date-format:: * timezone:: -* decimal-mark:: * newest-first:: * intra-day-reversed:: -* fields:: +* decimal-mark:: +* fields list:: * Field assignment:: * Field names:: -* if:: +* if block:: +* Matchers:: * if table:: -* end:: -* include:: * balance-type:: -* Tips:: +* include:: +* Working with CSV:: +* CSV rules examples:: + + +File: hledger.info, Node: CSV rules cheatsheet, Next: separator, Up: CSV + +11.1 CSV rules cheatsheet +========================= + +The following kinds of rule can appear in the rules file, in any order. +(Blank lines and lines beginning with ‘#’ or ‘;’ or ‘*’ are ignored.) + +*‘separator’* declare the field separator, instead of + relying on file extension +*‘skip’* skip one or more header lines at start of file +*‘date-format’* declare how to parse CSV dates/date-times +*‘timezone’* declare the time zone of ambiguous CSV + date-times +*‘newest-first’* improve txn order when: there are multiple + records, newest first, all with the same date +*‘intra-day-reversed’* improve txn order when: same-day txns are in + opposite order to the overall file +*‘decimal-mark’* declare the decimal mark used in CSV amounts, + when ambiguous +*‘fields’ list* name CSV fields for easy reference, and + optionally assign their values to hledger + fields +*Field assignment* assign a CSV value or interpolated text value + to a hledger field +*‘if’ block* conditionally assign values to hledger fields, + or ‘skip’ a record or ‘end’ (skip rest of + file) +*‘if’ table* conditionally assign values to hledger fields, + using compact syntax +*‘balance-type’* select which type of balance + assertions/assignments to generate +*‘include’* inline another CSV rules file + + Working with CSV tips can be found below, including How CSV rules are +evaluated. + + +File: hledger.info, Node: separator, Next: skip, Prev: CSV rules cheatsheet, Up: CSV + +11.2 ‘separator’ +================ + +You can use the ‘separator’ rule to read other kinds of +character-separated data. The argument is any single separator +character, or the words ‘tab’ or ‘space’ (case insensitive). Eg, for +comma-separated values (CSV): + +separator , + + or for semicolon-separated values (SSV): + +separator ; + + or for tab-separated values (TSV): + +separator TAB + + If the input file has a ‘.csv’, ‘.ssv’ or ‘.tsv’ file extension (or a +‘csv:’, ‘ssv:’, ‘tsv:’ prefix), the appropriate separator will be +inferred automatically, and you won’t need this rule. + + +File: hledger.info, Node: skip, Next: date-format, Prev: separator, Up: CSV + +11.3 ‘skip’ +=========== + +skip N + + The word ‘skip’ followed by a number (or no number, meaning 1) tells +hledger to ignore this many non-empty lines at the start of the input +data. (Empty/blank lines are skipped automatically, so you don’t need +to count those.) You’ll need this whenever your CSV data contains +header lines. Header lines skipped in this way are ignored, and not +parsed as CSV. + + ‘skip’ can also be used inside if blocks (described below), to skip +individual data records. Note records skipped in this way are still +required to be valid CSV, even though otherwise ignored. + + +File: hledger.info, Node: date-format, Next: timezone, Prev: skip, Up: CSV + +11.4 ‘date-format’ +================== + +date-format DATEFMT + + This is a helper for the ‘date’ (and ‘date2’) fields. If your CSV +dates are not formatted like ‘YYYY-MM-DD’, ‘YYYY/MM/DD’ or ‘YYYY.MM.DD’, +you’ll need to add a date-format rule describing them with a +strptime-style date parsing pattern - see +https://hackage.haskell.org/package/time/docs/Data-Time-Format.html#v:formatTime. +The pattern must parse the CSV date value completely. Some examples: + +# MM/DD/YY +date-format %m/%d/%y + +# D/M/YYYY +# The - makes leading zeros optional. +date-format %-d/%-m/%Y + +# YYYY-Mmm-DD +date-format %Y-%h-%d + +# M/D/YYYY HH:MM AM some other junk +# Note the time and junk must be fully parsed, though only the date is used. +date-format %-m/%-d/%Y %l:%M %p some other junk + + +File: hledger.info, Node: timezone, Next: newest-first, Prev: date-format, Up: CSV + +11.5 ‘timezone’ +=============== + +timezone TIMEZONE + + When CSV contains date-times that are implicitly in some time zone +other than yours, but containing no explicit time zone information, you +can use this rule to declare the CSV’s native time zone, which helps +prevent off-by-one dates. + + When the CSV date-times do contain time zone information, you don’t +need this rule; instead, use ‘%Z’ in ‘date-format’ (or ‘%z’, ‘%EZ’, +‘%Ez’; see the formatTime link above). + + In either of these cases, hledger will do a time-zone-aware +conversion, localising the CSV date-times to your current system time +zone. If you prefer to localise to some other time zone, eg for +reproducibility, you can (on unix at least) set the output timezone with +the TZ environment variable, eg: + +$ TZ=-1000 hledger print -f foo.csv # or TZ=-1000 hledger import foo.csv + + ‘timezone’ currently does not understand timezone names, except +"UTC", "GMT", "EST", "EDT", "CST", "CDT", "MST", "MDT", "PST", or "PDT". +For others, use numeric format: +HHMM or -HHMM. + + +File: hledger.info, Node: newest-first, Next: intra-day-reversed, Prev: timezone, Up: CSV + +11.6 ‘newest-first’ +=================== + +hledger tries to ensure that the generated transactions will be ordered +chronologically, including intra-day transactions. Usually it can +auto-detect how the CSV records are ordered. But if it encounters CSV +where all records are on the same date, it assumes that the records are +oldest first. If in fact the CSV’s records are normally newest first, +like: + +2022-10-01, txn 3... +2022-10-01, txn 2... +2022-10-01, txn 1... + + you can add the ‘newest-first’ rule to help hledger generate the +transactions in correct order. + +# same-day CSV records are newest first +newest-first + + +File: hledger.info, Node: intra-day-reversed, Next: decimal-mark, Prev: newest-first, Up: CSV + +11.7 ‘intra-day-reversed’ +========================= + +CSV records for each day are sometimes ordered in reverse compared to +the overall date order. Eg, here dates are newest first, but the +transactions on each date are oldest first: + +2022-10-02, txn 3... +2022-10-02, txn 4... +2022-10-01, txn 1... +2022-10-01, txn 2... + + In this situation, add the ‘intra-day-reversed’ rule, and hledger +will compensate, improving the order of transactions. + +# transactions within each day are reversed with respect to the overall date order +intra-day-reversed + + +File: hledger.info, Node: decimal-mark, Next: fields list, Prev: intra-day-reversed, Up: CSV + +11.8 ‘decimal-mark’ +=================== + +decimal-mark . + + or: + +decimal-mark , + + hledger automatically accepts either period or comma as a decimal +mark when parsing numbers (cf Amounts). However if any numbers in the +CSV contain digit group marks, such as thousand-separating commas, you +should declare the decimal mark explicitly with this rule, to avoid +misparsed numbers. + + +File: hledger.info, Node: fields list, Next: Field assignment, Prev: decimal-mark, Up: CSV + +11.9 ‘fields’ list +================== + +fields FIELDNAME1, FIELDNAME2, ... + + A fields list (the word ‘fields’ followed by comma-separated field +names) is optional, but convenient. It does two things: + + 1. It names the CSV field in each column. This can be convenient if + you are referencing them in other rules, so you can say + ‘%SomeField’ instead of remembering ‘%13’. + + 2. Whenever you use one of the special hledger field names (described + below), it assigns the CSV value in this position to that hledger + field. This is the quickest way to populate hledger’s fields and + build a transaction. + + Here’s an example that says "use the 1st, 2nd and 4th fields as the +transaction’s date, description and amount; name the last two fields for +later reference; and ignore the others": + +fields date, description, , amount, , , somefield, anotherfield + + In a fields list, the separator is always comma; it is unrelated to +the CSV file’s separator. Also: + + • There must be least two items in the list (at least one comma). + • Field names may not contain spaces. Spaces before/after field + names are optional. + • Field names may contain ‘_’ (underscore) or ‘-’ (hyphen). + • Fields you don’t care about can be given a dummy name or an empty + name. + + If the CSV contains column headings, it’s convenient to use these for +your field names, suitably modified (eg lower-cased with spaces replaced +by underscores). + + Sometimes you may want to alter a CSV field name to avoid assigning +to a hledger field with the same name. Eg you could call the CSV’s +"balance" field ‘balance_’ to avoid directly setting hledger’s ‘balance’ +field (and generating a balance assertion). + + +File: hledger.info, Node: Field assignment, Next: Field names, Prev: fields list, Up: CSV + +11.10 Field assignment +====================== + +HLEDGERFIELD FIELDVALUE + + Field assignments are the more flexible way to assign CSV values to +hledger fields. They can be used instead of or in addition to a fields +list (see above). + + To assign a value to a hledger field, write the field name (any of +the standard hledger field/pseudo-field names, defined below), a space, +followed by a text value on the same line. This text value may +interpolate CSV fields, referenced by their 1-based position in the CSV +record (‘%N’), or by the name they were given in the fields list +(‘%CSVFIELD’). + + Some examples: + +# set the amount to the 4th CSV field, with " USD" appended +amount %4 USD + +# combine three fields to make a comment, containing note: and date: tags +comment note: %somefield - %anotherfield, date: %1 + + Tips: + + • Interpolation strips outer whitespace (so a CSV value like ‘" 1 "’ + becomes ‘1’ when interpolated) (#1051). + • Interpolations always refer to a CSV field - you can’t interpolate + a hledger field. (See Referencing other fields below). + + +File: hledger.info, Node: Field names, Next: if block, Prev: Field assignment, Up: CSV + +11.11 Field names +================= + +Note the two kinds of field names mentioned here, and used only in +hledger CSV rules files: + + 1. *CSV field names* (‘CSVFIELD’ in these docs): you can optionally + name the CSV columns for easy reference (since hledger doesn’t yet + automatically recognise column headings in a CSV file), by writing + arbitrary names in a ‘fields’ list, eg: + + fields When, What, Some_Id, Net, Total, Foo, Bar + + 2. Special *hledger field names* (‘HLEDGERFIELD’ in these docs): you + must set at least some of these to generate the hledger transaction + from a CSV record, by writing them as the left hand side of a field + assignment, eg: + + date %When + code %Some_Id + description %What + comment %Foo %Bar + amount1 $ %Total + + or directly in a ‘fields’ list: + + fields date, description, code, , amount1, Foo, Bar + currency $ + comment %Foo %Bar + + Here are all the special hledger field names available, and what +happens when you assign values to them: + +* Menu: + +* date field:: +* date2 field:: +* status field:: +* code field:: +* description field:: +* comment field:: +* account field:: +* amount field:: +* currency field:: +* balance field:: + + +File: hledger.info, Node: date field, Next: date2 field, Up: Field names + +11.11.1 date field +------------------ + +Assigning to ‘date’ sets the transaction date. + + +File: hledger.info, Node: date2 field, Next: status field, Prev: date field, Up: Field names + +11.11.2 date2 field +------------------- + +‘date2’ sets the transaction’s secondary date, if any. + + +File: hledger.info, Node: status field, Next: code field, Prev: date2 field, Up: Field names + +11.11.3 status field +-------------------- + +‘status’ sets the transaction’s status, if any. + + +File: hledger.info, Node: code field, Next: description field, Prev: status field, Up: Field names + +11.11.4 code field +------------------ + +‘code’ sets the transaction’s code, if any. + + +File: hledger.info, Node: description field, Next: comment field, Prev: code field, Up: Field names + +11.11.5 description field +------------------------- + +‘description’ sets the transaction’s description, if any. + + +File: hledger.info, Node: comment field, Next: account field, Prev: description field, Up: Field names + +11.11.6 comment field +--------------------- + +‘comment’ sets the transaction’s comment, if any. + + ‘commentN’, where N is a number, sets the Nth posting’s comment. + + You can assign multi-line comments by writing literal ‘\n’ in the +code. A comment starting with ‘\n’ will begin on a new line. + + Comments can contain tags, as usual. + + +File: hledger.info, Node: account field, Next: amount field, Prev: comment field, Up: Field names + +11.11.7 account field +--------------------- + +Assigning to ‘accountN’, where N is 1 to 99, sets the account name of +the Nth posting, and causes that posting to be generated. + + Most often there are two postings, so you’ll want to set ‘account1’ +and ‘account2’. Typically ‘account1’ is associated with the CSV file, +and is set once with a top-level assignment, while ‘account2’ is set +based on each transaction’s description, in conditional rules. + + If a posting’s account name is left unset but its amount is set (see +below), a default account name will be chosen (like "expenses:unknown" +or "income:unknown"). + + +File: hledger.info, Node: amount field, Next: currency field, Prev: account field, Up: Field names + +11.11.8 amount field +-------------------- + +There are several "amount" field name variants, useful for different +situations: + + • ‘amountN’ sets the amount of the Nth posting, and causes that + posting to be generated. By assigning to ‘amount1’, ‘amount2’, ... + etc. you can generate up to 99 postings. Posting numbers don’t + have to be consecutive; in certain situations using a high number + might be helpful to influence the layout of postings. + + • ‘amountN-in’ and ‘amountN-out’ should be used instead, as a pair, + when and only when the amount must be obtained from two CSV fields. + Eg when the CSV has separate Debit and Credit fields instead of a + single Amount field. Note: + + • Don’t think "-in is for the first posting and -out is for the + second posting" - that’s not correct. Think: "‘amountN-in’ + and ‘amountN-out’ together detect the amount for posting N, by + inspecting two CSV fields at once." + • hledger assumes both CSV fields are unsigned, and will + automatically negate the -out value. + • It also expects that at least one of the values is empty or + zero, so it knows which one to ignore. If that’s not the case + you’ll need an if rule (see Setting amounts below). + + • ‘amount’, with no posting number (and similarly, ‘amount-in’ and + ‘amount-out’ with no number) are an older syntax. We keep them for + backwards compatibility, and because they have special behaviour + that is sometimes convenient: + + • They set the amount of posting 1 and (negated) the amount of + posting 2. + • Posting 2’s amount will be converted to cost if it has a cost + price. + • Any of the newer rules for posting 1 or 2 (like ‘amount1’, or + ‘amount2-in’ and ‘amount2-out’) will take precedence. This + allows incrementally migrating old rules files to the new + syntax. + + There’s more to say about amount-setting that doesn’t fit here; +please see also "Setting amounts" below. + + +File: hledger.info, Node: currency field, Next: balance field, Prev: amount field, Up: Field names + +11.11.9 currency field +---------------------- + +‘currency’ sets a currency symbol, to be prepended to all postings’ +amounts. You can use this if the CSV amounts do not have a currency +symbol, eg if it is in a separate column. + + ‘currencyN’ prepends a currency symbol to just the Nth posting’s +amount. + + +File: hledger.info, Node: balance field, Prev: currency field, Up: Field names + +11.11.10 balance field +---------------------- + +‘balanceN’ sets a balance assertion amount (or if the posting amount is +left empty, a balance assignment) on posting N. + + ‘balance’ is a compatibility spelling for hledger <1.17; it is +equivalent to ‘balance1’. + + You can adjust the type of assertion/assignment with the +‘balance-type’ rule (see below). + + See Tips below for more about setting amounts and currency. + + +File: hledger.info, Node: if block, Next: Matchers, Prev: Field names, Up: CSV + +11.12 ‘if’ block +================ + +Rules can be applied conditionally, depending on patterns in the CSV +data. This allows flexibility; in particular, it is how you can +categorise transactions, selecting an appropriate account name based on +their description (for example). There are two ways to write +conditional rules: "if blocks", described here, and "if tables", +described below. + + An if block is the word ‘if’ and one or more "matcher" expressions +(can be a word or phrase), one per line, starting either on the same or +next line; followed by one or more indented rules. Eg, + +if MATCHER + RULE + + or + +if +MATCHER +MATCHER +MATCHER + RULE + RULE + + If any of the matchers succeeds, all of the indented rules will be +applied. They are usually field assignments, but the following special +rules may also be used within an if block: + + • ‘skip’ - skips the matched CSV record (generating no transaction + from it) + • ‘end’ - skips the rest of the current CSV file. + + Some examples: + +# if the record contains "groceries", set account2 to "expenses:groceries" +if groceries + account2 expenses:groceries + +# if the record contains any of these phrases, set account2 and a transaction comment as shown +if +monthly service fee +atm transaction fee +banking thru software + account2 expenses:business:banking + comment XXX deductible ? check it + +# if an empty record is seen (assuming five fields), ignore the rest of the CSV file +if ,,,, + end + + +File: hledger.info, Node: Matchers, Next: if table, Prev: if block, Up: CSV + +11.13 Matchers +============== + +There are two kinds: + + 1. A record matcher is a word or single-line text fragment or regular + expression (‘REGEX’), which hledger will try to match + case-insensitively anywhere within the CSV record. + Eg: ‘whole foods’ + + 2. A field matcher is preceded with a percent sign and CSV field name + (‘%CSVFIELD REGEX’). hledger will try to match these just within + the named CSV field. + Eg: ‘%date 2023’ + + The regular expression is (as usual in hledger) a POSIX extended +regular expression, that also supports GNU word boundaries (‘\b’, ‘\B’, +‘\<’, ‘\>’), and nothing else. If you have trouble, see "Regular +expressions" in the hledger manual +(https://hledger.org/hledger.html#regular-expressions). + + With record matchers, it’s important to know that the record matched +is not the original CSV record, but a modified one: separators will be +converted to commas, and enclosing double quotes (but not enclosing +whitespace) are removed. So for example, when reading an SSV file, if +the original record was: + +2020-01-01; "Acme, Inc."; 1,000 + + the regex would see, and try to match, this modified record text: + +2020-01-01,Acme, Inc., 1,000 + + When an if block has multiple matchers, they are combined as follows: + + • By default they are OR’d (any one of them can match) + • When a matcher is preceded by ampersand (‘&’) it will be AND’ed + with the previous matcher (both of them must match). + + There’s not yet an easy syntax to negate a matcher. + + +File: hledger.info, Node: if table, Next: balance-type, Prev: Matchers, Up: CSV + +11.14 ‘if’ table +================ + +"if tables" are an alternative to if blocks; they can express many +matchers and field assignments in a more compact tabular format, like +this: + +if,HLEDGERFIELD1,HLEDGERFIELD2,... +MATCHERA,VALUE1,VALUE2,... +MATCHERB,VALUE1,VALUE2,... +MATCHERC,VALUE1,VALUE2,... + + + The first character after ‘if’ is taken to be the separator for the +rest of the table. It should be a non-alphanumeric character like ‘,’ +or ‘|’ that does not appear anywhere else in the table. (Note: it is +unrelated to the CSV file’s separator.) Whitespace can be used in the +matcher lines for readability, but not in the if line currently. The +table must be terminated by an empty line (or end of file). Each line +must contain the same number of separators; empty values are allowed. + + The above means: try all of the matchers; whenever a matcher +succeeds, assign all of the values on that line to the corresponding +hledger fields; later lines can overrider earlier ones. It is +equivalent to this sequence of if blocks: + +if MATCHERA + HLEDGERFIELD1 VALUE1 + HLEDGERFIELD2 VALUE2 + ... + +if MATCHERB + HLEDGERFIELD1 VALUE1 + HLEDGERFIELD2 VALUE2 + ... + +if MATCHERC + HLEDGERFIELD1 VALUE1 + HLEDGERFIELD2 VALUE2 + ... + + Example: + +if,account2,comment +atm transaction fee,expenses:business:banking,deductible? check it +%description groceries,expenses:groceries, +2020/01/12.*Plumbing LLC,expenses:house:upkeep,emergency plumbing call-out + + +File: hledger.info, Node: balance-type, Next: include, Prev: if table, Up: CSV + +11.15 ‘balance-type’ +==================== + +Balance assertions generated by assigning to balanceN are of the simple +‘=’ type by default, which is a single-commodity, subaccount-excluding +assertion. You may find the subaccount-including variants more useful, +eg if you have created some virtual subaccounts of checking to help with +budgeting. You can select a different type of assertion with the +‘balance-type’ rule: + +# balance assertions will consider all commodities and all subaccounts +balance-type ==* + + Here are the balance assertion types for quick reference: + += single commodity, exclude subaccounts +=* single commodity, include subaccounts +== multi commodity, exclude subaccounts +==* multi commodity, include subaccounts + + +File: hledger.info, Node: include, Next: Working with CSV, Prev: balance-type, Up: CSV + +11.16 ‘include’ +=============== + +include RULESFILE + + This includes the contents of another CSV rules file at this point. +‘RULESFILE’ is an absolute file path or a path relative to the current +file’s directory. This can be useful for sharing common rules between +several rules files, eg: + +# someaccount.csv.rules + +## someaccount-specific rules +fields date,description,amount +account1 assets:someaccount +account2 expenses:misc + +## common rules +include categorisation.rules + + +File: hledger.info, Node: Working with CSV, Next: CSV rules examples, Prev: include, Up: CSV + +11.17 Working with CSV +====================== + +Some tips: + +* Menu: + * Rapid feedback:: * Valid CSV:: * File Extension:: +* Reading CSV from standard input:: * Reading multiple CSV files:: * Valid transactions:: * Deduplicating importing:: @@ -3526,54 +4272,402 @@ you’ll need to adjust. * Amount decimal places:: * Referencing other fields:: * How CSV rules are evaluated:: +* Well factored rules::  -File: hledger.info, Node: Examples, Next: CSV rules, Up: CSV +File: hledger.info, Node: Rapid feedback, Next: Valid CSV, Up: Working with CSV -11.1 Examples -============= +11.17.1 Rapid feedback +---------------------- -Here are some sample hledger CSV rules files. See also the full -collection at: -https://github.com/simonmichael/hledger/tree/master/examples/csv +It’s a good idea to get rapid feedback while creating/troubleshooting +CSV rules. Here’s a good way, using entr from eradman.com/entrproject: + +$ ls foo.csv* | entr bash -c 'echo ----; hledger -f foo.csv print desc:SOMEDESC' + + A desc: query (eg) is used to select just one, or a few, transactions +of interest. "bash -c" is used to run multiple commands, so we can echo +a separator each time the command re-runs, making it easier to read the +output. + + +File: hledger.info, Node: Valid CSV, Next: File Extension, Prev: Rapid feedback, Up: Working with CSV + +11.17.2 Valid CSV +----------------- + +Note that hledger will only accept valid CSV conforming to RFC 4180, and +equivalent SSV and TSV formats (like RFC 4180 but with semicolon or tab +as separators). This means, eg: + + • Values may be enclosed in double quotes, or not. Enclosing in + single quotes is not allowed. (Eg ‘'A','B'’ is rejected.) + • When values are enclosed in double quotes, spaces outside the + quotes are not allowed. (Eg ‘"A", "B"’ is rejected.) + • When values are not enclosed in quotes, they may not contain double + quotes. (Eg ‘A"A, B’ is rejected.) + + If your CSV/SSV/TSV is not valid in this sense, you’ll need to +transform it before reading with hledger. Try using sed, or a more +permissive CSV parser like python’s csv lib. + + +File: hledger.info, Node: File Extension, Next: Reading CSV from standard input, Prev: Valid CSV, Up: Working with CSV + +11.17.3 File Extension +---------------------- + +To help hledger choose the CSV file reader and show the right error +messages (and choose the right field separator character by default), +it’s best if CSV/SSV/TSV files are named with a ‘.csv’, ‘.ssv’ or ‘.tsv’ +filename extension. (More about this at Data formats.) + + When reading files with the "wrong" extension, you can ensure the CSV +reader (and the default field separator) by prefixing the file path with +‘csv:’, ‘ssv:’ or ‘tsv:’: Eg: + +$ hledger -f ssv:foo.dat print + + You can also override the default field separator with a separator +rule if needed. + + +File: hledger.info, Node: Reading CSV from standard input, Next: Reading multiple CSV files, Prev: File Extension, Up: Working with CSV + +11.17.4 Reading CSV from standard input +--------------------------------------- + +You’ll need the file format prefix when reading CSV from stdin also, +since hledger assumes journal format by default. Eg: + +$ cat foo.dat | hledger -f ssv:- print + + +File: hledger.info, Node: Reading multiple CSV files, Next: Valid transactions, Prev: Reading CSV from standard input, Up: Working with CSV + +11.17.5 Reading multiple CSV files +---------------------------------- + +If you use multiple ‘-f’ options to read multiple CSV files at once, +hledger will look for a correspondingly-named rules file for each CSV +file. But if you use the ‘--rules-file’ option, that rules file will be +used for all the CSV files. + + +File: hledger.info, Node: Valid transactions, Next: Deduplicating importing, Prev: Reading multiple CSV files, Up: Working with CSV + +11.17.6 Valid transactions +-------------------------- + +After reading a CSV file, hledger post-processes and validates the +generated journal entries as it would for a journal file - balancing +them, applying balance assignments, and canonicalising amount styles. +Any errors at this stage will be reported in the usual way, displaying +the problem entry. + + There is one exception: balance assertions, if you have generated +them, will not be checked, since normally these will work only when the +CSV data is part of the main journal. If you do need to check balance +assertions generated from CSV right away, pipe into another hledger: + +$ hledger -f file.csv print | hledger -f- print + + +File: hledger.info, Node: Deduplicating importing, Next: Setting amounts, Prev: Valid transactions, Up: Working with CSV + +11.17.7 Deduplicating, importing +-------------------------------- + +When you download a CSV file periodically, eg to get your latest bank +transactions, the new file may overlap with the old one, containing some +of the same records. + + The import command will (a) detect the new transactions, and (b) +append just those transactions to your main journal. It is idempotent, +so you don’t have to remember how many times you ran it or with which +version of the CSV. (It keeps state in a hidden ‘.latest.FILE.csv’ +file.) This is the easiest way to import CSV data. Eg: + +# download the latest CSV files, then run this command. +# Note, no -f flags needed here. +$ hledger import *.csv [--dry] + + This method works for most CSV files. (Where records have a stable +chronological order, and new records appear only at the new end.) + + A number of other tools and workflows, hledger-specific and +otherwise, exist for converting, deduplicating, classifying and managing +CSV data. See: + + • https://hledger.org/cookbook.html#setups-and-workflows + • https://plaintextaccounting.org -> data import/conversion + + +File: hledger.info, Node: Setting amounts, Next: Amount signs, Prev: Deduplicating importing, Up: Working with CSV + +11.17.8 Setting amounts +----------------------- + +Continuing from amount field above, here are more tips on handling +various amount-setting situations: + + 1. *If the amount is in a single CSV field:* + + a. *If its sign indicates direction of flow:* + Assign it to ‘amountN’, to set the Nth posting’s amount. N is + usually 1 or 2 but can go up to 99. + + b. *If another field indicates direction of flow:* + Use one or more conditional rules to set the appropriate + amount sign. Eg: + + # assume a withdrawal unless Type contains "deposit": + amount1 -%Amount + if %Type deposit + amount1 %Amount + + 2. *If the amount is in one of two CSV fields (eg Debit and Credit):* + + a. *If both fields are unsigned:* + Assign the fields to ‘amountN-in’ and ‘amountN-out’. This + sets posting N’s amount to whichever of these has a non-zero + value. If it’s the -out value, the amount will be negated. + + b. *If either field is signed:* + Use a conditional rule to flip the sign when needed. Eg + below, the -out value already has a minus sign so we undo + hledger’s automatic negating by negating once more (but only + if the field is non-empty, so that we don’t leave a minus sign + by itself): + + fields date, description, amount1-in, amount1-out + if %amount1-out [1-9] + amount1-out -%amount1-out + + c. *If both fields can contain a non-zero value (or both can be + empty):* + The -in/-out rules normally choose the value which is + non-zero/non-empty. Some value pairs can be ambiguous, such + as ‘1’ and ‘none’. For such cases, use conditional rules to + help select the amount. Eg, to handle the above you could + select the value containing non-zero digits: + + fields date, description, in, out + if %in [1-9] + amount1 %in + if %out [1-9] + amount1 %out + + 3. *If you want posting 2’s amount converted to cost:* + Use the unnumbered ‘amount’ (or ‘amount-in’ and ‘amount-out’) + syntax. + + 4. *If the CSV has only balance amounts, not transaction amounts:* + Assign to ‘balanceN’, to set a balance assignment on the Nth + posting, causing the posting’s amount to be calculated + automatically. ‘balance’ with no number is equivalent to + ‘balance1’. In this situation hledger is more likely to guess the + wrong default account name, so you may need to set that explicitly. + + +File: hledger.info, Node: Amount signs, Next: Setting currency/commodity, Prev: Setting amounts, Up: Working with CSV + +11.17.9 Amount signs +-------------------- + +There is some special handling for amount signs, to simplify parsing and +sign-flipping: + + • *If an amount value begins with a plus sign:* + that will be removed: ‘+AMT’ becomes ‘AMT’ + + • *If an amount value is parenthesised:* + it will be de-parenthesised and sign-flipped: ‘(AMT)’ becomes + ‘-AMT’ + + • *If an amount value has two minus signs (or two sets of + parentheses, or a minus sign and parentheses):* + they cancel out and will be removed: ‘--AMT’ or ‘-(AMT)’ becomes + ‘AMT’ + + • *If an amount value contains just a sign (or just a set of + parentheses):* + that is removed, making it an empty value. ‘"+"’ or ‘"-"’ or + ‘"()"’ becomes ‘""’. + + +File: hledger.info, Node: Setting currency/commodity, Next: Amount decimal places, Prev: Amount signs, Up: Working with CSV + +11.17.10 Setting currency/commodity +----------------------------------- + +If the currency/commodity symbol is included in the CSV’s amount +field(s): + +2020-01-01,foo,$123.00 + + you don’t have to do anything special for the commodity symbol, it +will be assigned as part of the amount. Eg: + +fields date,description,amount + +2020-01-01 foo + expenses:unknown $123.00 + income:unknown $-123.00 + + If the currency is provided as a separate CSV field: + +2020-01-01,foo,USD,123.00 + + You can assign that to the ‘currency’ pseudo-field, which has the +special effect of prepending itself to every amount in the transaction +(on the left, with no separating space): + +fields date,description,currency,amount + +2020-01-01 foo + expenses:unknown USD123.00 + income:unknown USD-123.00 + + Or, you can use a field assignment to construct the amount yourself, +with more control. Eg to put the symbol on the right, and separated by +a space: + +fields date,description,cur,amt +amount %amt %cur + +2020-01-01 foo + expenses:unknown 123.00 USD + income:unknown -123.00 USD + + Note we used a temporary field name (‘cur’) that is not ‘currency’ - +that would trigger the prepending effect, which we don’t want here. + + +File: hledger.info, Node: Amount decimal places, Next: Referencing other fields, Prev: Setting currency/commodity, Up: Working with CSV + +11.17.11 Amount decimal places +------------------------------ + +Like amounts in a journal file, the amounts generated by CSV rules like +‘amount1’ influence commodity display styles, such as the number of +decimal places displayed in reports. + + The original amounts as written in the CSV file do not affect display +style (because we don’t yet reliably know their commodity). + + +File: hledger.info, Node: Referencing other fields, Next: How CSV rules are evaluated, Prev: Amount decimal places, Up: Working with CSV + +11.17.12 Referencing other fields +--------------------------------- + +In field assignments, you can interpolate only CSV fields, not hledger +fields. In the example below, there’s both a CSV field and a hledger +field named amount1, but %amount1 always means the CSV field, not the +hledger field: + +# Name the third CSV field "amount1" +fields date,description,amount1 + +# Set hledger's amount1 to the CSV amount1 field followed by USD +amount1 %amount1 USD + +# Set comment to the CSV amount1 (not the amount1 assigned above) +comment %amount1 + + Here, since there’s no CSV amount1 field, %amount1 will produce a +literal "amount1": + +fields date,description,csvamount +amount1 %csvamount USD +# Can't interpolate amount1 here +comment %amount1 + + When there are multiple field assignments to the same hledger field, +only the last one takes effect. Here, comment’s value will be be B, or +C if "something" is matched, but never A: + +comment A +comment B +if something + comment C + + +File: hledger.info, Node: How CSV rules are evaluated, Next: Well factored rules, Prev: Referencing other fields, Up: Working with CSV + +11.17.13 How CSV rules are evaluated +------------------------------------ + +Here’s how to think of CSV rules being evaluated (if you really need +to). First, + + • ‘include’ - all includes are inlined, from top to bottom, depth + first. (At each include point the file is inlined and scanned for + further includes, recursively, before proceeding.) + + Then "global" rules are evaluated, top to bottom. If a rule is +repeated, the last one wins: + + • ‘skip’ (at top level) + • ‘date-format’ + • ‘newest-first’ + • ‘fields’ - names the CSV fields, optionally sets up initial + assignments to hledger fields + + Then for each CSV record in turn: + + • test all ‘if’ blocks. If any of them contain a ‘end’ rule, skip + all remaining CSV records. Otherwise if any of them contain a + ‘skip’ rule, skip that many CSV records. If there are multiple + matched ‘skip’ rules, the first one wins. + • collect all field assignments at top level and in matched ‘if’ + blocks. When there are multiple assignments for a field, keep only + the last one. + • compute a value for each hledger field - either the one that was + assigned to it (and interpolate the %CSVFIELD references), or a + default + • generate a hledger transaction (journal entry) from these values. + + This is all part of the CSV reader, one of several readers hledger +can use to parse input files. When all files have been read +successfully, the transactions are passed as input to whichever hledger +command the user specified. + + +File: hledger.info, Node: Well factored rules, Prev: How CSV rules are evaluated, Up: Working with CSV + +11.17.14 Well factored rules +---------------------------- + +Some things than can help reduce duplication and complexity in rules +files: + + • Extracting common rules usable with multiple CSV files into a + ‘common.rules’, and adding ‘include common.rules’ to each CSV’s + rules file. + + • Splitting if blocks into smaller if blocks, extracting the + frequently used parts. + + +File: hledger.info, Node: CSV rules examples, Prev: Working with CSV, Up: CSV + +11.18 CSV rules examples +======================== * Menu: -* Basic:: * Bank of Ireland:: +* Coinbase:: * Amazon:: * Paypal::  -File: hledger.info, Node: Basic, Next: Bank of Ireland, Up: Examples +File: hledger.info, Node: Bank of Ireland, Next: Coinbase, Up: CSV rules examples -11.1.1 Basic ------------- - -At minimum, the rules file must identify the date and amount fields, and -often it also specifies the date format and how many header lines there -are. Here’s a simple CSV file and a rules file for it: - -Date, Description, Id, Amount -12/11/2019, Foo, 123, 10.23 - -# basic.csv.rules -skip 1 -fields date, description, _, amount -date-format %d/%m/%Y - -$ hledger print -f basic.csv -2019-11-12 Foo - expenses:unknown 10.23 - income:unknown -10.23 - - Default account names are chosen, since we didn’t set them. - - -File: hledger.info, Node: Bank of Ireland, Next: Amazon, Prev: Basic, Up: Examples - -11.1.2 Bank of Ireland ----------------------- +11.18.1 Bank of Ireland +----------------------- Here’s a CSV with two amount fields (Debit and Credit), and a balance field, which we can use to add balance assertions, which is not @@ -3623,10 +4717,37 @@ reading directly from CSV, but they will be checked if these entries are imported into a journal file.  -File: hledger.info, Node: Amazon, Next: Paypal, Prev: Bank of Ireland, Up: Examples +File: hledger.info, Node: Coinbase, Next: Amazon, Prev: Bank of Ireland, Up: CSV rules examples -11.1.3 Amazon -------------- +11.18.2 Coinbase +---------------- + +A simple example with some CSV from Coinbase. The spot price is +recorded using cost notation. The legacy ‘amount’ field name +conveniently sets amount 2 (posting 2’s amount) to the total cost. + +# Timestamp,Transaction Type,Asset,Quantity Transacted,Spot Price Currency,Spot Price at Transaction,Subtotal,Total (inclusive of fees and/or spread),Fees and/or Spread,Notes +# 2021-12-30T06:57:59Z,Receive,USDC,100,GBP,0.740000,"","","","Received 100.00 USDC from an external account" + +# coinbase.csv.rules +skip 1 +fields Timestamp,Transaction_Type,Asset,Quantity_Transacted,Spot_Price_Currency,Spot_Price_at_Transaction,Subtotal,Total,Fees_Spread,Notes +date %Timestamp +date-format %Y-%m-%dT%T%Z +description %Notes +account1 assets:coinbase:cc +amount %Quantity_Transacted %Asset @ %Spot_Price_at_Transaction %Spot_Price_Currency + +$ hledger print -f coinbase.csv +2021-12-30 Received 100.00 USDC from an external account + assets:coinbase:cc 100 USDC @ 0.740000 GBP + income:unknown -74.000000 GBP + + +File: hledger.info, Node: Amazon, Next: Paypal, Prev: Coinbase, Up: CSV rules examples + +11.18.3 Amazon +-------------- Here we convert amazon.com order history, and use an if block to generate a third posting if there’s a fee. (In practice you’d probably @@ -3681,10 +4802,10 @@ $ hledger -f amazon-orders.csv print expenses:fees $1.00  -File: hledger.info, Node: Paypal, Prev: Amazon, Up: Examples +File: hledger.info, Node: Paypal, Prev: Amazon, Up: CSV rules examples -11.1.4 Paypal -------------- +11.18.4 Paypal +-------------- Here’s a real-world rules file for (customised) Paypal CSV, with some Paypal-specific rules, and a second rules file included: @@ -3834,1035 +4955,6 @@ $ hledger -f paypal-custom.csv print revenues:foss donations:darcshub $-10.00 ; business: expenses:banking:paypal $0.59 ; business: - -File: hledger.info, Node: CSV rules, Next: separator, Prev: Examples, Up: CSV - -11.2 CSV rules -============== - -The following kinds of rule can appear in the rules file, in any order. -Blank lines and lines beginning with ‘#’ or ‘;’ or ‘*’ are ignored. - -*‘separator’* a custom field separator -*‘skip’* skip one or more header lines or matched CSV - records -*‘date-format’* how to parse dates in CSV records -*‘timezone’* declare the time zone of ambiguous CSV - date-times -*‘decimal-mark’* the decimal mark used in CSV amounts, if - ambiguous -*‘newest-first’* improve txn order when there are multiple - records, newest first, all with the same - date -*‘intra-day-reversed’* improve txn order when each day’s txns are - reverse of the overall date order -*‘balance-type’* choose which type of balance assignments to - use -*‘fields’* name CSV fields, assign them to hledger - fields -*Field assignment* assign a value to one hledger field, with - interpolation -*Field names* hledger field names, used in the fields list - and field assignments -*‘if’* apply some rules to CSV records matched by - patterns -*‘if’ table* apply some rules to CSV records matched by - patterns, alternate syntax -*‘end’* skip the remaining CSV records -*‘include’* inline another CSV rules file - - -File: hledger.info, Node: separator, Next: skip, Prev: CSV rules, Up: CSV - -11.3 ‘separator’ -================ - -You can use the ‘separator’ rule to read other kinds of -character-separated data. The argument is any single separator -character, or the words ‘tab’ or ‘space’ (case insensitive). Eg, for -comma-separated values (CSV): - -separator , - - or for semicolon-separated values (SSV): - -separator ; - - or for tab-separated values (TSV): - -separator TAB - - If the input file has a ‘.csv’, ‘.ssv’ or ‘.tsv’ file extension (or a -‘csv:’, ‘ssv:’, ‘tsv:’ prefix), the appropriate separator will be -inferred automatically, and you won’t need this rule. - - -File: hledger.info, Node: skip, Next: date-format, Prev: separator, Up: CSV - -11.4 ‘skip’ -=========== - -skip N - - The word ‘skip’ followed by a number (or no number, meaning 1) tells -hledger to ignore this many non-empty lines at the start of the input -data. (Empty/blank lines are skipped automatically, so you don’t need -to count those.) You’ll need this whenever your CSV data contains -header lines. Header lines skipped in this way are ignored, and not -parsed as CSV. - - ‘skip’ can also be used inside if blocks (described below), to skip -individual data records. Note records skipped in this way are still -required to be valid CSV, even though otherwise ignored. - - -File: hledger.info, Node: date-format, Next: timezone, Prev: skip, Up: CSV - -11.5 ‘date-format’ -================== - -date-format DATEFMT - - This is a helper for the ‘date’ (and ‘date2’) fields. If your CSV -dates are not formatted like ‘YYYY-MM-DD’, ‘YYYY/MM/DD’ or ‘YYYY.MM.DD’, -you’ll need to add a date-format rule describing them with a -strptime-style date parsing pattern - see -https://hackage.haskell.org/package/time/docs/Data-Time-Format.html#v:formatTime. -The pattern must parse the CSV date value completely. Some examples: - -# MM/DD/YY -date-format %m/%d/%y - -# D/M/YYYY -# The - makes leading zeros optional. -date-format %-d/%-m/%Y - -# YYYY-Mmm-DD -date-format %Y-%h-%d - -# M/D/YYYY HH:MM AM some other junk -# Note the time and junk must be fully parsed, though only the date is used. -date-format %-m/%-d/%Y %l:%M %p some other junk - - -File: hledger.info, Node: timezone, Next: decimal-mark, Prev: date-format, Up: CSV - -11.6 ‘timezone’ -=============== - -timezone TIMEZONE - - When CSV contains date-times that are implicitly in some time zone -other than yours, but containing no explicit time zone information, you -can use this rule to declare the CSV’s native time zone, which helps -prevent off-by-one dates. - - When the CSV date-times do contain time zone information, you don’t -need this rule; instead, use ‘%Z’ in ‘date-format’ (or ‘%z’, ‘%EZ’, -‘%Ez’; see the formatTime link above). - - In either of these cases, hledger will do a time-zone-aware -conversion, localising the CSV date-times to your current system time -zone. If you prefer to localise to some other time zone, eg for -reproducibility, you can (on unix at least) set the output timezone with -the TZ environment variable, eg: - -$ TZ=-1000 hledger print -f foo.csv # or TZ=-1000 hledger import foo.csv - - ‘timezone’ currently does not understand timezone names, except -"UTC", "GMT", "EST", "EDT", "CST", "CDT", "MST", "MDT", "PST", or "PDT". -For others, use numeric format: +HHMM or -HHMM. - - -File: hledger.info, Node: decimal-mark, Next: newest-first, Prev: timezone, Up: CSV - -11.7 ‘decimal-mark’ -=================== - -decimal-mark . - - or: - -decimal-mark , - - hledger automatically accepts either period or comma as a decimal -mark when parsing numbers (cf Amounts). However if any numbers in the -CSV contain digit group marks, such as thousand-separating commas, you -should declare the decimal mark explicitly with this rule, to avoid -misparsed numbers. - - -File: hledger.info, Node: newest-first, Next: intra-day-reversed, Prev: decimal-mark, Up: CSV - -11.8 ‘newest-first’ -=================== - -hledger tries to ensure that the generated transactions will be ordered -chronologically, including intra-day transactions. Usually it can -auto-detect how the CSV records are ordered. But if it encounters CSV -where all records are on the same date, it assumes that the records are -oldest first. If in fact the CSV’s records are normally newest first, -like: - -2022-10-01, txn 3... -2022-10-01, txn 2... -2022-10-01, txn 1... - - you can add the ‘newest-first’ rule to help hledger generate the -transactions in correct order. - -# same-day CSV records are newest first -newest-first - - -File: hledger.info, Node: intra-day-reversed, Next: fields, Prev: newest-first, Up: CSV - -11.9 ‘intra-day-reversed’ -========================= - -CSV records for each day are sometimes ordered in reverse compared to -the overall date order. Eg, here dates are newest first, but the -transactions on each date are oldest first: - -2022-10-02, txn 3... -2022-10-02, txn 4... -2022-10-01, txn 1... -2022-10-01, txn 2... - - In this situation, add the ‘intra-day-reversed’ rule, and hledger -will compensate, improving the order of transactions. - -# transactions within each day are reversed with respect to the overall date order -intra-day-reversed - - -File: hledger.info, Node: fields, Next: Field assignment, Prev: intra-day-reversed, Up: CSV - -11.10 ‘fields’ -============== - -fields FIELDNAME1, FIELDNAME2, ... - - A fields list (the word "fields" followed by comma-separated field -names) is the quick way to assign CSV field values to hledger fields. -(The other way is field assignments, see below.) A fields list does -does two things: - - 1. It names the CSV fields. This is optional, but can be convenient - later for interpolating them. - - 2. Whenever you use a standard hledger field name (defined below), the - CSV value is assigned to that part of the hledger transaction. - - Here’s an example that says "use the 1st, 2nd and 4th fields as the -transaction’s date, description and amount; name the last two fields for -later reference; and ignore the others": - -fields date, description, , amount, , , somefield, anotherfield - - Tips: - - • The fields list always use commas, even if your CSV data uses - another separator character. - • Currently there must be least two items in the list (at least one - comma). - • Field names may not contain spaces. Spaces before/after field - names are optional. - • Field names may contain ‘_’ (underscore) or ‘-’ (hyphen). - • If the CSV contains column headings, it’s a good idea to use these, - suitably modified, as the basis for your field names (eg - lower-cased, with underscores instead of spaces). - • If some heading names match standard hledger fields, but you don’t - want to set the hledger fields directly, alter those names, eg by - appending an underscore. - • Fields you don’t care about can be given a dummy name (eg: ‘_’ ), - or no name. - - -File: hledger.info, Node: Field assignment, Next: Field names, Prev: fields, Up: CSV - -11.11 Field assignment -====================== - -HLEDGERFIELDNAME FIELDVALUE - - Field assignments are the more flexible way to assign CSV values to -hledger fields. They can be used instead of or in addition to a fields -list (see above). - - To assign a value to a hledger field, write the field name (any of -the standard hledger field/pseudo-field names, defined below), a space, -followed by a text value on the same line. This text value may -interpolate CSV fields, referenced by their 1-based position in the CSV -record (‘%N’), or by the name they were given in the fields list -(‘%CSVFIELDNAME’). - - Some examples: - -# set the amount to the 4th CSV field, with " USD" appended -amount %4 USD - -# combine three fields to make a comment, containing note: and date: tags -comment note: %somefield - %anotherfield, date: %1 - - Tips: - - • Interpolation strips outer whitespace (so a CSV value like ‘" 1 "’ - becomes ‘1’ when interpolated) (#1051). - • Interpolations always refer to a CSV field - you can’t interpolate - a hledger field. (See Referencing other fields below). - - -File: hledger.info, Node: Field names, Next: if, Prev: Field assignment, Up: CSV - -11.12 Field names -================= - -Here are the standard hledger field (and pseudo-field) names, which you -can use in a fields list or in field assignments. For more about the -transaction parts they refer to, see Transactions. - -* Menu: - -* date field:: -* date2 field:: -* status field:: -* code field:: -* description field:: -* comment field:: -* account field:: -* amount field:: -* currency field:: -* balance field:: - - -File: hledger.info, Node: date field, Next: date2 field, Up: Field names - -11.12.1 date field ------------------- - -Assigning to ‘date’ sets the transaction date. - - -File: hledger.info, Node: date2 field, Next: status field, Prev: date field, Up: Field names - -11.12.2 date2 field -------------------- - -‘date2’ sets the transaction’s secondary date, if any. - - -File: hledger.info, Node: status field, Next: code field, Prev: date2 field, Up: Field names - -11.12.3 status field --------------------- - -‘status’ sets the transaction’s status, if any. - - -File: hledger.info, Node: code field, Next: description field, Prev: status field, Up: Field names - -11.12.4 code field ------------------- - -‘code’ sets the transaction’s code, if any. - - -File: hledger.info, Node: description field, Next: comment field, Prev: code field, Up: Field names - -11.12.5 description field -------------------------- - -‘description’ sets the transaction’s description, if any. - - -File: hledger.info, Node: comment field, Next: account field, Prev: description field, Up: Field names - -11.12.6 comment field ---------------------- - -‘comment’ sets the transaction’s comment, if any. - - ‘commentN’, where N is a number, sets the Nth posting’s comment. - - You can assign multi-line comments by writing literal ‘\n’ in the -code. A comment starting with ‘\n’ will begin on a new line. - - Comments can contain tags, as usual. - - -File: hledger.info, Node: account field, Next: amount field, Prev: comment field, Up: Field names - -11.12.7 account field ---------------------- - -Assigning to ‘accountN’, where N is 1 to 99, sets the account name of -the Nth posting, and causes that posting to be generated. - - Most often there are two postings, so you’ll want to set ‘account1’ -and ‘account2’. Typically ‘account1’ is associated with the CSV file, -and is set once with a top-level assignment, while ‘account2’ is set -based on each transaction’s description, and in conditional blocks. - - If a posting’s account name is left unset but its amount is set (see -below), a default account name will be chosen (like "expenses:unknown" -or "income:unknown"). - - -File: hledger.info, Node: amount field, Next: currency field, Prev: account field, Up: Field names - -11.12.8 amount field --------------------- - -‘amountN’ sets the amount of the Nth posting, and causes that posting to -be generated. By assigning to ‘amount1’, ‘amount2’, ... etc. you can -generate up to 99 postings. - - ‘amountN-in’ and ‘amountN-out’ can be used instead, if the CSV uses -separate fields for debits and credits (inflows and outflows). hledger -assumes both of these CSV fields are unsigned, and will automatically -negate the "-out" value. It also requires that at least one of them is -either empty or zero. See "Setting amounts" below for more on this -topic. - - ‘amount’, or ‘amount-in’ and ‘amount-out’ are a legacy mode, to keep -pre-hledger-1.17 CSV rules files working (and for occasional -convenience). They are suitable only for two-posting transactions; they -set both posting 1’s and posting 2’s amount. Posting 2’s amount will be -negated, and also converted to cost if there’s a cost price. - - Note: it might sound as if amount-in is for one posting and -amount-out for the other posting, but no; use the -in and -out rules -together for the same posting, producing one amount from two CSV fields. - - If you have an existing rules file using the unnumbered form, you -might want to use the numbered form in certain conditional blocks, -without having to update and retest all the old rules. To facilitate -this, posting 1 ignores ‘amount’/‘amount-in’/‘amount-out’ if any of -‘amount1’/‘amount1-in’/‘amount1-out’ are assigned, and posting 2 ignores -them if any of ‘amount2’/‘amount2-in’/‘amount2-out’ are assigned, -avoiding conflicts. - - -File: hledger.info, Node: currency field, Next: balance field, Prev: amount field, Up: Field names - -11.12.9 currency field ----------------------- - -‘currency’ sets a currency symbol, to be prepended to all postings’ -amounts. You can use this if the CSV amounts do not have a currency -symbol, eg if it is in a separate column. - - ‘currencyN’ prepends a currency symbol to just the Nth posting’s -amount. - - -File: hledger.info, Node: balance field, Prev: currency field, Up: Field names - -11.12.10 balance field ----------------------- - -‘balanceN’ sets a balance assertion amount (or if the posting amount is -left empty, a balance assignment) on posting N. - - ‘balance’ is a compatibility spelling for hledger <1.17; it is -equivalent to ‘balance1’. - - You can adjust the type of assertion/assignment with the -‘balance-type’ rule (see below). - - See Tips below for more about setting amounts and currency. - - -File: hledger.info, Node: if, Next: if table, Prev: Field names, Up: CSV - -11.13 ‘if’ -========== - -if MATCHER - RULE - -if -MATCHER -MATCHER -MATCHER - RULE - RULE - - Conditional blocks ("if blocks") are a block of rules that are -applied only to CSV records which match certain patterns. They are -often used for customising account names based on transaction -descriptions. - -* Menu: - -* Matching the whole record:: -* Matching individual fields:: -* Combining matchers:: -* Rules applied on successful match:: - - -File: hledger.info, Node: Matching the whole record, Next: Matching individual fields, Up: if - -11.13.1 Matching the whole record ---------------------------------- - -Each MATCHER can be a record matcher, which looks like this: - -REGEX - - REGEX is a case-insensitive regular expression that tries to match -anywhere within the CSV record. It is a POSIX ERE (extended regular -expression) that also supports GNU word boundaries (‘\b’, ‘\B’, ‘\<’, -‘\>’), and nothing else. If you have trouble, be sure to check our doc: -https://hledger.org/hledger.html#regular-expressions - - Important note: the record that is matched is not the original -record, but a synthetic one, with any enclosing double quotes (but not -enclosing whitespace) removed, and always comma-separated (which means -that a field containing a comma will appear like two fields). Eg, if -the original record is ‘2020-01-01; "Acme, Inc."; 1,000’, the REGEX will -actually see ‘2020-01-01,Acme, Inc., 1,000’). - - -File: hledger.info, Node: Matching individual fields, Next: Combining matchers, Prev: Matching the whole record, Up: if - -11.13.2 Matching individual fields ----------------------------------- - -Or, MATCHER can be a field matcher, like this: - -%CSVFIELD REGEX - - which matches just the content of a particular CSV field. CSVFIELD -is a percent sign followed by the field’s name or column number, like -‘%date’ or ‘%1’. - - -File: hledger.info, Node: Combining matchers, Next: Rules applied on successful match, Prev: Matching individual fields, Up: if - -11.13.3 Combining matchers --------------------------- - -A single matcher can be written on the same line as the "if"; or -multiple matchers can be written on the following lines, non-indented. -Multiple matchers are OR’d (any one of them can match), unless one -begins with an ‘&’ symbol, in which case it is AND’ed with the previous -matcher. - -if -MATCHER -& MATCHER - RULE - - -File: hledger.info, Node: Rules applied on successful match, Prev: Combining matchers, Up: if - -11.13.4 Rules applied on successful match ------------------------------------------ - -After the patterns there should be one or more rules to apply, all -indented by at least one space. Three kinds of rule are allowed in -conditional blocks: - - • field assignments (to set a hledger field) - • skip (to skip the matched CSV record) - • end (to skip all remaining CSV records). - - Examples: - -# if the CSV record contains "groceries", set account2 to "expenses:groceries" -if groceries - account2 expenses:groceries - -# if the CSV record contains any of these patterns, set account2 and comment as shown -if -monthly service fee -atm transaction fee -banking thru software - account2 expenses:business:banking - comment XXX deductible ? check it - - -File: hledger.info, Node: if table, Next: end, Prev: if, Up: CSV - -11.14 ‘if’ table -================ - -if,CSVFIELDNAME1,CSVFIELDNAME2,...,CSVFIELDNAMEn -MATCHER1,VALUE11,VALUE12,...,VALUE1n -MATCHER2,VALUE21,VALUE22,...,VALUE2n -MATCHER3,VALUE31,VALUE32,...,VALUE3n - - - Conditional tables ("if tables") are a different syntax to specify -field assignments that will be applied only to CSV records which match -certain patterns. - - MATCHER could be either field or record matcher, as described above. -When MATCHER matches, values from that row would be assigned to the CSV -fields named on the ‘if’ line, in the same order. - - Therefore ‘if’ table is exactly equivalent to a sequence of of ‘if’ -blocks: - -if MATCHER1 - CSVFIELDNAME1 VALUE11 - CSVFIELDNAME2 VALUE12 - ... - CSVFIELDNAMEn VALUE1n - -if MATCHER2 - CSVFIELDNAME1 VALUE21 - CSVFIELDNAME2 VALUE22 - ... - CSVFIELDNAMEn VALUE2n - -if MATCHER3 - CSVFIELDNAME1 VALUE31 - CSVFIELDNAME2 VALUE32 - ... - CSVFIELDNAMEn VALUE3n - - Each line starting with MATCHER should contain enough (possibly -empty) values for all the listed fields. - - Rules would be checked and applied in the order they are listed in -the table and, like with ‘if’ blocks, later rules (in the same or -another table) or ‘if’ blocks could override the effect of any rule. - - Instead of ’,’ you can use a variety of other non-alphanumeric -characters as a separator. First character after ‘if’ is taken to be -the separator for the rest of the table. It is the responsibility of -the user to ensure that separator does not occur inside MATCHERs and -values - there is no way to escape separator. - - Example: - -if,account2,comment -atm transaction fee,expenses:business:banking,deductible? check it -%description groceries,expenses:groceries, -2020/01/12.*Plumbing LLC,expenses:house:upkeep,emergency plumbing call-out - - -File: hledger.info, Node: end, Next: include, Prev: if table, Up: CSV - -11.15 ‘end’ -=========== - -This rule can be used inside if blocks (only), to make hledger stop -reading this CSV file and move on to the next input file, or to command -execution. Eg: - -# ignore everything following the first empty record -if ,,,, - end - - -File: hledger.info, Node: include, Next: balance-type, Prev: end, Up: CSV - -11.16 ‘include’ -=============== - -include RULESFILE - - This includes the contents of another CSV rules file at this point. -‘RULESFILE’ is an absolute file path or a path relative to the current -file’s directory. This can be useful for sharing common rules between -several rules files, eg: - -# someaccount.csv.rules - -## someaccount-specific rules -fields date,description,amount -account1 assets:someaccount -account2 expenses:misc - -## common rules -include categorisation.rules - - -File: hledger.info, Node: balance-type, Next: Tips, Prev: include, Up: CSV - -11.17 ‘balance-type’ -==================== - -Balance assertions generated by assigning to balanceN are of the simple -‘=’ type by default, which is a single-commodity, subaccount-excluding -assertion. You may find the subaccount-including variants more useful, -eg if you have created some virtual subaccounts of checking to help with -budgeting. You can select a different type of assertion with the -‘balance-type’ rule: - -# balance assertions will consider all commodities and all subaccounts -balance-type ==* - - Here are the balance assertion types for quick reference: - -= single commodity, exclude subaccounts -=* single commodity, include subaccounts -== multi commodity, exclude subaccounts -==* multi commodity, include subaccounts - - -File: hledger.info, Node: Tips, Next: Rapid feedback, Prev: balance-type, Up: CSV - -11.18 Tips -========== - - -File: hledger.info, Node: Rapid feedback, Next: Valid CSV, Prev: Tips, Up: CSV - -11.19 Rapid feedback -==================== - -It’s a good idea to get rapid feedback while creating/troubleshooting -CSV rules. Here’s a good way, using entr from eradman.com/entrproject: - -$ ls foo.csv* | entr bash -c 'echo ----; hledger -f foo.csv print desc:SOMEDESC' - - A desc: query (eg) is used to select just one, or a few, transactions -of interest. "bash -c" is used to run multiple commands, so we can echo -a separator each time the command re-runs, making it easier to read the -output. - - -File: hledger.info, Node: Valid CSV, Next: File Extension, Prev: Rapid feedback, Up: CSV - -11.20 Valid CSV -=============== - -Note that hledger will only accept valid CSV conforming to RFC 4180, and -equivalent SSV and TSV formats (like RFC 4180 but with semicolon or tab -as separators). This means, eg: - - • Values may be enclosed in double quotes, or not. Enclosing in - single quotes is not allowed. (Eg ‘'A','B'’ is rejected.) - • When values are enclosed in double quotes, spaces outside the - quotes are not allowed. (Eg ‘"A", "B"’ is rejected.) - • When values are not enclosed in quotes, they may not contain double - quotes. (Eg ‘A"A, B’ is rejected.) - - If your CSV/SSV/TSV is not valid in this sense, you’ll need to -transform it before reading with hledger. Try using sed, or a more -permissive CSV parser like python’s csv lib. - - -File: hledger.info, Node: File Extension, Next: Reading multiple CSV files, Prev: Valid CSV, Up: CSV - -11.21 File Extension -==================== - -To help hledger identify the format and show the right error messages, -CSV/SSV/TSV files should normally be named with a ‘.csv’, ‘.ssv’ or -‘.tsv’ filename extension. Or, the file path should be prefixed with -‘csv:’, ‘ssv:’ or ‘tsv:’. Eg: - -$ hledger -f foo.ssv print - - or: - -$ cat foo | hledger -f ssv:- foo - - You can override the file extension with a separator rule if needed. -See also: Input files in the hledger manual. - - -File: hledger.info, Node: Reading multiple CSV files, Next: Valid transactions, Prev: File Extension, Up: CSV - -11.22 Reading multiple CSV files -================================ - -If you use multiple ‘-f’ options to read multiple CSV files at once, -hledger will look for a correspondingly-named rules file for each CSV -file. But if you use the ‘--rules-file’ option, that rules file will be -used for all the CSV files. - - -File: hledger.info, Node: Valid transactions, Next: Deduplicating importing, Prev: Reading multiple CSV files, Up: CSV - -11.23 Valid transactions -======================== - -After reading a CSV file, hledger post-processes and validates the -generated journal entries as it would for a journal file - balancing -them, applying balance assignments, and canonicalising amount styles. -Any errors at this stage will be reported in the usual way, displaying -the problem entry. - - There is one exception: balance assertions, if you have generated -them, will not be checked, since normally these will work only when the -CSV data is part of the main journal. If you do need to check balance -assertions generated from CSV right away, pipe into another hledger: - -$ hledger -f file.csv print | hledger -f- print - - -File: hledger.info, Node: Deduplicating importing, Next: Setting amounts, Prev: Valid transactions, Up: CSV - -11.24 Deduplicating, importing -============================== - -When you download a CSV file periodically, eg to get your latest bank -transactions, the new file may overlap with the old one, containing some -of the same records. - - The import command will (a) detect the new transactions, and (b) -append just those transactions to your main journal. It is idempotent, -so you don’t have to remember how many times you ran it or with which -version of the CSV. (It keeps state in a hidden ‘.latest.FILE.csv’ -file.) This is the easiest way to import CSV data. Eg: - -# download the latest CSV files, then run this command. -# Note, no -f flags needed here. -$ hledger import *.csv [--dry] - - This method works for most CSV files. (Where records have a stable -chronological order, and new records appear only at the new end.) - - A number of other tools and workflows, hledger-specific and -otherwise, exist for converting, deduplicating, classifying and managing -CSV data. See: - - • https://hledger.org/cookbook.html#setups-and-workflows - • https://plaintextaccounting.org -> data import/conversion - - -File: hledger.info, Node: Setting amounts, Next: Amount signs, Prev: Deduplicating importing, Up: CSV - -11.25 Setting amounts -===================== - -Some tips on using the amount-setting rules discussed above. - - Here are the ways to set a posting’s amount: - - 1. *If the CSV has a single amount field:* - Assign (via a fields list or a field assignment) to ‘amountN’. - This sets the Nth posting’s amount. N is usually 1 or 2 but can go - up to 99. - - 2. *If the CSV has separate amount fields for debit & credit (in & - out):* - - a. *If both fields are unsigned:* - Assign to ‘amountN-in’ and ‘amountN-out’. This sets posting - N’s amount to whichever of these has a non-zero value, and - negates the "-out" value. - - b. *If either field is signed (can contain a minus sign):* - Use a conditional rule to flip the sign (of non-empty values). - Since hledger always negates amountN-out, if it was already - negative, we must undo that by negating once more (but only if - the field is non-empty): - - fields date, description, amount1-in, amount1-out - if %amount1-out [1-9] - amount1-out -%amount1-out - - c. *If both fields, or neither field, can contain a non-zero - value:* - hledger normally expects exactly one of the fields to have a - non-zero value. Eg, the ‘amountN-in’/‘amountN-out’ rules - would reject value pairs like these: - - "", "" - "0", "0" - "1", "none" - - So, use smarter conditional rules to set the amount from the - appropriate field. Eg, these rules would make it use only the - value containing non-zero digits, handling the above: - - fields date, description, in, out - if %in [1-9] - amount1 %in - if %out [1-9] - amount1 %out - - 3. *If you want posting 2’s amount converted to cost:* - Assign to ‘amount’ (or to ‘amount-in’ and ‘amount-out’). (This is - the legacy numberless syntax, which sets amount1 and amount2 and - converts amount2 to cost.) - - 4. *If the CSV has the balance instead of the transaction amount:* - Assign to ‘balanceN’, which sets posting N’s amount indirectly via - a balance assignment. (Old syntax: ‘balance’, equivalent to - ‘balance1’.) - - • *If hledger guesses the wrong default account name:* - When setting the amount via balance assertion, hledger may - guess the wrong default account name. So, set the account - name explicitly, eg: - - fields date, description, balance1 - account1 assets:checking - - -File: hledger.info, Node: Amount signs, Next: Setting currency/commodity, Prev: Setting amounts, Up: CSV - -11.26 Amount signs -================== - -There is some special handling for amount signs, to simplify parsing and -sign-flipping: - - • *If an amount value begins with a plus sign:* - that will be removed: ‘+AMT’ becomes ‘AMT’ - - • *If an amount value is parenthesised:* - it will be de-parenthesised and sign-flipped: ‘(AMT)’ becomes - ‘-AMT’ - - • *If an amount value has two minus signs (or two sets of - parentheses, or a minus sign and parentheses):* - they cancel out and will be removed: ‘--AMT’ or ‘-(AMT)’ becomes - ‘AMT’ - - • *If an amount value contains just a sign (or just a set of - parentheses):* - that is removed, making it an empty value. ‘"+"’ or ‘"-"’ or - ‘"()"’ becomes ‘""’. - - -File: hledger.info, Node: Setting currency/commodity, Next: Amount decimal places, Prev: Amount signs, Up: CSV - -11.27 Setting currency/commodity -================================ - -If the currency/commodity symbol is included in the CSV’s amount -field(s): - -2020-01-01,foo,$123.00 - - you don’t have to do anything special for the commodity symbol, it -will be assigned as part of the amount. Eg: - -fields date,description,amount - -2020-01-01 foo - expenses:unknown $123.00 - income:unknown $-123.00 - - If the currency is provided as a separate CSV field: - -2020-01-01,foo,USD,123.00 - - You can assign that to the ‘currency’ pseudo-field, which has the -special effect of prepending itself to every amount in the transaction -(on the left, with no separating space): - -fields date,description,currency,amount - -2020-01-01 foo - expenses:unknown USD123.00 - income:unknown USD-123.00 - - Or, you can use a field assignment to construct the amount yourself, -with more control. Eg to put the symbol on the right, and separated by -a space: - -fields date,description,cur,amt -amount %amt %cur - -2020-01-01 foo - expenses:unknown 123.00 USD - income:unknown -123.00 USD - - Note we used a temporary field name (‘cur’) that is not ‘currency’ - -that would trigger the prepending effect, which we don’t want here. - - -File: hledger.info, Node: Amount decimal places, Next: Referencing other fields, Prev: Setting currency/commodity, Up: CSV - -11.28 Amount decimal places -=========================== - -Like amounts in a journal file, the amounts generated by CSV rules like -‘amount1’ influence commodity display styles, such as the number of -decimal places displayed in reports. - - The original amounts as written in the CSV file do not affect display -style (because we don’t yet reliably know their commodity). - - -File: hledger.info, Node: Referencing other fields, Next: How CSV rules are evaluated, Prev: Amount decimal places, Up: CSV - -11.29 Referencing other fields -============================== - -In field assignments, you can interpolate only CSV fields, not hledger -fields. In the example below, there’s both a CSV field and a hledger -field named amount1, but %amount1 always means the CSV field, not the -hledger field: - -# Name the third CSV field "amount1" -fields date,description,amount1 - -# Set hledger's amount1 to the CSV amount1 field followed by USD -amount1 %amount1 USD - -# Set comment to the CSV amount1 (not the amount1 assigned above) -comment %amount1 - - Here, since there’s no CSV amount1 field, %amount1 will produce a -literal "amount1": - -fields date,description,csvamount -amount1 %csvamount USD -# Can't interpolate amount1 here -comment %amount1 - - When there are multiple field assignments to the same hledger field, -only the last one takes effect. Here, comment’s value will be be B, or -C if "something" is matched, but never A: - -comment A -comment B -if something - comment C - - -File: hledger.info, Node: How CSV rules are evaluated, Prev: Referencing other fields, Up: CSV - -11.30 How CSV rules are evaluated -================================= - -Here’s how to think of CSV rules being evaluated (if you really need -to). First, - - • ‘include’ - all includes are inlined, from top to bottom, depth - first. (At each include point the file is inlined and scanned for - further includes, recursively, before proceeding.) - - Then "global" rules are evaluated, top to bottom. If a rule is -repeated, the last one wins: - - • ‘skip’ (at top level) - • ‘date-format’ - • ‘newest-first’ - • ‘fields’ - names the CSV fields, optionally sets up initial - assignments to hledger fields - - Then for each CSV record in turn: - - • test all ‘if’ blocks. If any of them contain a ‘end’ rule, skip - all remaining CSV records. Otherwise if any of them contain a - ‘skip’ rule, skip that many CSV records. If there are multiple - matched ‘skip’ rules, the first one wins. - • collect all field assignments at top level and in matched ‘if’ - blocks. When there are multiple assignments for a field, keep only - the last one. - • compute a value for each hledger field - either the one that was - assigned to it (and interpolate the %CSVFIELDNAME references), or a - default - • generate a synthetic hledger transaction from these values. - - This is all part of the CSV reader, one of several readers hledger -can use to parse input files. When all files have been read -successfully, the transactions are passed as input to whichever hledger -command the user specified. -  File: hledger.info, Node: Timeclock, Next: Timedot, Prev: CSV, Up: Top @@ -6017,6 +6109,7 @@ postings: subaccounts of these) 4. the equity postings’ amounts must exactly match the non-equity postings’ amounts + 5. all of the amounts must be explicit, with none missing Multiple such exchanges can coexist within a single transaction, should you need that. @@ -6837,11 +6930,8 @@ journal file. • payees - show unique payee segments of transaction descriptions • prices - show market price records • *print* - show transactions (journal entries) - • print-unique - show only transactions with unique descriptions • *register (reg)* - show postings in one or more accounts & running total - • register-match - show a recent posting that best matches a - description • stats - show journal statistics • tags - show tag names • test - run self tests @@ -6883,9 +6973,7 @@ hledger’s commands list, with a ‘+’ mark: * payees:: * prices:: * print:: -* print-unique:: * register:: -* register-match:: * rewrite:: * roi:: * stats:: @@ -7069,8 +7157,13 @@ revenues/expenses. ‘aregister’ requires one argument: the account to report on. You can write either the full account name, or a case-insensitive regular expression which will select the alphabetically first matched account. -(Eg if you have ‘assets:aaa:checking’ and ‘assets:bbb:checking’ -accounts, ‘hledger areg checking’ would select ‘assets:aaa:checking’.) + + When there are multiple matches, the alphabetically-first choice can +be surprising; eg if you have ‘assets:per:checking 1’ and +‘assets:biz:checking 2’ accounts, ‘hledger areg checking’ would select +‘assets:biz:checking 2’. It’s just a convenience to save typing, so if +in doubt, write the full account name, or a distinctive substring that +matches uniquely. Transactions involving subaccounts of this account will also be shown. ‘aregister’ ignores depth limits, so its final total will always @@ -9079,7 +9172,7 @@ known prices. Prices can be filtered by a query. Price amounts are displayed with their full precision.  -File: hledger.info, Node: print, Next: print-unique, Prev: prices, Up: PART 4 COMMANDS +File: hledger.info, Node: print, Next: register, Prev: prices, Up: PART 4 COMMANDS 24.22 print =========== @@ -9160,11 +9253,11 @@ keeping the output parseable. With ‘-B’/‘--cost’, amounts with costs are converted to cost using that price. This can be used for troubleshooting. - With ‘-m DESC’/‘--match=DESC’, print does a fuzzy search for the one -transaction whose description is most similar to DESC, also preferring -recent tranactions. DESC should contain at least two characters. If -there is no similar-enough match, no transaction will be shown and the -program exit code will be non-zero. + With ‘-m DESC’/‘--match=DESC’, print does a fuzzy search for one +recent transaction whose description is most similar to DESC. DESC +should contain at least two characters. If there is no similar-enough +match, no transaction will be shown and the program exit code will be +non-zero. With ‘--new’, hledger prints only transactions it has not seen on a previous run. This uses the same deduplication system as the ‘import’ @@ -9204,30 +9297,9 @@ $ hledger print -Ocsv zero or greater amounts under debit.)  -File: hledger.info, Node: print-unique, Next: register, Prev: print, Up: PART 4 COMMANDS +File: hledger.info, Node: register, Next: rewrite, Prev: print, Up: PART 4 COMMANDS -24.23 print-unique -================== - -print-unique -Print transactions which do not reuse an already-seen description. - - Example: - -$ cat unique.journal -1/1 test - (acct:one) 1 -2/2 test - (acct:two) 2 -$ LEDGER_FILE=unique.journal hledger print-unique -(-f option not supported) -2015/01/01 test - (acct:one) 1 - - -File: hledger.info, Node: register, Next: register-match, Prev: print-unique, Up: PART 4 COMMANDS - -24.24 register +24.23 register ============== register, reg @@ -9251,7 +9323,7 @@ $ hledger register checking 2008/06/02 save assets:bank:checking $-1 $1 2008/12/31 pay off assets:bank:checking $-1 0 - With –date2, it shows and sorts by secondary date instead. + With ‘--date2’, it shows and sorts by secondary date instead. For performance reasons, column widths are chosen based on the first 1000 lines; this means unusually wide values in later lines can cause @@ -9324,6 +9396,11 @@ these will be adjusted outward if necessary to contain a whole number of intervals. This ensures that the first and last intervals are full length and comparable to the others in the report. + With ‘-m DESC’/‘--match=DESC’, register does a fuzzy search for one +recent posting whose description is most similar to DESC. DESC should +contain at least two characters. If there is no similar-enough match, +no posting will be shown and the program exit code will be non-zero. + * Menu: * Custom register output:: @@ -9331,7 +9408,7 @@ length and comparable to the others in the report.  File: hledger.info, Node: Custom register output, Up: register -24.24.1 Custom register output +24.23.1 Custom register output ------------------------------ register uses the full terminal width by default, except on windows. @@ -9361,22 +9438,9 @@ options The output formats supported are ‘txt’, ‘csv’, and (experimental) ‘json’.  -File: hledger.info, Node: register-match, Next: rewrite, Prev: register, Up: PART 4 COMMANDS +File: hledger.info, Node: rewrite, Next: roi, Prev: register, Up: PART 4 COMMANDS -24.25 register-match -==================== - -register-match -Print the one posting whose transaction description is closest to DESC, -in the style of the register command. If there are multiple equally -good matches, it shows the most recent. Query options (options, not -arguments) can be used to restrict the search space. Helps -ledger-autosync detect already-seen transactions when importing. - - -File: hledger.info, Node: rewrite, Next: roi, Prev: register-match, Up: PART 4 COMMANDS - -24.26 rewrite +24.24 rewrite ============= rewrite @@ -9430,7 +9494,7 @@ commodity.  File: hledger.info, Node: Re-write rules in a file, Next: Diff output format, Up: rewrite -24.26.1 Re-write rules in a file +24.24.1 Re-write rules in a file -------------------------------- During the run this tool will execute so called "Automated Transactions" @@ -9468,7 +9532,7 @@ postings.  File: hledger.info, Node: Diff output format, Next: rewrite vs print --auto, Prev: Re-write rules in a file, Up: rewrite -24.26.2 Diff output format +24.24.2 Diff output format -------------------------- To use this tool for batch modification of your journal files you may @@ -9509,7 +9573,7 @@ output from ‘hledger print’.  File: hledger.info, Node: rewrite vs print --auto, Prev: Diff output format, Up: rewrite -24.26.3 rewrite vs. print –auto +24.24.3 rewrite vs. print –auto ------------------------------- This command predates print –auto, and currently does much the same @@ -9529,7 +9593,7 @@ thing, but with these differences:  File: hledger.info, Node: roi, Next: stats, Prev: rewrite, Up: PART 4 COMMANDS -24.27 roi +24.25 roi ========= roi @@ -9578,7 +9642,7 @@ display, regardless of the length of reporting interval.  File: hledger.info, Node: Spaces and special characters in --inv and --pnl, Next: Semantics of --inv and --pnl, Up: roi -24.27.1 Spaces and special characters in ‘--inv’ and +24.25.1 Spaces and special characters in ‘--inv’ and ---------------------------------------------------- ‘--pnl’ Note that ‘--inv’ and ‘--pnl’’s argument is a query, and queries @@ -9597,7 +9661,7 @@ $ hledger roi --inv="'Assets:Test 1'" --pnl="'Equity:Unrealized Profit and Loss'  File: hledger.info, Node: Semantics of --inv and --pnl, Next: IRR and TWR explained, Prev: Spaces and special characters in --inv and --pnl, Up: roi -24.27.2 Semantics of ‘--inv’ and ‘--pnl’ +24.25.2 Semantics of ‘--inv’ and ‘--pnl’ ---------------------------------------- Query supplied to ‘--inv’ has to match all transactions that are related @@ -9651,7 +9715,7 @@ postings in the example below would be classifed as:  File: hledger.info, Node: IRR and TWR explained, Prev: Semantics of --inv and --pnl, Up: roi -24.27.3 IRR and TWR explained +24.25.3 IRR and TWR explained ----------------------------- "ROI" stands for "return on investment". Traditionally this was @@ -9718,7 +9782,7 @@ your investment.  File: hledger.info, Node: stats, Next: tags, Prev: roi, Up: PART 4 COMMANDS -24.28 stats +24.26 stats =========== stats @@ -9759,7 +9823,7 @@ Throughput : 8342 txns/s  File: hledger.info, Node: tags, Next: test, Prev: stats, Up: PART 4 COMMANDS -24.29 tags +24.27 tags ========== tags @@ -9790,7 +9854,7 @@ transactions also acquire tags from their postings.  File: hledger.info, Node: test, Prev: tags, Up: PART 4 COMMANDS -24.30 test +24.28 test ========== test @@ -10243,602 +10307,594 @@ close command.  Tag Table: Node: Top210 -Node: PART 1 USER INTERFACE3967 -Ref: #part-1-user-interface4108 -Node: Options4108 -Ref: #options4227 -Node: General options4369 -Ref: #general-options4494 -Node: Command options8944 -Ref: #command-options9095 -Node: Command arguments9511 -Ref: #command-arguments9669 -Node: Special characters10571 -Ref: #special-characters10734 -Node: Single escaping shell metacharacters10897 -Ref: #single-escaping-shell-metacharacters11138 -Node: Double escaping regular expression metacharacters11773 -Ref: #double-escaping-regular-expression-metacharacters12084 -Node: Triple escaping for add-on commands12654 -Ref: #triple-escaping-for-add-on-commands12914 -Node: Less escaping13582 -Ref: #less-escaping13736 -Node: Unicode characters14074 -Ref: #unicode-characters14239 -Node: Regular expressions15671 -Ref: #regular-expressions15811 -Node: Environment17653 -Ref: #environment17764 -Node: Input19375 -Ref: #input19475 -Node: Data formats20038 -Ref: #data-formats20151 -Node: Multiple files21603 -Ref: #multiple-files21740 -Node: Strict mode22225 -Ref: #strict-mode22335 -Node: Commands23089 -Ref: #commands23190 -Node: Add-on commands23674 -Ref: #add-on-commands23776 -Node: Output24943 -Ref: #output25046 -Node: Output destination25162 -Ref: #output-destination25293 -Node: Output format25722 -Ref: #output-format25868 -Node: CSV output27440 -Ref: #csv-output27556 -Node: HTML output27661 -Ref: #html-output27799 -Node: JSON output27899 -Ref: #json-output28037 -Node: SQL output28969 -Ref: #sql-output29085 -Node: Commodity styles29608 -Ref: #commodity-styles29748 -Node: Colour30355 -Ref: #colour30473 -Node: Box-drawing30907 -Ref: #box-drawing31031 -Node: Debug output31345 -Ref: #debug-output31456 -Node: Limitations32131 -Ref: #limitations32251 -Node: Troubleshooting33026 -Ref: #troubleshooting33167 -Node: PART 2 DATA FORMATS35689 -Ref: #part-2-data-formats35836 -Node: Journal35836 -Ref: #journal35947 -Node: Journal cheatsheet36534 -Ref: #journal-cheatsheet36675 -Node: About journal format41124 -Ref: #about-journal-format41286 -Node: Comments42824 -Ref: #comments42956 -Node: Transactions43810 -Ref: #transactions43935 -Node: Dates44969 -Ref: #dates45078 -Node: Simple dates45123 -Ref: #simple-dates45241 -Node: Posting dates45778 -Ref: #posting-dates45898 -Node: Status46881 -Ref: #status46984 -Node: Code48736 -Ref: #code48841 -Node: Description49073 -Ref: #description49206 -Node: Payee and note49528 -Ref: #payee-and-note49636 -Node: Transaction comments49983 -Ref: #transaction-comments50138 -Node: Postings50509 -Ref: #postings50644 -Node: Account names51653 -Ref: #account-names51785 -Node: Amounts52297 -Ref: #amounts52414 -Node: Decimal marks digit group marks53403 -Ref: #decimal-marks-digit-group-marks53580 -Node: Commodity54604 -Ref: #commodity54793 -Node: Directives influencing number parsing and display55775 -Ref: #directives-influencing-number-parsing-and-display56036 -Node: Commodity display style56498 -Ref: #commodity-display-style56706 -Node: Rounding58917 -Ref: #rounding59037 -Node: Costs59338 -Ref: #costs59456 -Node: Other cost/lot notations61487 -Ref: #other-costlot-notations61621 -Node: Balance assertions64374 -Ref: #balance-assertions64527 -Node: Assertions and ordering65620 -Ref: #assertions-and-ordering65811 -Node: Assertions and multiple included files66513 -Ref: #assertions-and-multiple-included-files66775 -Node: Assertions and multiple -f files67283 -Ref: #assertions-and-multiple--f-files67536 -Node: Assertions and commodities67945 -Ref: #assertions-and-commodities68169 -Node: Assertions and prices69359 -Ref: #assertions-and-prices69567 -Node: Assertions and subaccounts69998 -Ref: #assertions-and-subaccounts70221 -Node: Assertions and virtual postings70563 -Ref: #assertions-and-virtual-postings70803 -Node: Assertions and auto postings70943 -Ref: #assertions-and-auto-postings71175 -Node: Assertions and precision71846 -Ref: #assertions-and-precision72030 -Node: Posting comments72297 -Ref: #posting-comments72445 -Node: Tags72830 -Ref: #tags72946 -Node: Tag values74151 -Ref: #tag-values74242 -Node: Directives75015 -Ref: #directives75135 -Node: Directive effects77003 -Ref: #directive-effects77159 -Node: Directives and multiple files80242 -Ref: #directives-and-multiple-files80422 -Node: Accounts81132 -Ref: #accounts81259 -Node: Account comments82445 -Ref: #account-comments82588 -Node: Account subdirectives83108 -Ref: #account-subdirectives83292 -Node: Account error checking83434 -Ref: #account-error-checking83625 -Node: Account display order84844 -Ref: #account-display-order85025 -Node: Account types86164 -Ref: #account-types86298 -Node: Account aliases90090 -Ref: #account-aliases90232 -Node: Basic aliases91282 -Ref: #basic-aliases91415 -Node: Regex aliases92169 -Ref: #regex-aliases92328 -Node: Combining aliases93222 -Ref: #combining-aliases93402 -Node: Aliases and multiple files94696 -Ref: #aliases-and-multiple-files94892 -Node: end aliases95477 -Ref: #end-aliases95672 -Node: Aliases can generate bad account names95821 -Ref: #aliases-can-generate-bad-account-names96061 -Node: Aliases and account types96658 -Ref: #aliases-and-account-types96852 -Node: Commodities97554 -Ref: #commodities97692 -Node: Commodity error checking100296 -Ref: #commodity-error-checking100436 -Node: Decimal mark100965 -Ref: #decimal-mark101103 -Node: Include files101504 -Ref: #include-files101646 -Node: Market prices102594 -Ref: #market-prices102730 -Node: Payees103509 -Ref: #payees103639 -Node: Periodic transactions103943 -Ref: #periodic-transactions104102 -Node: Periodic rule syntax105815 -Ref: #periodic-rule-syntax105995 -Node: Periodic rules and relative dates106470 -Ref: #periodic-rules-and-relative-dates106738 -Node: Two spaces between period expression and description!107277 -Ref: #two-spaces-between-period-expression-and-description107556 -Node: Other syntax108246 -Ref: #other-syntax108372 -Node: Auto postings109023 -Ref: #auto-postings109159 -Node: Auto postings and multiple files111655 -Ref: #auto-postings-and-multiple-files111857 -Node: Auto postings and dates112074 -Ref: #auto-postings-and-dates112346 -Node: Auto postings and transaction balancing / inferred amounts / balance assertions112521 -Ref: #auto-postings-and-transaction-balancing-inferred-amounts-balance-assertions112860 -Node: Auto posting tags113367 -Ref: #auto-posting-tags113580 -Node: Balance assignments114242 -Ref: #balance-assignments114422 -Node: Balance assignments and prices115756 -Ref: #balance-assignments-and-prices115926 -Node: Bracketed posting dates116137 -Ref: #bracketed-posting-dates116329 -Node: Default commodity116873 -Ref: #default-commodity117056 -Node: Default parent account118607 -Ref: #default-parent-account118789 -Node: Default year119498 -Ref: #default-year119658 -Node: Secondary dates120471 -Ref: #secondary-dates120628 -Node: Star comments121460 -Ref: #star-comments121622 -Node: Valuation expressions122162 -Ref: #valuation-expressions122341 -Node: Virtual postings122463 -Ref: #virtual-postings122644 -Node: Other Ledger directives124216 -Ref: #other-ledger-directives124381 -Node: CSV124951 -Ref: #csv125044 -Node: Examples126783 -Ref: #examples126879 -Node: Basic127087 -Ref: #basic127189 -Node: Bank of Ireland127735 -Ref: #bank-of-ireland127872 -Node: Amazon129340 -Ref: #amazon129460 -Node: Paypal131185 -Ref: #paypal131281 -Node: CSV rules138927 -Ref: #csv-rules139043 -Node: separator140759 -Ref: #separator140879 -Node: skip141457 -Ref: #skip141569 -Node: date-format142152 -Ref: #date-format142277 -Node: timezone143023 -Ref: #timezone143150 -Node: decimal-mark144183 -Ref: #decimal-mark-1144319 -Node: newest-first144658 -Ref: #newest-first144804 -Node: intra-day-reversed145388 -Ref: #intra-day-reversed145540 -Node: fields146037 -Ref: #fields146171 -Node: Field assignment147773 -Ref: #field-assignment147912 -Node: Field names148969 -Ref: #field-names149094 -Node: date field149473 -Ref: #date-field149591 -Node: date2 field149643 -Ref: #date2-field149784 -Node: status field149846 -Ref: #status-field149989 -Node: code field150044 -Ref: #code-field150189 -Node: description field150240 -Ref: #description-field150400 -Node: comment field150465 -Ref: #comment-field150620 -Node: account field150933 -Ref: #account-field151083 -Node: amount field151684 -Ref: #amount-field151833 -Node: currency field153425 -Ref: #currency-field153578 -Node: balance field153847 -Ref: #balance-field153979 -Node: if154367 -Ref: #if154474 -Node: Matching the whole record154875 -Ref: #matching-the-whole-record155044 -Node: Matching individual fields155871 -Ref: #matching-individual-fields156069 -Node: Combining matchers156303 -Ref: #combining-matchers156493 -Node: Rules applied on successful match156814 -Ref: #rules-applied-on-successful-match156999 -Node: if table157659 -Ref: #if-table157770 -Node: end159536 -Ref: #end159642 -Node: include159866 -Ref: #include159984 -Node: balance-type160434 -Ref: #balance-type160563 -Node: Tips161271 -Ref: #tips161383 -Node: Rapid feedback161383 -Ref: #rapid-feedback161512 -Node: Valid CSV161968 -Ref: #valid-csv162097 -Node: File Extension162851 -Ref: #file-extension163002 -Node: Reading multiple CSV files163455 -Ref: #reading-multiple-csv-files163639 -Node: Valid transactions163888 -Ref: #valid-transactions164065 -Node: Deduplicating importing164693 -Ref: #deduplicating-importing164871 -Node: Setting amounts165917 -Ref: #setting-amounts166071 -Node: Amount signs168571 -Ref: #amount-signs168722 -Node: Setting currency/commodity169461 -Ref: #setting-currencycommodity169646 -Node: Amount decimal places170838 -Ref: #amount-decimal-places171025 -Node: Referencing other fields171343 -Ref: #referencing-other-fields171537 -Node: How CSV rules are evaluated172440 -Ref: #how-csv-rules-are-evaluated172610 -Node: Timeclock174121 -Ref: #timeclock174226 -Node: Timedot176394 -Ref: #timedot176517 -Node: PART 3 REPORTING CONCEPTS181368 -Ref: #part-3-reporting-concepts181532 -Node: Time periods181532 -Ref: #time-periods181666 -Node: Report start & end date181764 -Ref: #report-start-end-date181916 -Node: Smart dates183665 -Ref: #smart-dates183818 -Node: Report intervals185740 -Ref: #report-intervals185898 -Node: Period expressions187719 -Ref: #period-expressions187861 -Node: Period expressions with a report interval189712 -Ref: #period-expressions-with-a-report-interval189946 -Node: More complex report intervals191101 -Ref: #more-complex-report-intervals191352 -Node: Intervals with custom start date192034 -Ref: #intervals-with-custom-start-date192268 -Node: Periods or dates ?193928 -Ref: #periods-or-dates194132 -Node: Events on multiple weekdays194588 -Ref: #events-on-multiple-weekdays194769 -Node: Depth195678 -Ref: #depth195780 -Node: Queries196100 -Ref: #queries196202 -Node: Query types197179 -Ref: #query-types197300 -Node: Combining query terms200640 -Ref: #combining-query-terms200817 -Node: Queries and command options201915 -Ref: #queries-and-command-options202120 -Node: Queries and account aliases202385 -Ref: #queries-and-account-aliases202590 -Node: Queries and valuation202722 -Ref: #queries-and-valuation202917 -Node: Querying with account aliases203156 -Ref: #querying-with-account-aliases203367 -Node: Querying with cost or value203509 -Ref: #querying-with-cost-or-value203686 -Node: Pivoting203995 -Ref: #pivoting204109 -Node: Generating data205597 -Ref: #generating-data205729 -Node: Forecasting206227 -Ref: #forecasting206352 -Node: Budgeting209239 -Ref: #budgeting209359 -Node: Cost reporting209630 -Ref: #cost-reporting209758 -Node: -B Convert to cost210873 -Ref: #b-convert-to-cost211029 -Node: Equity conversion postings212437 -Ref: #equity-conversion-postings212651 -Node: Inferring equity postings from cost213280 -Ref: #inferring-equity-postings-from-cost213529 -Node: Inferring cost from equity postings214503 -Ref: #inferring-cost-from-equity-postings214751 -Node: When to infer cost/equity216794 -Ref: #when-to-infer-costequity217012 -Node: How to record conversions217420 -Ref: #how-to-record-conversions217612 -Node: Conversion with implicit cost217903 -Ref: #conversion-with-implicit-cost218108 -Node: Conversion with explicit cost219013 -Ref: #conversion-with-explicit-cost219258 -Node: Conversion with equity postings219685 -Ref: #conversion-with-equity-postings219954 -Node: Conversion with equity postings and explicit cost220791 -Ref: #conversion-with-equity-postings-and-explicit-cost221058 -Node: Cost tips221681 -Ref: #cost-tips221807 -Node: Valuation222566 -Ref: #valuation222690 -Node: -V Value223476 -Ref: #v-value223602 -Node: -X Value in specified commodity223801 -Ref: #x-value-in-specified-commodity223996 -Node: Valuation date224153 -Ref: #valuation-date224324 -Node: Finding market price224763 -Ref: #finding-market-price224968 -Node: --infer-market-prices market prices from transactions226148 -Ref: #infer-market-prices-market-prices-from-transactions226426 -Node: Valuation commodity229272 -Ref: #valuation-commodity229485 -Node: Simple valuation examples230730 -Ref: #simple-valuation-examples230928 -Node: --value Flexible valuation231591 -Ref: #value-flexible-valuation231797 -Node: More valuation examples233483 -Ref: #more-valuation-examples233692 -Node: Interaction of valuation and queries235703 -Ref: #interaction-of-valuation-and-queries235944 -Node: Effect of valuation on reports236424 -Ref: #effect-of-valuation-on-reports236621 -Node: PART 4 COMMANDS244380 -Ref: #part-4-commands244523 -Node: accounts247444 -Ref: #accounts-1247551 -Node: activity249535 -Ref: #activity249654 -Node: add250037 -Ref: #add250147 -Node: aregister253012 -Ref: #aregister253133 -Node: aregister and custom posting dates255888 -Ref: #aregister-and-custom-posting-dates256054 -Node: balance256622 -Ref: #balance256748 -Node: balance features257756 -Ref: #balance-features257896 -Node: Simple balance report260026 -Ref: #simple-balance-report260211 -Node: Balance report line format261856 -Ref: #balance-report-line-format262058 -Node: Filtered balance report264308 -Ref: #filtered-balance-report264500 -Node: List or tree mode264827 -Ref: #list-or-tree-mode264995 -Node: Depth limiting266370 -Ref: #depth-limiting266536 -Node: Dropping top-level accounts267153 -Ref: #dropping-top-level-accounts267353 -Node: Showing declared accounts267667 -Ref: #showing-declared-accounts267866 -Node: Sorting by amount268407 -Ref: #sorting-by-amount268574 -Node: Percentages269264 -Ref: #percentages269423 -Node: Multi-period balance report269993 -Ref: #multi-period-balance-report270193 -Node: Balance change end balance272586 -Ref: #balance-change-end-balance272795 -Node: Balance report types274243 -Ref: #balance-report-types274424 -Node: Calculation type274940 -Ref: #calculation-type275095 -Node: Accumulation type275626 -Ref: #accumulation-type275806 -Node: Valuation type276734 -Ref: #valuation-type276922 -Node: Combining balance report types277989 -Ref: #combining-balance-report-types278183 -Node: Budget report280087 -Ref: #budget-report280239 -Node: Budget report start date285541 -Ref: #budget-report-start-date285719 -Node: Budgets and subaccounts287081 -Ref: #budgets-and-subaccounts287288 -Node: Selecting budget goals290774 -Ref: #selecting-budget-goals290946 -Node: Data layout291988 -Ref: #data-layout292138 -Node: Useful balance reports300079 -Ref: #useful-balance-reports300229 -Node: balancesheet301382 -Ref: #balancesheet301527 -Node: balancesheetequity302901 -Ref: #balancesheetequity303059 -Node: cashflow304516 -Ref: #cashflow304647 -Node: check306137 -Ref: #check306249 -Node: Basic checks307061 -Ref: #basic-checks307179 -Node: Strict checks307717 -Ref: #strict-checks307858 -Node: Other checks308299 -Ref: #other-checks308439 -Node: Custom checks308950 -Ref: #custom-checks309105 -Node: More about specific checks309526 -Ref: #more-about-specific-checks309686 -Node: close310418 -Ref: #close310529 -Node: close and costs312669 -Ref: #close-and-costs312795 -Node: close date313187 -Ref: #close-date313370 -Node: Example close asset/liability accounts for file transition314162 -Ref: #example-close-assetliability-accounts-for-file-transition314463 -Node: Hiding opening/closing transactions315322 -Ref: #hiding-openingclosing-transactions315593 -Node: close and balance assertions316980 -Ref: #close-and-balance-assertions317238 -Node: Example close revenue/expense accounts to retained earnings318614 -Ref: #example-close-revenueexpense-accounts-to-retained-earnings318892 -Node: codes319802 -Ref: #codes319919 -Node: commodities320801 -Ref: #commodities-1320937 -Node: descriptions321019 -Ref: #descriptions321156 -Node: diff321460 -Ref: #diff321575 -Node: files322626 -Ref: #files322735 -Node: help322882 -Ref: #help322991 -Node: import323986 -Ref: #import324109 -Node: Deduplication325224 -Ref: #deduplication325349 -Node: Import testing327271 -Ref: #import-testing327436 -Node: Importing balance assignments327928 -Ref: #importing-balance-assignments328134 -Node: Commodity display styles328791 -Ref: #commodity-display-styles328964 -Node: incomestatement329093 -Ref: #incomestatement329235 -Node: notes330613 -Ref: #notes330735 -Node: payees331103 -Ref: #payees-1331218 -Node: prices331750 -Ref: #prices331865 -Node: print332174 -Ref: #print332293 -Node: print-unique337779 -Ref: #print-unique337914 -Node: register338199 -Ref: #register338335 -Node: Custom register output343153 -Ref: #custom-register-output343284 -Node: register-match344659 -Ref: #register-match344802 -Node: rewrite345153 -Ref: #rewrite345277 -Node: Re-write rules in a file347197 -Ref: #re-write-rules-in-a-file347360 -Node: Diff output format348513 -Ref: #diff-output-format348696 -Node: rewrite vs print --auto349808 -Ref: #rewrite-vs.-print---auto349970 -Node: roi350544 -Ref: #roi350651 -Node: Spaces and special characters in --inv and --pnl352416 -Ref: #spaces-and-special-characters-in---inv-and---pnl352664 -Node: Semantics of --inv and --pnl353162 -Ref: #semantics-of---inv-and---pnl353409 -Node: IRR and TWR explained355287 -Ref: #irr-and-twr-explained355447 -Node: stats358559 -Ref: #stats358667 -Node: tags360070 -Ref: #tags-1360177 -Node: test361199 -Ref: #test361292 -Node: PART 5 COMMON TASKS362047 -Ref: #part-5-common-tasks362180 -Node: Getting help362454 -Ref: #getting-help362595 -Node: Constructing command lines363359 -Ref: #constructing-command-lines363560 -Node: Starting a journal file364241 -Ref: #starting-a-journal-file364448 -Node: Setting opening balances365646 -Ref: #setting-opening-balances365851 -Node: Recording transactions369004 -Ref: #recording-transactions369193 -Node: Reconciling369749 -Ref: #reconciling369901 -Node: Reporting372214 -Ref: #reporting372363 -Node: Migrating to a new file376352 -Ref: #migrating-to-a-new-file376509 +Node: PART 1 USER INTERFACE3947 +Ref: #part-1-user-interface4088 +Node: Options4088 +Ref: #options4207 +Node: General options4349 +Ref: #general-options4474 +Node: Command options8924 +Ref: #command-options9075 +Node: Command arguments9491 +Ref: #command-arguments9649 +Node: Special characters10551 +Ref: #special-characters10714 +Node: Single escaping shell metacharacters10877 +Ref: #single-escaping-shell-metacharacters11118 +Node: Double escaping regular expression metacharacters11753 +Ref: #double-escaping-regular-expression-metacharacters12064 +Node: Triple escaping for add-on commands12634 +Ref: #triple-escaping-for-add-on-commands12894 +Node: Less escaping13562 +Ref: #less-escaping13716 +Node: Unicode characters14054 +Ref: #unicode-characters14219 +Node: Regular expressions15651 +Ref: #regular-expressions15791 +Node: Environment17633 +Ref: #environment17744 +Node: Input19355 +Ref: #input19455 +Node: Data formats20018 +Ref: #data-formats20131 +Node: Multiple files21583 +Ref: #multiple-files21720 +Node: Strict mode22205 +Ref: #strict-mode22315 +Node: Commands23069 +Ref: #commands23170 +Node: Add-on commands23654 +Ref: #add-on-commands23756 +Node: Output24923 +Ref: #output25026 +Node: Output destination25142 +Ref: #output-destination25273 +Node: Output format25702 +Ref: #output-format25848 +Node: CSV output27420 +Ref: #csv-output27536 +Node: HTML output27641 +Ref: #html-output27779 +Node: JSON output27879 +Ref: #json-output28017 +Node: SQL output28949 +Ref: #sql-output29065 +Node: Commodity styles29588 +Ref: #commodity-styles29728 +Node: Colour30335 +Ref: #colour30453 +Node: Box-drawing30887 +Ref: #box-drawing31011 +Node: Debug output31325 +Ref: #debug-output31436 +Node: Limitations32111 +Ref: #limitations32231 +Node: Troubleshooting33006 +Ref: #troubleshooting33147 +Node: PART 2 DATA FORMATS35669 +Ref: #part-2-data-formats35816 +Node: Journal35816 +Ref: #journal35927 +Node: Journal cheatsheet36514 +Ref: #journal-cheatsheet36655 +Node: About journal format41104 +Ref: #about-journal-format41266 +Node: Comments42804 +Ref: #comments42936 +Node: Transactions43790 +Ref: #transactions43915 +Node: Dates44949 +Ref: #dates45058 +Node: Simple dates45103 +Ref: #simple-dates45221 +Node: Posting dates45758 +Ref: #posting-dates45878 +Node: Status46861 +Ref: #status46964 +Node: Code48716 +Ref: #code48821 +Node: Description49053 +Ref: #description49186 +Node: Payee and note49508 +Ref: #payee-and-note49616 +Node: Transaction comments49963 +Ref: #transaction-comments50118 +Node: Postings50489 +Ref: #postings50624 +Node: Account names51633 +Ref: #account-names51765 +Node: Amounts53467 +Ref: #amounts53584 +Node: Decimal marks digit group marks54573 +Ref: #decimal-marks-digit-group-marks54750 +Node: Commodity55774 +Ref: #commodity55963 +Node: Directives influencing number parsing and display56945 +Ref: #directives-influencing-number-parsing-and-display57206 +Node: Commodity display style57668 +Ref: #commodity-display-style57876 +Node: Rounding60087 +Ref: #rounding60207 +Node: Costs60508 +Ref: #costs60626 +Node: Other cost/lot notations62657 +Ref: #other-costlot-notations62791 +Node: Balance assertions65544 +Ref: #balance-assertions65697 +Node: Assertions and ordering66790 +Ref: #assertions-and-ordering66981 +Node: Assertions and multiple included files67683 +Ref: #assertions-and-multiple-included-files67945 +Node: Assertions and multiple -f files68453 +Ref: #assertions-and-multiple--f-files68706 +Node: Assertions and commodities69115 +Ref: #assertions-and-commodities69339 +Node: Assertions and prices70529 +Ref: #assertions-and-prices70737 +Node: Assertions and subaccounts71168 +Ref: #assertions-and-subaccounts71391 +Node: Assertions and virtual postings71733 +Ref: #assertions-and-virtual-postings71973 +Node: Assertions and auto postings72113 +Ref: #assertions-and-auto-postings72345 +Node: Assertions and precision73016 +Ref: #assertions-and-precision73200 +Node: Posting comments73467 +Ref: #posting-comments73615 +Node: Tags74000 +Ref: #tags74116 +Node: Tag values75321 +Ref: #tag-values75412 +Node: Directives76185 +Ref: #directives76305 +Node: Directive effects78173 +Ref: #directive-effects78329 +Node: Directives and multiple files81412 +Ref: #directives-and-multiple-files81592 +Node: Accounts82302 +Ref: #accounts82429 +Node: Account comments83849 +Ref: #account-comments83992 +Node: Account subdirectives84512 +Ref: #account-subdirectives84696 +Node: Account error checking84838 +Ref: #account-error-checking85029 +Node: Account display order86248 +Ref: #account-display-order86429 +Node: Account types87568 +Ref: #account-types87702 +Node: Account aliases91494 +Ref: #account-aliases91636 +Node: Basic aliases92686 +Ref: #basic-aliases92819 +Node: Regex aliases93573 +Ref: #regex-aliases93732 +Node: Combining aliases94626 +Ref: #combining-aliases94806 +Node: Aliases and multiple files96100 +Ref: #aliases-and-multiple-files96296 +Node: end aliases96881 +Ref: #end-aliases97076 +Node: Aliases can generate bad account names97225 +Ref: #aliases-can-generate-bad-account-names97465 +Node: Aliases and account types98062 +Ref: #aliases-and-account-types98256 +Node: Commodities98958 +Ref: #commodities99096 +Node: Commodity error checking101700 +Ref: #commodity-error-checking101840 +Node: Decimal mark102369 +Ref: #decimal-mark102507 +Node: Include files102908 +Ref: #include-files103050 +Node: Market prices103998 +Ref: #market-prices104134 +Node: Payees104913 +Ref: #payees105043 +Node: Periodic transactions105347 +Ref: #periodic-transactions105506 +Node: Periodic rule syntax107219 +Ref: #periodic-rule-syntax107399 +Node: Periodic rules and relative dates107874 +Ref: #periodic-rules-and-relative-dates108142 +Node: Two spaces between period expression and description!108681 +Ref: #two-spaces-between-period-expression-and-description108960 +Node: Other syntax109650 +Ref: #other-syntax109776 +Node: Auto postings110427 +Ref: #auto-postings110563 +Node: Auto postings and multiple files113059 +Ref: #auto-postings-and-multiple-files113261 +Node: Auto postings and dates113478 +Ref: #auto-postings-and-dates113750 +Node: Auto postings and transaction balancing / inferred amounts / balance assertions113925 +Ref: #auto-postings-and-transaction-balancing-inferred-amounts-balance-assertions114264 +Node: Auto posting tags114771 +Ref: #auto-posting-tags114984 +Node: Balance assignments115646 +Ref: #balance-assignments115826 +Node: Balance assignments and prices117160 +Ref: #balance-assignments-and-prices117330 +Node: Bracketed posting dates117541 +Ref: #bracketed-posting-dates117733 +Node: Default commodity118277 +Ref: #default-commodity118460 +Node: Default parent account120011 +Ref: #default-parent-account120193 +Node: Default year120902 +Ref: #default-year121062 +Node: Secondary dates121875 +Ref: #secondary-dates122032 +Node: Star comments122864 +Ref: #star-comments123026 +Node: Valuation expressions123566 +Ref: #valuation-expressions123745 +Node: Virtual postings123867 +Ref: #virtual-postings124048 +Node: Other Ledger directives125620 +Ref: #other-ledger-directives125785 +Node: CSV126355 +Ref: #csv126448 +Node: CSV rules cheatsheet128551 +Ref: #csv-rules-cheatsheet128683 +Node: separator130451 +Ref: #separator130582 +Node: skip131160 +Ref: #skip131272 +Node: date-format131855 +Ref: #date-format131980 +Node: timezone132726 +Ref: #timezone132853 +Node: newest-first133886 +Ref: #newest-first134028 +Node: intra-day-reversed134612 +Ref: #intra-day-reversed134770 +Node: decimal-mark135267 +Ref: #decimal-mark-1135412 +Node: fields list135751 +Ref: #fields-list135892 +Node: Field assignment137615 +Ref: #field-assignment137759 +Node: Field names138808 +Ref: #field-names138939 +Node: date field140160 +Ref: #date-field140278 +Node: date2 field140330 +Ref: #date2-field140471 +Node: status field140533 +Ref: #status-field140676 +Node: code field140731 +Ref: #code-field140876 +Node: description field140927 +Ref: #description-field141087 +Node: comment field141152 +Ref: #comment-field141307 +Node: account field141620 +Ref: #account-field141770 +Node: amount field142366 +Ref: #amount-field142515 +Node: currency field144620 +Ref: #currency-field144773 +Node: balance field145042 +Ref: #balance-field145174 +Node: if block145562 +Ref: #if-block145687 +Node: Matchers147111 +Ref: #matchers147225 +Node: if table148755 +Ref: #if-table148881 +Node: balance-type150317 +Ref: #balance-type150450 +Node: include151158 +Ref: #include151289 +Node: Working with CSV151739 +Ref: #working-with-csv151886 +Node: Rapid feedback152257 +Ref: #rapid-feedback152390 +Node: Valid CSV152846 +Ref: #valid-csv152992 +Node: File Extension153746 +Ref: #file-extension153919 +Node: Reading CSV from standard input154509 +Ref: #reading-csv-from-standard-input154733 +Node: Reading multiple CSV files154899 +Ref: #reading-multiple-csv-files155117 +Node: Valid transactions155366 +Ref: #valid-transactions155560 +Node: Deduplicating importing156188 +Ref: #deduplicating-importing156383 +Node: Setting amounts157429 +Ref: #setting-amounts157600 +Node: Amount signs160123 +Ref: #amount-signs160291 +Node: Setting currency/commodity161030 +Ref: #setting-currencycommodity161234 +Node: Amount decimal places162426 +Ref: #amount-decimal-places162632 +Node: Referencing other fields162950 +Ref: #referencing-other-fields163163 +Node: How CSV rules are evaluated164066 +Ref: #how-csv-rules-are-evaluated164283 +Node: Well factored rules165796 +Ref: #well-factored-rules165964 +Node: CSV rules examples166302 +Ref: #csv-rules-examples166437 +Node: Bank of Ireland166502 +Ref: #bank-of-ireland166639 +Node: Coinbase168107 +Ref: #coinbase168245 +Node: Amazon169298 +Ref: #amazon169423 +Node: Paypal171148 +Ref: #paypal171256 +Node: Timeclock178902 +Ref: #timeclock179007 +Node: Timedot181175 +Ref: #timedot181298 +Node: PART 3 REPORTING CONCEPTS186149 +Ref: #part-3-reporting-concepts186313 +Node: Time periods186313 +Ref: #time-periods186447 +Node: Report start & end date186545 +Ref: #report-start-end-date186697 +Node: Smart dates188446 +Ref: #smart-dates188599 +Node: Report intervals190521 +Ref: #report-intervals190679 +Node: Period expressions192500 +Ref: #period-expressions192642 +Node: Period expressions with a report interval194493 +Ref: #period-expressions-with-a-report-interval194727 +Node: More complex report intervals195882 +Ref: #more-complex-report-intervals196133 +Node: Intervals with custom start date196815 +Ref: #intervals-with-custom-start-date197049 +Node: Periods or dates ?198709 +Ref: #periods-or-dates198913 +Node: Events on multiple weekdays199369 +Ref: #events-on-multiple-weekdays199550 +Node: Depth200459 +Ref: #depth200561 +Node: Queries200881 +Ref: #queries200983 +Node: Query types201960 +Ref: #query-types202081 +Node: Combining query terms205421 +Ref: #combining-query-terms205598 +Node: Queries and command options206696 +Ref: #queries-and-command-options206901 +Node: Queries and account aliases207166 +Ref: #queries-and-account-aliases207371 +Node: Queries and valuation207503 +Ref: #queries-and-valuation207698 +Node: Querying with account aliases207937 +Ref: #querying-with-account-aliases208148 +Node: Querying with cost or value208290 +Ref: #querying-with-cost-or-value208467 +Node: Pivoting208776 +Ref: #pivoting208890 +Node: Generating data210378 +Ref: #generating-data210510 +Node: Forecasting211008 +Ref: #forecasting211133 +Node: Budgeting214020 +Ref: #budgeting214140 +Node: Cost reporting214411 +Ref: #cost-reporting214539 +Node: -B Convert to cost215654 +Ref: #b-convert-to-cost215810 +Node: Equity conversion postings217218 +Ref: #equity-conversion-postings217432 +Node: Inferring equity postings from cost218061 +Ref: #inferring-equity-postings-from-cost218310 +Node: Inferring cost from equity postings219284 +Ref: #inferring-cost-from-equity-postings219532 +Node: When to infer cost/equity221635 +Ref: #when-to-infer-costequity221853 +Node: How to record conversions222261 +Ref: #how-to-record-conversions222453 +Node: Conversion with implicit cost222744 +Ref: #conversion-with-implicit-cost222949 +Node: Conversion with explicit cost223854 +Ref: #conversion-with-explicit-cost224099 +Node: Conversion with equity postings224526 +Ref: #conversion-with-equity-postings224795 +Node: Conversion with equity postings and explicit cost225632 +Ref: #conversion-with-equity-postings-and-explicit-cost225899 +Node: Cost tips226522 +Ref: #cost-tips226648 +Node: Valuation227407 +Ref: #valuation227531 +Node: -V Value228317 +Ref: #v-value228443 +Node: -X Value in specified commodity228642 +Ref: #x-value-in-specified-commodity228837 +Node: Valuation date228994 +Ref: #valuation-date229165 +Node: Finding market price229604 +Ref: #finding-market-price229809 +Node: --infer-market-prices market prices from transactions230989 +Ref: #infer-market-prices-market-prices-from-transactions231267 +Node: Valuation commodity234113 +Ref: #valuation-commodity234326 +Node: Simple valuation examples235571 +Ref: #simple-valuation-examples235769 +Node: --value Flexible valuation236432 +Ref: #value-flexible-valuation236638 +Node: More valuation examples238324 +Ref: #more-valuation-examples238533 +Node: Interaction of valuation and queries240544 +Ref: #interaction-of-valuation-and-queries240785 +Node: Effect of valuation on reports241265 +Ref: #effect-of-valuation-on-reports241462 +Node: PART 4 COMMANDS249221 +Ref: #part-4-commands249364 +Node: accounts252096 +Ref: #accounts-1252203 +Node: activity254187 +Ref: #activity254306 +Node: add254689 +Ref: #add254799 +Node: aregister257664 +Ref: #aregister257785 +Node: aregister and custom posting dates260770 +Ref: #aregister-and-custom-posting-dates260936 +Node: balance261504 +Ref: #balance261630 +Node: balance features262638 +Ref: #balance-features262778 +Node: Simple balance report264908 +Ref: #simple-balance-report265093 +Node: Balance report line format266738 +Ref: #balance-report-line-format266940 +Node: Filtered balance report269190 +Ref: #filtered-balance-report269382 +Node: List or tree mode269709 +Ref: #list-or-tree-mode269877 +Node: Depth limiting271252 +Ref: #depth-limiting271418 +Node: Dropping top-level accounts272035 +Ref: #dropping-top-level-accounts272235 +Node: Showing declared accounts272549 +Ref: #showing-declared-accounts272748 +Node: Sorting by amount273289 +Ref: #sorting-by-amount273456 +Node: Percentages274146 +Ref: #percentages274305 +Node: Multi-period balance report274875 +Ref: #multi-period-balance-report275075 +Node: Balance change end balance277468 +Ref: #balance-change-end-balance277677 +Node: Balance report types279125 +Ref: #balance-report-types279306 +Node: Calculation type279822 +Ref: #calculation-type279977 +Node: Accumulation type280508 +Ref: #accumulation-type280688 +Node: Valuation type281616 +Ref: #valuation-type281804 +Node: Combining balance report types282871 +Ref: #combining-balance-report-types283065 +Node: Budget report284969 +Ref: #budget-report285121 +Node: Budget report start date290423 +Ref: #budget-report-start-date290601 +Node: Budgets and subaccounts291963 +Ref: #budgets-and-subaccounts292170 +Node: Selecting budget goals295656 +Ref: #selecting-budget-goals295828 +Node: Data layout296870 +Ref: #data-layout297020 +Node: Useful balance reports304961 +Ref: #useful-balance-reports305111 +Node: balancesheet306264 +Ref: #balancesheet306409 +Node: balancesheetequity307783 +Ref: #balancesheetequity307941 +Node: cashflow309398 +Ref: #cashflow309529 +Node: check311019 +Ref: #check311131 +Node: Basic checks311943 +Ref: #basic-checks312061 +Node: Strict checks312599 +Ref: #strict-checks312740 +Node: Other checks313181 +Ref: #other-checks313321 +Node: Custom checks313832 +Ref: #custom-checks313987 +Node: More about specific checks314408 +Ref: #more-about-specific-checks314568 +Node: close315300 +Ref: #close315411 +Node: close and costs317551 +Ref: #close-and-costs317677 +Node: close date318069 +Ref: #close-date318252 +Node: Example close asset/liability accounts for file transition319044 +Ref: #example-close-assetliability-accounts-for-file-transition319345 +Node: Hiding opening/closing transactions320204 +Ref: #hiding-openingclosing-transactions320475 +Node: close and balance assertions321862 +Ref: #close-and-balance-assertions322120 +Node: Example close revenue/expense accounts to retained earnings323496 +Ref: #example-close-revenueexpense-accounts-to-retained-earnings323774 +Node: codes324684 +Ref: #codes324801 +Node: commodities325683 +Ref: #commodities-1325819 +Node: descriptions325901 +Ref: #descriptions326038 +Node: diff326342 +Ref: #diff326457 +Node: files327508 +Ref: #files327617 +Node: help327764 +Ref: #help327873 +Node: import328868 +Ref: #import328991 +Node: Deduplication330106 +Ref: #deduplication330231 +Node: Import testing332153 +Ref: #import-testing332318 +Node: Importing balance assignments332810 +Ref: #importing-balance-assignments333016 +Node: Commodity display styles333673 +Ref: #commodity-display-styles333846 +Node: incomestatement333975 +Ref: #incomestatement334117 +Node: notes335495 +Ref: #notes335617 +Node: payees335985 +Ref: #payees-1336100 +Node: prices336632 +Ref: #prices336747 +Node: print337056 +Ref: #print337171 +Node: register342623 +Ref: #register342745 +Node: Custom register output347858 +Ref: #custom-register-output347989 +Node: rewrite349364 +Ref: #rewrite349482 +Node: Re-write rules in a file351402 +Ref: #re-write-rules-in-a-file351565 +Node: Diff output format352718 +Ref: #diff-output-format352901 +Node: rewrite vs print --auto354013 +Ref: #rewrite-vs.-print---auto354175 +Node: roi354749 +Ref: #roi354856 +Node: Spaces and special characters in --inv and --pnl356621 +Ref: #spaces-and-special-characters-in---inv-and---pnl356869 +Node: Semantics of --inv and --pnl357367 +Ref: #semantics-of---inv-and---pnl357614 +Node: IRR and TWR explained359492 +Ref: #irr-and-twr-explained359652 +Node: stats362764 +Ref: #stats362872 +Node: tags364275 +Ref: #tags-1364382 +Node: test365404 +Ref: #test365497 +Node: PART 5 COMMON TASKS366252 +Ref: #part-5-common-tasks366385 +Node: Getting help366659 +Ref: #getting-help366800 +Node: Constructing command lines367564 +Ref: #constructing-command-lines367765 +Node: Starting a journal file368446 +Ref: #starting-a-journal-file368653 +Node: Setting opening balances369851 +Ref: #setting-opening-balances370056 +Node: Recording transactions373209 +Ref: #recording-transactions373398 +Node: Reconciling373954 +Ref: #reconciling374106 +Node: Reporting376419 +Ref: #reporting376568 +Node: Migrating to a new file380557 +Ref: #migrating-to-a-new-file380714  End Tag Table diff --git a/hledger/hledger.txt b/hledger/hledger.txt index 91f391b12..5a679e1e4 100644 --- a/hledger/hledger.txt +++ b/hledger/hledger.txt @@ -25,8 +25,8 @@ INTRODUCTION use hledger productively, but when you have a question about function- ality, this doc should answer it. It is detailed, so do skip ahead or skim when needed. You can read it on hledger.org, or as an info manual - or man page on your system (each has benefits). You can also get it - from hledger itself with + or man page on your system. You can also get it from hledger itself + with hledger --man, hledger --info or hledger help [TOPIC]. The main function of the hledger CLI is to read plain text files @@ -1161,16 +1161,50 @@ Journal amount, the amount will be considered part of the account name. Account names - Account names typically have several parts separated by a full colon, - from which hledger derives a hierarchical chart of accounts. They can - be anything you like, but in finance there are traditionally five top- - level accounts: assets, liabilities, revenue, expenses, and equity. + Accounts are the main way of categorising things in hledger. As in + Double Entry Bookkeeping, they can represent real world accounts (such + as a bank account), or more abstract categories such as "money borrowed + from Frank" or "money spent on electricity". - Account names may contain single spaces, eg: assets:accounts receiv- - able. Because of this, they must always be followed by two or more - spaces (or newline). + You can use any account names you like, but we usually start with the + traditional accounting categories, which in english are assets, liabil- + ities, equity, revenues, expenses. (You might see these referred to as + A, L, E, R, X for short.) - Account names can be aliased. + For more precise reporting, we usually divide the top level accounts + into more detailed subaccounts, by writing a full colon between account + name parts. For example, from the account names assets:bank:checking + and expenses:food, hledger will infer this hierarchy of five accounts: + + assets + assets:bank + assets:bank:checking + expenses + expenses:food + + Shown as an outline, the hierarchical tree structure is more clear: + + assets + bank + checking + expenses + food + + hledger reports can summarise the account tree to any depth, so you can + go as deep as you like with subcategories, but keeping your account + names relatively simple may be best when starting out. + + Account names may be capitalised or not; they may contain letters, num- + bers, symbols, or single spaces. Note, when an account name and an + amount are written on the same line, they must be separated by two or + more spaces (or tabs). + + Parentheses or brackets enclosing the full account name indicate vir- + tual postings, described below. Parentheses or brackets internal to + the account name have no special meaning. + + Account names can be altered temporarily or permanently by account + aliases. Amounts After the account name, there is usually an amount. (Important: @@ -1703,6 +1737,7 @@ Journal GENERATING DATA: Generate recurring transactions or budget ~ goals + Generate extra postings on transactions = CHECKING FOR ERRORS: Define valid entities to provide more error account, commodity, payee @@ -1745,7 +1780,6 @@ Journal were written inline. Command line alternative: multiple -f/--file payee Declares a payee name, for checking all entries in all files. N - P Declares the market price of a commodity on some date, for value N reports. ~ Declares a periodic transaction rule that generates future N @@ -1812,6 +1846,12 @@ Journal account assets:bank:checking + Note, however, that accounts declared in account directives are not + allowed to have surrounding brackets and parentheses, unlike accounts + used in postings. So the following journal will not parse: + + account (assets:bank:checking) + Account comments Text following two or more spaces and ; at the end of an account direc- tive line, and/or following ; on indented lines immediately below it, @@ -2776,6 +2816,7 @@ Journal See also https://hledger.org/ledger.html for a detailed hledger/Ledger syntax comparison. + CSV hledger can read CSV files (Character Separated Value - usually comma, semicolon, or tab) containing dated records, automatically converting @@ -2783,9 +2824,9 @@ CSV (To learn about writing CSV, see CSV output.) - Note, for best error messages when reading CSV/TSV/SSV files, make sure - they have a corresponding .csv, .tsv or .ssv file extension or use a - hledger file prefix (see File Extension below). + For best error messages when reading CSV/TSV/SSV files, make sure they + have a corresponding .csv, .tsv or .ssv file extension or use a hledger + file prefix (see File Extension below). Each CSV file must be described by a corresponding rules file. This contains rules describing the CSV data (header line, fields lay- @@ -2800,16 +2841,8 @@ CSV file is found, hledger will create a sample rules file, which you'll need to adjust. - There's an introductory Importing CSV data tutorial on hledger.org. - - Examples - Here are some sample hledger CSV rules files. See also the full col- - lection at: - https://github.com/simonmichael/hledger/tree/master/examples/csv - - Basic - At minimum, the rules file must identify the date and amount fields, - and often it also specifies the date format and how many header lines + At minimum, the rules file must identify the date and amount fields, + and often it also specifies the date format and how many header lines there are. Here's a simple CSV file and a rules file for it: Date, Description, Id, Amount @@ -2817,7 +2850,7 @@ CSV # basic.csv.rules skip 1 - fields date, description, _, amount + fields date, description, , amount date-format %d/%m/%Y $ hledger print -f basic.csv @@ -2825,8 +2858,866 @@ CSV expenses:unknown 10.23 income:unknown -10.23 - Default account names are chosen, since we didn't set them. + There's an introductory Importing CSV data tutorial on hledger.org, and + more CSV rules examples below, and a larger collection at + https://github.com/simonmichael/hledger/tree/master/examples/csv. + CSV rules cheatsheet + The following kinds of rule can appear in the rules file, in any order. + (Blank lines and lines beginning with # or ; or * are ignored.) + + + separator declare the field separator, instead of rely- + ing on file extension + skip skip one or more header lines at start of file + date-format declare how to parse CSV dates/date-times + timezone declare the time zone of ambiguous CSV date- + times + newest-first improve txn order when: there are multiple + records, newest first, all with the same date + intra-day-reversed improve txn order when: same-day txns are in + opposite order to the overall file + decimal-mark declare the decimal mark used in CSV amounts, + when ambiguous + fields list name CSV fields for easy reference, and + optionally assign their values to hledger + fields + Field assignment assign a CSV value or interpolated text value + to a hledger field + if block conditionally assign values to hledger fields, + or skip a record or end (skip rest of file) + if table conditionally assign values to hledger fields, + using compact syntax + balance-type select which type of balance asser- + tions/assignments to generate + include inline another CSV rules file + + Working with CSV tips can be found below, including How CSV rules are + evaluated. + + separator + You can use the separator rule to read other kinds of character-sepa- + rated data. The argument is any single separator character, or the + words tab or space (case insensitive). Eg, for comma-separated values + (CSV): + + separator , + + or for semicolon-separated values (SSV): + + separator ; + + or for tab-separated values (TSV): + + separator TAB + + If the input file has a .csv, .ssv or .tsv file extension (or a csv:, + ssv:, tsv: prefix), the appropriate separator will be inferred automat- + ically, and you won't need this rule. + + skip + skip N + + The word skip followed by a number (or no number, meaning 1) tells + hledger to ignore this many non-empty lines at the start of the input + data. (Empty/blank lines are skipped automatically, so you don't need + to count those.) You'll need this whenever your CSV data contains + header lines. Header lines skipped in this way are ignored, and not + parsed as CSV. + + skip can also be used inside if blocks (described below), to skip indi- + vidual data records. Note records skipped in this way are still + required to be valid CSV, even though otherwise ignored. + + date-format + date-format DATEFMT + + This is a helper for the date (and date2) fields. If your CSV dates + are not formatted like YYYY-MM-DD, YYYY/MM/DD or YYYY.MM.DD, you'll + need to add a date-format rule describing them with a strptime-style + date parsing pattern - see https://hackage.haskell.org/pack- + age/time/docs/Data-Time-Format.html#v:formatTime. The pattern must + parse the CSV date value completely. Some examples: + + # MM/DD/YY + date-format %m/%d/%y + + # D/M/YYYY + # The - makes leading zeros optional. + date-format %-d/%-m/%Y + + # YYYY-Mmm-DD + date-format %Y-%h-%d + + # M/D/YYYY HH:MM AM some other junk + # Note the time and junk must be fully parsed, though only the date is used. + date-format %-m/%-d/%Y %l:%M %p some other junk + + timezone + timezone TIMEZONE + + When CSV contains date-times that are implicitly in some time zone + other than yours, but containing no explicit time zone information, you + can use this rule to declare the CSV's native time zone, which helps + prevent off-by-one dates. + + When the CSV date-times do contain time zone information, you don't + need this rule; instead, use %Z in date-format (or %z, %EZ, %Ez; see + the formatTime link above). + + In either of these cases, hledger will do a time-zone-aware conversion, + localising the CSV date-times to your current system time zone. If you + prefer to localise to some other time zone, eg for reproducibility, you + can (on unix at least) set the output timezone with the TZ environment + variable, eg: + + $ TZ=-1000 hledger print -f foo.csv # or TZ=-1000 hledger import foo.csv + + timezone currently does not understand timezone names, except "UTC", + "GMT", "EST", "EDT", "CST", "CDT", "MST", "MDT", "PST", or "PDT". For + others, use numeric format: +HHMM or -HHMM. + + newest-first + hledger tries to ensure that the generated transactions will be ordered + chronologically, including intra-day transactions. Usually it can + auto-detect how the CSV records are ordered. But if it encounters CSV + where all records are on the same date, it assumes that the records are + oldest first. If in fact the CSV's records are normally newest first, + like: + + 2022-10-01, txn 3... + 2022-10-01, txn 2... + 2022-10-01, txn 1... + + you can add the newest-first rule to help hledger generate the transac- + tions in correct order. + + # same-day CSV records are newest first + newest-first + + intra-day-reversed + CSV records for each day are sometimes ordered in reverse compared to + the overall date order. Eg, here dates are newest first, but the + transactions on each date are oldest first: + + 2022-10-02, txn 3... + 2022-10-02, txn 4... + 2022-10-01, txn 1... + 2022-10-01, txn 2... + + In this situation, add the intra-day-reversed rule, and hledger will + compensate, improving the order of transactions. + + # transactions within each day are reversed with respect to the overall date order + intra-day-reversed + + decimal-mark + decimal-mark . + + or: + + decimal-mark , + + hledger automatically accepts either period or comma as a decimal mark + when parsing numbers (cf Amounts). However if any numbers in the CSV + contain digit group marks, such as thousand-separating commas, you + should declare the decimal mark explicitly with this rule, to avoid + misparsed numbers. + + fields list + fields FIELDNAME1, FIELDNAME2, ... + + A fields list (the word fields followed by comma-separated field names) + is optional, but convenient. It does two things: + + 1. It names the CSV field in each column. This can be convenient if + you are referencing them in other rules, so you can say %SomeField + instead of remembering %13. + + 2. Whenever you use one of the special hledger field names (described + below), it assigns the CSV value in this position to that hledger + field. This is the quickest way to populate hledger's fields and + build a transaction. + + Here's an example that says "use the 1st, 2nd and 4th fields as the + transaction's date, description and amount; name the last two fields + for later reference; and ignore the others": + + fields date, description, , amount, , , somefield, anotherfield + + In a fields list, the separator is always comma; it is unrelated to the + CSV file's separator. Also: + + o There must be least two items in the list (at least one comma). + + o Field names may not contain spaces. Spaces before/after field names + are optional. + + o Field names may contain _ (underscore) or - (hyphen). + + o Fields you don't care about can be given a dummy name or an empty + name. + + If the CSV contains column headings, it's convenient to use these for + your field names, suitably modified (eg lower-cased with spaces + replaced by underscores). + + Sometimes you may want to alter a CSV field name to avoid assigning to + a hledger field with the same name. Eg you could call the CSV's "bal- + ance" field balance_ to avoid directly setting hledger's balance field + (and generating a balance assertion). + + Field assignment + HLEDGERFIELD FIELDVALUE + + Field assignments are the more flexible way to assign CSV values to + hledger fields. They can be used instead of or in addition to a fields + list (see above). + + To assign a value to a hledger field, write the field name (any of the + standard hledger field/pseudo-field names, defined below), a space, + followed by a text value on the same line. This text value may inter- + polate CSV fields, referenced by their 1-based position in the CSV + record (%N), or by the name they were given in the fields list (%CSV- + FIELD). + + Some examples: + + # set the amount to the 4th CSV field, with " USD" appended + amount %4 USD + + # combine three fields to make a comment, containing note: and date: tags + comment note: %somefield - %anotherfield, date: %1 + + Tips: + + o Interpolation strips outer whitespace (so a CSV value like " 1 " + becomes 1 when interpolated) (#1051). + + o Interpolations always refer to a CSV field - you can't interpolate a + hledger field. (See Referencing other fields below). + + Field names + Note the two kinds of field names mentioned here, and used only in + hledger CSV rules files: + + 1. CSV field names (CSVFIELD in these docs): you can optionally name + the CSV columns for easy reference (since hledger doesn't yet auto- + matically recognise column headings in a CSV file), by writing arbi- + trary names in a fields list, eg: + + fields When, What, Some_Id, Net, Total, Foo, Bar + + 2. Special hledger field names (HLEDGERFIELD in these docs): you must + set at least some of these to generate the hledger transaction from + a CSV record, by writing them as the left hand side of a field + assignment, eg: + + date %When + code %Some_Id + description %What + comment %Foo %Bar + amount1 $ %Total + + or directly in a fields list: + + fields date, description, code, , amount1, Foo, Bar + currency $ + comment %Foo %Bar + + Here are all the special hledger field names available, and what hap- + pens when you assign values to them: + + date field + Assigning to date sets the transaction date. + + date2 field + date2 sets the transaction's secondary date, if any. + + status field + status sets the transaction's status, if any. + + code field + code sets the transaction's code, if any. + + description field + description sets the transaction's description, if any. + + comment field + comment sets the transaction's comment, if any. + + commentN, where N is a number, sets the Nth posting's comment. + + You can assign multi-line comments by writing literal \n in the code. + A comment starting with \n will begin on a new line. + + Comments can contain tags, as usual. + + account field + Assigning to accountN, where N is 1 to 99, sets the account name of the + Nth posting, and causes that posting to be generated. + + Most often there are two postings, so you'll want to set account1 and + account2. Typically account1 is associated with the CSV file, and is + set once with a top-level assignment, while account2 is set based on + each transaction's description, in conditional rules. + + If a posting's account name is left unset but its amount is set (see + below), a default account name will be chosen (like "expenses:unknown" + or "income:unknown"). + + amount field + There are several "amount" field name variants, useful for different + situations: + + o amountN sets the amount of the Nth posting, and causes that posting + to be generated. By assigning to amount1, amount2, ... etc. you + can generate up to 99 postings. Posting numbers don't have to be + consecutive; in certain situations using a high number might be help- + ful to influence the layout of postings. + + o amountN-in and amountN-out should be used instead, as a pair, when + and only when the amount must be obtained from two CSV fields. Eg + when the CSV has separate Debit and Credit fields instead of a single + Amount field. Note: + + o Don't think "-in is for the first posting and -out is for the sec- + ond posting" - that's not correct. Think: "amountN-in and amountN- + out together detect the amount for posting N, by inspecting two CSV + fields at once." + + o hledger assumes both CSV fields are unsigned, and will automati- + cally negate the -out value. + + o It also expects that at least one of the values is empty or zero, + so it knows which one to ignore. If that's not the case you'll + need an if rule (see Setting amounts below). + + o amount, with no posting number (and similarly, amount-in and amount- + out with no number) are an older syntax. We keep them for backwards + compatibility, and because they have special behaviour that is some- + times convenient: + + o They set the amount of posting 1 and (negated) the amount of post- + ing 2. + + o Posting 2's amount will be converted to cost if it has a cost + price. + + o Any of the newer rules for posting 1 or 2 (like amount1, or + amount2-in and amount2-out) will take precedence. This allows + incrementally migrating old rules files to the new syntax. + + There's more to say about amount-setting that doesn't fit here; please + see also "Setting amounts" below. + + currency field + currency sets a currency symbol, to be prepended to all postings' + amounts. You can use this if the CSV amounts do not have a currency + symbol, eg if it is in a separate column. + + currencyN prepends a currency symbol to just the Nth posting's amount. + + balance field + balanceN sets a balance assertion amount (or if the posting amount is + left empty, a balance assignment) on posting N. + + balance is a compatibility spelling for hledger <1.17; it is equivalent + to balance1. + + You can adjust the type of assertion/assignment with the balance-type + rule (see below). + + See Tips below for more about setting amounts and currency. + + if block + Rules can be applied conditionally, depending on patterns in the CSV + data. This allows flexibility; in particular, it is how you can cate- + gorise transactions, selecting an appropriate account name based on + their description (for example). There are two ways to write condi- + tional rules: "if blocks", described here, and "if tables", described + below. + + An if block is the word if and one or more "matcher" expressions (can + be a word or phrase), one per line, starting either on the same or next + line; followed by one or more indented rules. Eg, + + if MATCHER + RULE + + or + + if + MATCHER + MATCHER + MATCHER + RULE + RULE + + If any of the matchers succeeds, all of the indented rules will be + applied. They are usually field assignments, but the following special + rules may also be used within an if block: + + o skip - skips the matched CSV record (generating no transaction from + it) + + o end - skips the rest of the current CSV file. + + Some examples: + + # if the record contains "groceries", set account2 to "expenses:groceries" + if groceries + account2 expenses:groceries + + # if the record contains any of these phrases, set account2 and a transaction comment as shown + if + monthly service fee + atm transaction fee + banking thru software + account2 expenses:business:banking + comment XXX deductible ? check it + + # if an empty record is seen (assuming five fields), ignore the rest of the CSV file + if ,,,, + end + + Matchers + There are two kinds: + + 1. A record matcher is a word or single-line text fragment or regular + expression (REGEX), which hledger will try to match case-insensi- + tively anywhere within the CSV record. + Eg: whole foods + + 2. A field matcher is preceded with a percent sign and CSV field name + (%CSVFIELD REGEX). hledger will try to match these just within the + named CSV field. + Eg: %date 2023 + + The regular expression is (as usual in hledger) a POSIX extended regu- + lar expression, that also supports GNU word boundaries (\b, \B, \<, + \>), and nothing else. If you have trouble, see "Regular expressions" + in the hledger manual (https://hledger.org/hledger.html#regular-expres- + sions). + + With record matchers, it's important to know that the record matched is + not the original CSV record, but a modified one: separators will be + converted to commas, and enclosing double quotes (but not enclosing + whitespace) are removed. So for example, when reading an SSV file, if + the original record was: + + 2020-01-01; "Acme, Inc."; 1,000 + + the regex would see, and try to match, this modified record text: + + 2020-01-01,Acme, Inc., 1,000 + + When an if block has multiple matchers, they are combined as follows: + + o By default they are OR'd (any one of them can match) + + o When a matcher is preceded by ampersand (&) it will be AND'ed with + the previous matcher (both of them must match). + + There's not yet an easy syntax to negate a matcher. + + if table + "if tables" are an alternative to if blocks; they can express many + matchers and field assignments in a more compact tabular format, like + this: + + if,HLEDGERFIELD1,HLEDGERFIELD2,... + MATCHERA,VALUE1,VALUE2,... + MATCHERB,VALUE1,VALUE2,... + MATCHERC,VALUE1,VALUE2,... + + + The first character after if is taken to be the separator for the rest + of the table. It should be a non-alphanumeric character like , or | + that does not appear anywhere else in the table. (Note: it is unre- + lated to the CSV file's separator.) Whitespace can be used in the + matcher lines for readability, but not in the if line currently. The + table must be terminated by an empty line (or end of file). Each line + must contain the same number of separators; empty values are allowed. + + The above means: try all of the matchers; whenever a matcher succeeds, + assign all of the values on that line to the corresponding hledger + fields; later lines can overrider earlier ones. It is equivalent to + this sequence of if blocks: + + if MATCHERA + HLEDGERFIELD1 VALUE1 + HLEDGERFIELD2 VALUE2 + ... + + if MATCHERB + HLEDGERFIELD1 VALUE1 + HLEDGERFIELD2 VALUE2 + ... + + if MATCHERC + HLEDGERFIELD1 VALUE1 + HLEDGERFIELD2 VALUE2 + ... + + Example: + + if,account2,comment + atm transaction fee,expenses:business:banking,deductible? check it + %description groceries,expenses:groceries, + 2020/01/12.*Plumbing LLC,expenses:house:upkeep,emergency plumbing call-out + + balance-type + Balance assertions generated by assigning to balanceN are of the simple + = type by default, which is a single-commodity, subaccount-excluding + assertion. You may find the subaccount-including variants more useful, + eg if you have created some virtual subaccounts of checking to help + with budgeting. You can select a different type of assertion with the + balance-type rule: + + # balance assertions will consider all commodities and all subaccounts + balance-type ==* + + Here are the balance assertion types for quick reference: + + = single commodity, exclude subaccounts + =* single commodity, include subaccounts + == multi commodity, exclude subaccounts + ==* multi commodity, include subaccounts + + include + include RULESFILE + + This includes the contents of another CSV rules file at this point. + RULESFILE is an absolute file path or a path relative to the current + file's directory. This can be useful for sharing common rules between + several rules files, eg: + + # someaccount.csv.rules + + ## someaccount-specific rules + fields date,description,amount + account1 assets:someaccount + account2 expenses:misc + + ## common rules + include categorisation.rules + + Working with CSV + Some tips: + + Rapid feedback + It's a good idea to get rapid feedback while creating/troubleshooting + CSV rules. Here's a good way, using entr from eradman.com/entrproject: + + $ ls foo.csv* | entr bash -c 'echo ----; hledger -f foo.csv print desc:SOMEDESC' + + A desc: query (eg) is used to select just one, or a few, transactions + of interest. "bash -c" is used to run multiple commands, so we can + echo a separator each time the command re-runs, making it easier to + read the output. + + Valid CSV + Note that hledger will only accept valid CSV conforming to RFC 4180, + and equivalent SSV and TSV formats (like RFC 4180 but with semicolon or + tab as separators). This means, eg: + + o Values may be enclosed in double quotes, or not. Enclosing in single + quotes is not allowed. (Eg 'A','B' is rejected.) + + o When values are enclosed in double quotes, spaces outside the quotes + are not allowed. (Eg "A", "B" is rejected.) + + o When values are not enclosed in quotes, they may not contain double + quotes. (Eg A"A, B is rejected.) + + If your CSV/SSV/TSV is not valid in this sense, you'll need to trans- + form it before reading with hledger. Try using sed, or a more permis- + sive CSV parser like python's csv lib. + + File Extension + To help hledger choose the CSV file reader and show the right error + messages (and choose the right field separator character by default), + it's best if CSV/SSV/TSV files are named with a .csv, .ssv or .tsv + filename extension. (More about this at Data formats.) + + When reading files with the "wrong" extension, you can ensure the CSV + reader (and the default field separator) by prefixing the file path + with csv:, ssv: or tsv:: Eg: + + $ hledger -f ssv:foo.dat print + + You can also override the default field separator with a separator rule + if needed. + + Reading CSV from standard input + You'll need the file format prefix when reading CSV from stdin also, + since hledger assumes journal format by default. Eg: + + $ cat foo.dat | hledger -f ssv:- print + + Reading multiple CSV files + If you use multiple -f options to read multiple CSV files at once, + hledger will look for a correspondingly-named rules file for each CSV + file. But if you use the --rules-file option, that rules file will be + used for all the CSV files. + + Valid transactions + After reading a CSV file, hledger post-processes and validates the gen- + erated journal entries as it would for a journal file - balancing them, + applying balance assignments, and canonicalising amount styles. Any + errors at this stage will be reported in the usual way, displaying the + problem entry. + + There is one exception: balance assertions, if you have generated them, + will not be checked, since normally these will work only when the CSV + data is part of the main journal. If you do need to check balance + assertions generated from CSV right away, pipe into another hledger: + + $ hledger -f file.csv print | hledger -f- print + + Deduplicating, importing + When you download a CSV file periodically, eg to get your latest bank + transactions, the new file may overlap with the old one, containing + some of the same records. + + The import command will (a) detect the new transactions, and (b) append + just those transactions to your main journal. It is idempotent, so you + don't have to remember how many times you ran it or with which version + of the CSV. (It keeps state in a hidden .latest.FILE.csv file.) This + is the easiest way to import CSV data. Eg: + + # download the latest CSV files, then run this command. + # Note, no -f flags needed here. + $ hledger import *.csv [--dry] + + This method works for most CSV files. (Where records have a stable + chronological order, and new records appear only at the new end.) + + A number of other tools and workflows, hledger-specific and otherwise, + exist for converting, deduplicating, classifying and managing CSV data. + See: + + o https://hledger.org/cookbook.html#setups-and-workflows + + o https://plaintextaccounting.org -> data import/conversion + + Setting amounts + Continuing from amount field above, here are more tips on handling var- + ious amount-setting situations: + + 1. If the amount is in a single CSV field: + + a. If its sign indicates direction of flow: + Assign it to amountN, to set the Nth posting's amount. N is usu- + ally 1 or 2 but can go up to 99. + + b. If another field indicates direction of flow: + Use one or more conditional rules to set the appropriate amount + sign. Eg: + + # assume a withdrawal unless Type contains "deposit": + amount1 -%Amount + if %Type deposit + amount1 %Amount + + 2. If the amount is in one of two CSV fields (eg Debit and Credit): + + a. If both fields are unsigned: + Assign the fields to amountN-in and amountN-out. This sets posting + N's amount to whichever of these has a non-zero value. If it's the + -out value, the amount will be negated. + + b. If either field is signed: + Use a conditional rule to flip the sign when needed. Eg below, the + -out value already has a minus sign so we undo hledger's automatic + negating by negating once more (but only if the field is non-empty, + so that we don't leave a minus sign by itself): + + fields date, description, amount1-in, amount1-out + if %amount1-out [1-9] + amount1-out -%amount1-out + + c. If both fields can contain a non-zero value (or both can be + empty): + The -in/-out rules normally choose the value which is non-zero/non- + empty. Some value pairs can be ambiguous, such as 1 and none. For + such cases, use conditional rules to help select the amount. Eg, + to handle the above you could select the value containing non-zero + digits: + + fields date, description, in, out + if %in [1-9] + amount1 %in + if %out [1-9] + amount1 %out + + 3. If you want posting 2's amount converted to cost: + Use the unnumbered amount (or amount-in and amount-out) syntax. + + 4. If the CSV has only balance amounts, not transaction amounts: + Assign to balanceN, to set a balance assignment on the Nth posting, + causing the posting's amount to be calculated automatically. balance + with no number is equivalent to balance1. In this situation hledger is + more likely to guess the wrong default account name, so you may need to + set that explicitly. + + Amount signs + There is some special handling for amount signs, to simplify parsing + and sign-flipping: + + o If an amount value begins with a plus sign: + that will be removed: +AMT becomes AMT + + o If an amount value is parenthesised: + it will be de-parenthesised and sign-flipped: (AMT) becomes -AMT + + o If an amount value has two minus signs (or two sets of parentheses, + or a minus sign and parentheses): + they cancel out and will be removed: --AMT or -(AMT) becomes AMT + + o If an amount value contains just a sign (or just a set of parenthe- + ses): + that is removed, making it an empty value. "+" or "-" or "()" becomes + "". + + Setting currency/commodity + If the currency/commodity symbol is included in the CSV's amount + field(s): + + 2020-01-01,foo,$123.00 + + you don't have to do anything special for the commodity symbol, it will + be assigned as part of the amount. Eg: + + fields date,description,amount + + 2020-01-01 foo + expenses:unknown $123.00 + income:unknown $-123.00 + + If the currency is provided as a separate CSV field: + + 2020-01-01,foo,USD,123.00 + + You can assign that to the currency pseudo-field, which has the special + effect of prepending itself to every amount in the transaction (on the + left, with no separating space): + + fields date,description,currency,amount + + 2020-01-01 foo + expenses:unknown USD123.00 + income:unknown USD-123.00 + + Or, you can use a field assignment to construct the amount yourself, + with more control. Eg to put the symbol on the right, and separated by + a space: + + fields date,description,cur,amt + amount %amt %cur + + 2020-01-01 foo + expenses:unknown 123.00 USD + income:unknown -123.00 USD + + Note we used a temporary field name (cur) that is not currency - that + would trigger the prepending effect, which we don't want here. + + Amount decimal places + Like amounts in a journal file, the amounts generated by CSV rules like + amount1 influence commodity display styles, such as the number of deci- + mal places displayed in reports. + + The original amounts as written in the CSV file do not affect display + style (because we don't yet reliably know their commodity). + + Referencing other fields + In field assignments, you can interpolate only CSV fields, not hledger + fields. In the example below, there's both a CSV field and a hledger + field named amount1, but %amount1 always means the CSV field, not the + hledger field: + + # Name the third CSV field "amount1" + fields date,description,amount1 + + # Set hledger's amount1 to the CSV amount1 field followed by USD + amount1 %amount1 USD + + # Set comment to the CSV amount1 (not the amount1 assigned above) + comment %amount1 + + Here, since there's no CSV amount1 field, %amount1 will produce a lit- + eral "amount1": + + fields date,description,csvamount + amount1 %csvamount USD + # Can't interpolate amount1 here + comment %amount1 + + When there are multiple field assignments to the same hledger field, + only the last one takes effect. Here, comment's value will be be B, or + C if "something" is matched, but never A: + + comment A + comment B + if something + comment C + + How CSV rules are evaluated + Here's how to think of CSV rules being evaluated (if you really need + to). First, + + o include - all includes are inlined, from top to bottom, depth first. + (At each include point the file is inlined and scanned for further + includes, recursively, before proceeding.) + + Then "global" rules are evaluated, top to bottom. If a rule is + repeated, the last one wins: + + o skip (at top level) + + o date-format + + o newest-first + + o fields - names the CSV fields, optionally sets up initial assignments + to hledger fields + + Then for each CSV record in turn: + + o test all if blocks. If any of them contain a end rule, skip all + remaining CSV records. Otherwise if any of them contain a skip rule, + skip that many CSV records. If there are multiple matched skip + rules, the first one wins. + + o collect all field assignments at top level and in matched if blocks. + When there are multiple assignments for a field, keep only the last + one. + + o compute a value for each hledger field - either the one that was + assigned to it (and interpolate the %CSVFIELD references), or a + default + + o generate a hledger transaction (journal entry) from these values. + + This is all part of the CSV reader, one of several readers hledger can + use to parse input files. When all files have been read successfully, + the transactions are passed as input to whichever hledger command the + user specified. + + + Well factored rules + Some things than can help reduce duplication and complexity in rules + files: + + o Extracting common rules usable with multiple CSV files into a com- + mon.rules, and adding include common.rules to each CSV's rules file. + + o Splitting if blocks into smaller if blocks, extracting the frequently + used parts. + + CSV rules examples Bank of Ireland Here's a CSV with two amount fields (Debit and Credit), and a balance field, which we can use to add balance assertions, which is not neces- @@ -2875,6 +3766,28 @@ CSV ing directly from CSV, but they will be checked if these entries are imported into a journal file. + Coinbase + A simple example with some CSV from Coinbase. The spot price is + recorded using cost notation. The legacy amount field name conve- + niently sets amount 2 (posting 2's amount) to the total cost. + + # Timestamp,Transaction Type,Asset,Quantity Transacted,Spot Price Currency,Spot Price at Transaction,Subtotal,Total (inclusive of fees and/or spread),Fees and/or Spread,Notes + # 2021-12-30T06:57:59Z,Receive,USDC,100,GBP,0.740000,"","","","Received 100.00 USDC from an external account" + + # coinbase.csv.rules + skip 1 + fields Timestamp,Transaction_Type,Asset,Quantity_Transacted,Spot_Price_Currency,Spot_Price_at_Transaction,Subtotal,Total,Fees_Spread,Notes + date %Timestamp + date-format %Y-%m-%dT%T%Z + description %Notes + account1 assets:coinbase:cc + amount %Quantity_Transacted %Asset @ %Spot_Price_at_Transaction %Spot_Price_Currency + + $ hledger print -f coinbase.csv + 2021-12-30 Received 100.00 USDC from an external account + assets:coinbase:cc 100 USDC @ 0.740000 GBP + income:unknown -74.000000 GBP + Amazon Here we convert amazon.com order history, and use an if block to gener- ate a third posting if there's a fee. (In practice you'd probably get @@ -3077,820 +3990,15 @@ CSV revenues:foss donations:darcshub $-10.00 ; business: expenses:banking:paypal $0.59 ; business: - - CSV rules - The following kinds of rule can appear in the rules file, in any order. - Blank lines and lines beginning with # or ; or * are ignored. - - - separator a custom field separator - - skip skip one or more header lines or matched CSV - records - date-format how to parse dates in CSV records - timezone declare the time zone of ambiguous CSV date- - times - decimal-mark the decimal mark used in CSV amounts, if - ambiguous - newest-first improve txn order when there are multiple - records, newest first, all with the same date - intra-day-reversed improve txn order when each day's txns are - reverse of the overall date order - balance-type choose which type of balance assignments to - use - fields name CSV fields, assign them to hledger - fields - Field assignment assign a value to one hledger field, with - interpolation - Field names hledger field names, used in the fields list - and field assignments - if apply some rules to CSV records matched by - patterns - if table apply some rules to CSV records matched by - patterns, alternate syntax - end skip the remaining CSV records - include inline another CSV rules file - - separator - You can use the separator rule to read other kinds of character-sepa- - rated data. The argument is any single separator character, or the - words tab or space (case insensitive). Eg, for comma-separated values - (CSV): - - separator , - - or for semicolon-separated values (SSV): - - separator ; - - or for tab-separated values (TSV): - - separator TAB - - If the input file has a .csv, .ssv or .tsv file extension (or a csv:, - ssv:, tsv: prefix), the appropriate separator will be inferred automat- - ically, and you won't need this rule. - - skip - skip N - - The word skip followed by a number (or no number, meaning 1) tells - hledger to ignore this many non-empty lines at the start of the input - data. (Empty/blank lines are skipped automatically, so you don't need - to count those.) You'll need this whenever your CSV data contains - header lines. Header lines skipped in this way are ignored, and not - parsed as CSV. - - skip can also be used inside if blocks (described below), to skip indi- - vidual data records. Note records skipped in this way are still - required to be valid CSV, even though otherwise ignored. - - date-format - date-format DATEFMT - - This is a helper for the date (and date2) fields. If your CSV dates - are not formatted like YYYY-MM-DD, YYYY/MM/DD or YYYY.MM.DD, you'll - need to add a date-format rule describing them with a strptime-style - date parsing pattern - see https://hackage.haskell.org/pack- - age/time/docs/Data-Time-Format.html#v:formatTime. The pattern must - parse the CSV date value completely. Some examples: - - # MM/DD/YY - date-format %m/%d/%y - - # D/M/YYYY - # The - makes leading zeros optional. - date-format %-d/%-m/%Y - - # YYYY-Mmm-DD - date-format %Y-%h-%d - - # M/D/YYYY HH:MM AM some other junk - # Note the time and junk must be fully parsed, though only the date is used. - date-format %-m/%-d/%Y %l:%M %p some other junk - - timezone - timezone TIMEZONE - - When CSV contains date-times that are implicitly in some time zone - other than yours, but containing no explicit time zone information, you - can use this rule to declare the CSV's native time zone, which helps - prevent off-by-one dates. - - When the CSV date-times do contain time zone information, you don't - need this rule; instead, use %Z in date-format (or %z, %EZ, %Ez; see - the formatTime link above). - - In either of these cases, hledger will do a time-zone-aware conversion, - localising the CSV date-times to your current system time zone. If you - prefer to localise to some other time zone, eg for reproducibility, you - can (on unix at least) set the output timezone with the TZ environment - variable, eg: - - $ TZ=-1000 hledger print -f foo.csv # or TZ=-1000 hledger import foo.csv - - timezone currently does not understand timezone names, except "UTC", - "GMT", "EST", "EDT", "CST", "CDT", "MST", "MDT", "PST", or "PDT". For - others, use numeric format: +HHMM or -HHMM. - - decimal-mark - decimal-mark . - - or: - - decimal-mark , - - hledger automatically accepts either period or comma as a decimal mark - when parsing numbers (cf Amounts). However if any numbers in the CSV - contain digit group marks, such as thousand-separating commas, you - should declare the decimal mark explicitly with this rule, to avoid - misparsed numbers. - - newest-first - hledger tries to ensure that the generated transactions will be ordered - chronologically, including intra-day transactions. Usually it can - auto-detect how the CSV records are ordered. But if it encounters CSV - where all records are on the same date, it assumes that the records are - oldest first. If in fact the CSV's records are normally newest first, - like: - - 2022-10-01, txn 3... - 2022-10-01, txn 2... - 2022-10-01, txn 1... - - you can add the newest-first rule to help hledger generate the transac- - tions in correct order. - - # same-day CSV records are newest first - newest-first - - intra-day-reversed - CSV records for each day are sometimes ordered in reverse compared to - the overall date order. Eg, here dates are newest first, but the - transactions on each date are oldest first: - - 2022-10-02, txn 3... - 2022-10-02, txn 4... - 2022-10-01, txn 1... - 2022-10-01, txn 2... - - In this situation, add the intra-day-reversed rule, and hledger will - compensate, improving the order of transactions. - - # transactions within each day are reversed with respect to the overall date order - intra-day-reversed - - fields - fields FIELDNAME1, FIELDNAME2, ... - - A fields list (the word "fields" followed by comma-separated field - names) is the quick way to assign CSV field values to hledger fields. - (The other way is field assignments, see below.) A fields list does - does two things: - - 1. It names the CSV fields. This is optional, but can be convenient - later for interpolating them. - - 2. Whenever you use a standard hledger field name (defined below), the - CSV value is assigned to that part of the hledger transaction. - - Here's an example that says "use the 1st, 2nd and 4th fields as the - transaction's date, description and amount; name the last two fields - for later reference; and ignore the others": - - fields date, description, , amount, , , somefield, anotherfield - - Tips: - - o The fields list always use commas, even if your CSV data uses another - separator character. - - o Currently there must be least two items in the list (at least one - comma). - - o Field names may not contain spaces. Spaces before/after field names - are optional. - - o Field names may contain _ (underscore) or - (hyphen). - - o If the CSV contains column headings, it's a good idea to use these, - suitably modified, as the basis for your field names (eg lower-cased, - with underscores instead of spaces). - - o If some heading names match standard hledger fields, but you don't - want to set the hledger fields directly, alter those names, eg by - appending an underscore. - - o Fields you don't care about can be given a dummy name (eg: _ ), or no - name. - - Field assignment - HLEDGERFIELDNAME FIELDVALUE - - Field assignments are the more flexible way to assign CSV values to - hledger fields. They can be used instead of or in addition to a fields - list (see above). - - To assign a value to a hledger field, write the field name (any of the - standard hledger field/pseudo-field names, defined below), a space, - followed by a text value on the same line. This text value may inter- - polate CSV fields, referenced by their 1-based position in the CSV - record (%N), or by the name they were given in the fields list (%CSV- - FIELDNAME). - - Some examples: - - # set the amount to the 4th CSV field, with " USD" appended - amount %4 USD - - # combine three fields to make a comment, containing note: and date: tags - comment note: %somefield - %anotherfield, date: %1 - - Tips: - - o Interpolation strips outer whitespace (so a CSV value like " 1 " - becomes 1 when interpolated) (#1051). - - o Interpolations always refer to a CSV field - you can't interpolate a - hledger field. (See Referencing other fields below). - - Field names - Here are the standard hledger field (and pseudo-field) names, which you - can use in a fields list or in field assignments. For more about the - transaction parts they refer to, see Transactions. - - date field - Assigning to date sets the transaction date. - - date2 field - date2 sets the transaction's secondary date, if any. - - status field - status sets the transaction's status, if any. - - code field - code sets the transaction's code, if any. - - description field - description sets the transaction's description, if any. - - comment field - comment sets the transaction's comment, if any. - - commentN, where N is a number, sets the Nth posting's comment. - - You can assign multi-line comments by writing literal \n in the code. - A comment starting with \n will begin on a new line. - - Comments can contain tags, as usual. - - account field - Assigning to accountN, where N is 1 to 99, sets the account name of the - Nth posting, and causes that posting to be generated. - - Most often there are two postings, so you'll want to set account1 and - account2. Typically account1 is associated with the CSV file, and is - set once with a top-level assignment, while account2 is set based on - each transaction's description, and in conditional blocks. - - If a posting's account name is left unset but its amount is set (see - below), a default account name will be chosen (like "expenses:unknown" - or "income:unknown"). - - amount field - amountN sets the amount of the Nth posting, and causes that posting to - be generated. By assigning to amount1, amount2, ... etc. you can - generate up to 99 postings. - - amountN-in and amountN-out can be used instead, if the CSV uses sepa- - rate fields for debits and credits (inflows and outflows). hledger - assumes both of these CSV fields are unsigned, and will automatically - negate the "-out" value. It also requires that at least one of them is - either empty or zero. See "Setting amounts" below for more on this - topic. - - amount, or amount-in and amount-out are a legacy mode, to keep pre- - hledger-1.17 CSV rules files working (and for occasional convenience). - They are suitable only for two-posting transactions; they set both - posting 1's and posting 2's amount. Posting 2's amount will be - negated, and also converted to cost if there's a cost price. - - Note: it might sound as if amount-in is for one posting and amount-out - for the other posting, but no; use the -in and -out rules together for - the same posting, producing one amount from two CSV fields. - - If you have an existing rules file using the unnumbered form, you might - want to use the numbered form in certain conditional blocks, without - having to update and retest all the old rules. To facilitate this, - posting 1 ignores amount/amount-in/amount-out if any of - amount1/amount1-in/amount1-out are assigned, and posting 2 ignores them - if any of amount2/amount2-in/amount2-out are assigned, avoiding con- - flicts. - - currency field - currency sets a currency symbol, to be prepended to all postings' - amounts. You can use this if the CSV amounts do not have a currency - symbol, eg if it is in a separate column. - - currencyN prepends a currency symbol to just the Nth posting's amount. - - balance field - balanceN sets a balance assertion amount (or if the posting amount is - left empty, a balance assignment) on posting N. - - balance is a compatibility spelling for hledger <1.17; it is equivalent - to balance1. - - You can adjust the type of assertion/assignment with the balance-type - rule (see below). - - See Tips below for more about setting amounts and currency. - - if - if MATCHER - RULE - - if - MATCHER - MATCHER - MATCHER - RULE - RULE - - Conditional blocks ("if blocks") are a block of rules that are applied - only to CSV records which match certain patterns. They are often used - for customising account names based on transaction descriptions. - - Matching the whole record - Each MATCHER can be a record matcher, which looks like this: - - REGEX - - REGEX is a case-insensitive regular expression that tries to match any- - where within the CSV record. It is a POSIX ERE (extended regular - expression) that also supports GNU word boundaries (\b, \B, \<, \>), - and nothing else. If you have trouble, be sure to check our doc: - https://hledger.org/hledger.html#regular-expressions - - Important note: the record that is matched is not the original record, - but a synthetic one, with any enclosing double quotes (but not enclos- - ing whitespace) removed, and always comma-separated (which means that a - field containing a comma will appear like two fields). Eg, if the - original record is 2020-01-01; "Acme, Inc."; 1,000, the REGEX will - actually see 2020-01-01,Acme, Inc., 1,000). - - Matching individual fields - Or, MATCHER can be a field matcher, like this: - - %CSVFIELD REGEX - - which matches just the content of a particular CSV field. CSVFIELD is - a percent sign followed by the field's name or column number, like - %date or %1. - - Combining matchers - A single matcher can be written on the same line as the "if"; or multi- - ple matchers can be written on the following lines, non-indented. Mul- - tiple matchers are OR'd (any one of them can match), unless one begins - with an & symbol, in which case it is AND'ed with the previous matcher. - - if - MATCHER - & MATCHER - RULE - - Rules applied on successful match - After the patterns there should be one or more rules to apply, all - indented by at least one space. Three kinds of rule are allowed in - conditional blocks: - - o field assignments (to set a hledger field) - - o skip (to skip the matched CSV record) - - o end (to skip all remaining CSV records). - - Examples: - - # if the CSV record contains "groceries", set account2 to "expenses:groceries" - if groceries - account2 expenses:groceries - - # if the CSV record contains any of these patterns, set account2 and comment as shown - if - monthly service fee - atm transaction fee - banking thru software - account2 expenses:business:banking - comment XXX deductible ? check it - - if table - if,CSVFIELDNAME1,CSVFIELDNAME2,...,CSVFIELDNAMEn - MATCHER1,VALUE11,VALUE12,...,VALUE1n - MATCHER2,VALUE21,VALUE22,...,VALUE2n - MATCHER3,VALUE31,VALUE32,...,VALUE3n - - - Conditional tables ("if tables") are a different syntax to specify - field assignments that will be applied only to CSV records which match - certain patterns. - - MATCHER could be either field or record matcher, as described above. - When MATCHER matches, values from that row would be assigned to the CSV - fields named on the if line, in the same order. - - Therefore if table is exactly equivalent to a sequence of of if blocks: - - if MATCHER1 - CSVFIELDNAME1 VALUE11 - CSVFIELDNAME2 VALUE12 - ... - CSVFIELDNAMEn VALUE1n - - if MATCHER2 - CSVFIELDNAME1 VALUE21 - CSVFIELDNAME2 VALUE22 - ... - CSVFIELDNAMEn VALUE2n - - if MATCHER3 - CSVFIELDNAME1 VALUE31 - CSVFIELDNAME2 VALUE32 - ... - CSVFIELDNAMEn VALUE3n - - Each line starting with MATCHER should contain enough (possibly empty) - values for all the listed fields. - - Rules would be checked and applied in the order they are listed in the - table and, like with if blocks, later rules (in the same or another ta- - ble) or if blocks could override the effect of any rule. - - Instead of ',' you can use a variety of other non-alphanumeric charac- - ters as a separator. First character after if is taken to be the sepa- - rator for the rest of the table. It is the responsibility of the user - to ensure that separator does not occur inside MATCHERs and values - - there is no way to escape separator. - - Example: - - if,account2,comment - atm transaction fee,expenses:business:banking,deductible? check it - %description groceries,expenses:groceries, - 2020/01/12.*Plumbing LLC,expenses:house:upkeep,emergency plumbing call-out - - end - This rule can be used inside if blocks (only), to make hledger stop - reading this CSV file and move on to the next input file, or to command - execution. Eg: - - # ignore everything following the first empty record - if ,,,, - end - - include - include RULESFILE - - This includes the contents of another CSV rules file at this point. - RULESFILE is an absolute file path or a path relative to the current - file's directory. This can be useful for sharing common rules between - several rules files, eg: - - # someaccount.csv.rules - - ## someaccount-specific rules - fields date,description,amount - account1 assets:someaccount - account2 expenses:misc - - ## common rules - include categorisation.rules - - balance-type - Balance assertions generated by assigning to balanceN are of the simple - = type by default, which is a single-commodity, subaccount-excluding - assertion. You may find the subaccount-including variants more useful, - eg if you have created some virtual subaccounts of checking to help - with budgeting. You can select a different type of assertion with the - balance-type rule: - - # balance assertions will consider all commodities and all subaccounts - balance-type ==* - - Here are the balance assertion types for quick reference: - - = single commodity, exclude subaccounts - =* single commodity, include subaccounts - == multi commodity, exclude subaccounts - ==* multi commodity, include subaccounts - - Tips - Rapid feedback - It's a good idea to get rapid feedback while creating/troubleshooting - CSV rules. Here's a good way, using entr from eradman.com/entrproject: - - $ ls foo.csv* | entr bash -c 'echo ----; hledger -f foo.csv print desc:SOMEDESC' - - A desc: query (eg) is used to select just one, or a few, transactions - of interest. "bash -c" is used to run multiple commands, so we can - echo a separator each time the command re-runs, making it easier to - read the output. - - Valid CSV - Note that hledger will only accept valid CSV conforming to RFC 4180, - and equivalent SSV and TSV formats (like RFC 4180 but with semicolon or - tab as separators). This means, eg: - - o Values may be enclosed in double quotes, or not. Enclosing in single - quotes is not allowed. (Eg 'A','B' is rejected.) - - o When values are enclosed in double quotes, spaces outside the quotes - are not allowed. (Eg "A", "B" is rejected.) - - o When values are not enclosed in quotes, they may not contain double - quotes. (Eg A"A, B is rejected.) - - If your CSV/SSV/TSV is not valid in this sense, you'll need to trans- - form it before reading with hledger. Try using sed, or a more permis- - sive CSV parser like python's csv lib. - - File Extension - To help hledger identify the format and show the right error messages, - CSV/SSV/TSV files should normally be named with a .csv, .ssv or .tsv - filename extension. Or, the file path should be prefixed with csv:, - ssv: or tsv:. Eg: - - $ hledger -f foo.ssv print - - or: - - $ cat foo | hledger -f ssv:- foo - - You can override the file extension with a separator rule if needed. - See also: Input files in the hledger manual. - - Reading multiple CSV files - If you use multiple -f options to read multiple CSV files at once, - hledger will look for a correspondingly-named rules file for each CSV - file. But if you use the --rules-file option, that rules file will be - used for all the CSV files. - - Valid transactions - After reading a CSV file, hledger post-processes and validates the gen- - erated journal entries as it would for a journal file - balancing them, - applying balance assignments, and canonicalising amount styles. Any - errors at this stage will be reported in the usual way, displaying the - problem entry. - - There is one exception: balance assertions, if you have generated them, - will not be checked, since normally these will work only when the CSV - data is part of the main journal. If you do need to check balance - assertions generated from CSV right away, pipe into another hledger: - - $ hledger -f file.csv print | hledger -f- print - - Deduplicating, importing - When you download a CSV file periodically, eg to get your latest bank - transactions, the new file may overlap with the old one, containing - some of the same records. - - The import command will (a) detect the new transactions, and (b) append - just those transactions to your main journal. It is idempotent, so you - don't have to remember how many times you ran it or with which version - of the CSV. (It keeps state in a hidden .latest.FILE.csv file.) This - is the easiest way to import CSV data. Eg: - - # download the latest CSV files, then run this command. - # Note, no -f flags needed here. - $ hledger import *.csv [--dry] - - This method works for most CSV files. (Where records have a stable - chronological order, and new records appear only at the new end.) - - A number of other tools and workflows, hledger-specific and otherwise, - exist for converting, deduplicating, classifying and managing CSV data. - See: - - o https://hledger.org/cookbook.html#setups-and-workflows - - o https://plaintextaccounting.org -> data import/conversion - - Setting amounts - Some tips on using the amount-setting rules discussed above. - - Here are the ways to set a posting's amount: - - 1. If the CSV has a single amount field: - Assign (via a fields list or a field assignment) to amountN. This sets - the Nth posting's amount. N is usually 1 or 2 but can go up to 99. - - 2. If the CSV has separate amount fields for debit & credit (in & out): - - a. If both fields are unsigned: - Assign to amountN-in and amountN-out. This sets posting N's amount - to whichever of these has a non-zero value, and negates the "-out" - value. - - b. If either field is signed (can contain a minus sign): - Use a conditional rule to flip the sign (of non-empty values). - Since hledger always negates amountN-out, if it was already nega- - tive, we must undo that by negating once more (but only if the - field is non-empty): - - fields date, description, amount1-in, amount1-out - if %amount1-out [1-9] - amount1-out -%amount1-out - - c. If both fields, or neither field, can contain a non-zero value: - hledger normally expects exactly one of the fields to have a non- - zero value. Eg, the amountN-in/amountN-out rules would reject - value pairs like these: - - "", "" - "0", "0" - "1", "none" - - So, use smarter conditional rules to set the amount from the appro- - priate field. Eg, these rules would make it use only the value - containing non-zero digits, handling the above: - - fields date, description, in, out - if %in [1-9] - amount1 %in - if %out [1-9] - amount1 %out - - 3. If you want posting 2's amount converted to cost: - Assign to amount (or to amount-in and amount-out). (This is the legacy - numberless syntax, which sets amount1 and amount2 and converts amount2 - to cost.) - - 4. If the CSV has the balance instead of the transaction amount: - Assign to balanceN, which sets posting N's amount indirectly via a bal- - ance assignment. (Old syntax: balance, equivalent to balance1.) - - o If hledger guesses the wrong default account name: - When setting the amount via balance assertion, hledger may guess - the wrong default account name. So, set the account name explic- - itly, eg: - - fields date, description, balance1 - account1 assets:checking - - Amount signs - There is some special handling for amount signs, to simplify parsing - and sign-flipping: - - o If an amount value begins with a plus sign: - that will be removed: +AMT becomes AMT - - o If an amount value is parenthesised: - it will be de-parenthesised and sign-flipped: (AMT) becomes -AMT - - o If an amount value has two minus signs (or two sets of parentheses, - or a minus sign and parentheses): - they cancel out and will be removed: --AMT or -(AMT) becomes AMT - - o If an amount value contains just a sign (or just a set of parenthe- - ses): - that is removed, making it an empty value. "+" or "-" or "()" becomes - "". - - Setting currency/commodity - If the currency/commodity symbol is included in the CSV's amount - field(s): - - 2020-01-01,foo,$123.00 - - you don't have to do anything special for the commodity symbol, it will - be assigned as part of the amount. Eg: - - fields date,description,amount - - 2020-01-01 foo - expenses:unknown $123.00 - income:unknown $-123.00 - - If the currency is provided as a separate CSV field: - - 2020-01-01,foo,USD,123.00 - - You can assign that to the currency pseudo-field, which has the special - effect of prepending itself to every amount in the transaction (on the - left, with no separating space): - - fields date,description,currency,amount - - 2020-01-01 foo - expenses:unknown USD123.00 - income:unknown USD-123.00 - - Or, you can use a field assignment to construct the amount yourself, - with more control. Eg to put the symbol on the right, and separated by - a space: - - fields date,description,cur,amt - amount %amt %cur - - 2020-01-01 foo - expenses:unknown 123.00 USD - income:unknown -123.00 USD - - Note we used a temporary field name (cur) that is not currency - that - would trigger the prepending effect, which we don't want here. - - Amount decimal places - Like amounts in a journal file, the amounts generated by CSV rules like - amount1 influence commodity display styles, such as the number of deci- - mal places displayed in reports. - - The original amounts as written in the CSV file do not affect display - style (because we don't yet reliably know their commodity). - - Referencing other fields - In field assignments, you can interpolate only CSV fields, not hledger - fields. In the example below, there's both a CSV field and a hledger - field named amount1, but %amount1 always means the CSV field, not the - hledger field: - - # Name the third CSV field "amount1" - fields date,description,amount1 - - # Set hledger's amount1 to the CSV amount1 field followed by USD - amount1 %amount1 USD - - # Set comment to the CSV amount1 (not the amount1 assigned above) - comment %amount1 - - Here, since there's no CSV amount1 field, %amount1 will produce a lit- - eral "amount1": - - fields date,description,csvamount - amount1 %csvamount USD - # Can't interpolate amount1 here - comment %amount1 - - When there are multiple field assignments to the same hledger field, - only the last one takes effect. Here, comment's value will be be B, or - C if "something" is matched, but never A: - - comment A - comment B - if something - comment C - - How CSV rules are evaluated - Here's how to think of CSV rules being evaluated (if you really need - to). First, - - o include - all includes are inlined, from top to bottom, depth first. - (At each include point the file is inlined and scanned for further - includes, recursively, before proceeding.) - - Then "global" rules are evaluated, top to bottom. If a rule is - repeated, the last one wins: - - o skip (at top level) - - o date-format - - o newest-first - - o fields - names the CSV fields, optionally sets up initial assignments - to hledger fields - - Then for each CSV record in turn: - - o test all if blocks. If any of them contain a end rule, skip all - remaining CSV records. Otherwise if any of them contain a skip rule, - skip that many CSV records. If there are multiple matched skip - rules, the first one wins. - - o collect all field assignments at top level and in matched if blocks. - When there are multiple assignments for a field, keep only the last - one. - - o compute a value for each hledger field - either the one that was - assigned to it (and interpolate the %CSVFIELDNAME references), or a - default - - o generate a synthetic hledger transaction from these values. - - This is all part of the CSV reader, one of several readers hledger can - use to parse input files. When all files have been read successfully, - the transactions are passed as input to whichever hledger command the - user specified. - - Timeclock The time logging format of timeclock.el, as read by hledger. - hledger can read time logs in timeclock format. As with Ledger, these + hledger can read time logs in timeclock format. As with Ledger, these are (a subset of) timeclock.el's format, containing clock-in and clock- - out entries as in the example below. The date is a simple date. The - time format is HH:MM[:SS][+-ZZZZ]. Seconds and timezone are optional. + out entries as in the example below. The date is a simple date. The + time format is HH:MM[:SS][+-ZZZZ]. Seconds and timezone are optional. The timezone, if present, must be four digits and is ignored (currently - the time is always interpreted as a local time). Lines beginning with + the time is always interpreted as a local time). Lines beginning with # or ; or *, and blank lines, are ignored. i 2015/03/30 09:00:00 some:account name optional description after two spaces @@ -3898,9 +4006,9 @@ Timeclock i 2015/03/31 22:21:45 another account o 2015/04/01 02:00:34 - hledger treats each clock-in/clock-out pair as a transaction posting - some number of hours to an account. Or if the session spans more than - one day, it is split into several transactions, one for each day. For + hledger treats each clock-in/clock-out pair as a transaction posting + some number of hours to an account. Or if the session spans more than + one day, it is split into several transactions, one for each day. For the above time log, hledger print generates these journal entries: $ hledger -f t.timeclock print @@ -3921,27 +4029,27 @@ Timeclock To generate time logs, ie to clock in and clock out, you could: - o use emacs and the built-in timeclock.el, or the extended timeclock- + o use emacs and the built-in timeclock.el, or the extended timeclock- x.el and perhaps the extras in ledgerutils.el o at the command line, use these bash aliases: shell alias ti="echo - i `date '+%Y-%m-%d %H:%M:%S'` \$* >>$TIMELOG" alias to="echo o + i `date '+%Y-%m-%d %H:%M:%S'` \$* >>$TIMELOG" alias to="echo o `date '+%Y-%m-%d %H:%M:%S'` >>$TIMELOG" o or use the old ti and to scripts in the ledger 2.x repository. These - rely on a "timeclock" executable which I think is just the ledger 2 + rely on a "timeclock" executable which I think is just the ledger 2 executable renamed. Timedot - timedot format is hledger's human-friendly time logging format. Com- + timedot format is hledger's human-friendly time logging format. Com- pared to timeclock format, it is o convenient for quick, approximate, and retroactive time logging o readable: you can see at a glance where time was spent. - A timedot file contains a series of day entries, which might look like + A timedot file contains a series of day entries, which might look like this: 2021-08-04 @@ -3949,7 +4057,7 @@ Timedot fos:hledger:timedot .. ; docs per:admin:finance - hledger reads this as three time transactions on this day, with each + hledger reads this as three time transactions on this day, with each dot representing a quarter-hour spent: $ hledger -f a.timedot print # .timedot file extension activates the timedot reader @@ -3972,47 +4080,47 @@ Timedot o a common transaction comment for this day, after a semicolon (;). - After the date line are zero or more optionally-indented time transac- + After the date line are zero or more optionally-indented time transac- tion lines, consisting of: o an account name - any word or phrase, usually a hledger-style account name. - o two or more spaces - a field separator, required if there is an + o two or more spaces - a field separator, required if there is an amount (as in journal format). - o a timedot amount - dots representing quarter hours, or a number rep- + o a timedot amount - dots representing quarter hours, or a number rep- resenting hours. o an optional comment beginning with semicolon. This is ignored. In more detail, timedot amounts can be: - o dots: zero or more period characters, each representing one quarter- - hour. Spaces are ignored and can be used for grouping. Eg: .... .. + o dots: zero or more period characters, each representing one quarter- + hour. Spaces are ignored and can be used for grouping. Eg: .... .. o a number, representing hours. Eg: 1.5 - o a number immediately followed by a unit symbol s, m, h, d, w, mo, or + o a number immediately followed by a unit symbol s, m, h, d, w, mo, or y, representing seconds, minutes, hours, days weeks, months or years. Eg 1.5h or 90m. The following equivalencies are assumed: - 60s = 1m, 60m = 1h, 24h = 1d, 7d = 1w, 30d = 1mo, 365d = 1y. (This - unit will not be visible in the generated transaction amount, which is + 60s = 1m, 60m = 1h, 24h = 1d, 7d = 1w, 30d = 1mo, 365d = 1y. (This + unit will not be visible in the generated transaction amount, which is always in hours.) - There is some added flexibility to help with keeping time log data in + There is some added flexibility to help with keeping time log data in the same file as your notes, todo lists, etc.: o Blank lines and lines beginning with # or ; are ignored. o Before the first date line, lines beginning with * are ignored. From - the first date line onward, a sequence of *'s followed by a space at - beginning of lines (ie, the headline prefix used by Emacs Org mode) - is ignored. This means the time log can be kept under an Org head- - line, and date lines or time transaction lines can be Org headlines. + the first date line onward, a sequence of *'s followed by a space at + beginning of lines (ie, the headline prefix used by Emacs Org mode) + is ignored. This means the time log can be kept under an Org head- + line, and date lines or time transaction lines can be Org headlines. - o Lines not ending with a double-space and amount are parsed as trans- - actions with zero amount. (Most hledger reports hide these by + o Lines not ending with a double-space and amount are parsed as trans- + actions with zero amount. (Most hledger reports hide these by default; add -E to see them.) More examples: @@ -4100,36 +4208,36 @@ Time periods transaction or posting date, and the report end date will be the latest transaction, posting, or market price date. - Often you will want to see a shorter time span, such as the current - month. You can specify a start and/or end date using -b/--begin, + Often you will want to see a shorter time span, such as the current + month. You can specify a start and/or end date using -b/--begin, -e/--end, -p/--period or a date: query (described below). All of these accept the smart date syntax (below). Some notes: - o End dates are exclusive, as in Ledger, so you should write the date + o End dates are exclusive, as in Ledger, so you should write the date after the last day you want to see in the report. - o As noted in reporting options: among start/end dates specified with + o As noted in reporting options: among start/end dates specified with options, the last (i.e. right-most) option takes precedence. - o The effective report start and end dates are the intersection of the - start/end dates from options and that from date: queries. That is, - date:2019-01 date:2019 -p'2000 to 2030' yields January 2019, the + o The effective report start and end dates are the intersection of the + start/end dates from options and that from date: queries. That is, + date:2019-01 date:2019 -p'2000 to 2030' yields January 2019, the smallest common time span. - o A report interval (see below) will adjust start/end dates, when + o A report interval (see below) will adjust start/end dates, when needed, so that they fall on subperiod boundaries. Examples: -b 2016/3/17 begin on St. Patrick's day 2016 - -e 12/1 end at the start of december 1st of the current year + -e 12/1 end at the start of december 1st of the current year (11/30 will be the last date included) -b thismonth all transactions on or after the 1st of the current month -p thismonth all transactions in the current month - date:2016/3/17.. the above written as queries instead (.. can also be + date:2016/3/17.. the above written as queries instead (.. can also be replaced with -) date:..12/1 date:thismonth.. @@ -4137,13 +4245,13 @@ Time periods Smart dates hledger's user interfaces accept a flexible "smart date" syntax. Smart - dates allow some english words, can be relative to today's date, and + dates allow some english words, can be relative to today's date, and can have less-significant date parts omitted (defaulting to 1). Examples: - 2004/10/1, 2004-01-01, exact date, several separators allowed. Year + 2004/10/1, 2004-01-01, exact date, several separators allowed. Year 2004.9.1 is 4+ digits, month is 1-12, day is 1-31 2004 start of year 2004/10 start of month @@ -4164,22 +4272,23 @@ Time periods n -n periods from the current period days/weeks/months/quar- ters/years ago + 20181201 8 digit YYYYMMDD with valid year month and day 201812 6 digit YYYYMM with valid year and month - Counterexamples - malformed digit sequences might give surprising + Counterexamples - malformed digit sequences might give surprising results: - 201813 6 digits with an invalid month is parsed as start of + 201813 6 digits with an invalid month is parsed as start of 6-digit year - 20181301 8 digits with an invalid month is parsed as start of + 20181301 8 digits with an invalid month is parsed as start of 8-digit year 20181232 8 digits with an invalid day gives an error 201801012 9+ digits beginning with a valid YYYYMMDD gives an error - Note "today's date" can be overridden with the --today option, in case - it's needed for testing or for recreating old reports. (Except for + Note "today's date" can be overridden with the --today option, in case + it's needed for testing or for recreating old reports. (Except for periodic transaction rules; those are not affected by --today.) Report intervals @@ -4200,51 +4309,52 @@ Time periods o -Y/--yearly - These standard intervals always start on natural interval boundaries: - eg --weekly starts on mondays, --monthly starts on the first of the + These standard intervals always start on natural interval boundaries: + eg --weekly starts on mondays, --monthly starts on the first of the month, --yearly always starts on January 1st, etc. - Certain more complex intervals, and more flexible boundary dates, can - be specified by -p/--period. These are described in period expres- + Certain more complex intervals, and more flexible boundary dates, can + be specified by -p/--period. These are described in period expres- sions, below. - Report intervals can only be specified by the flags above, and not by + Report intervals can only be specified by the flags above, and not by query arguments, currently. - Report intervals have another effect: multi-period reports are always - expanded to fill a whole number of subperiods. So if you use a report - interval (other than --daily), and you have specified a start or end - date, you may notice those dates being overridden (ie, the report - starts earlier than your requested start date, or ends later than your + Report intervals have another effect: multi-period reports are always + expanded to fill a whole number of subperiods. So if you use a report + interval (other than --daily), and you have specified a start or end + date, you may notice those dates being overridden (ie, the report + starts earlier than your requested start date, or ends later than your requested end date). This is done to ensure "full" first and last sub- periods, so that all subperiods' numbers are comparable. To summarise: - o In multiperiod reports, all subperiods are forced to be the same + o In multiperiod reports, all subperiods are forced to be the same length, to simplify reporting. o Reports with the standard --weekly/--monthly/--quarterly/--yearly - intervals are required to start on the first day of a - week/month/quarter/year. We'd like more flexibility here but it + intervals are required to start on the first day of a + week/month/quarter/year. We'd like more flexibility here but it isn't supported yet. - o --period (below) can specify more complex intervals, starting on any + o --period (below) can specify more complex intervals, starting on any date. Period expressions - The -p/--period option accepts period expressions, a shorthand way of - expressing a start date, end date, and/or report interval all at once. + The -p/--period option accepts period expressions, a shorthand way of + expressing a start date, end date, and/or report interval all at once. - Here's a basic period expression specifying the first quarter of 2009. - Note, hledger always treats start dates as inclusive and end dates as + Here's a basic period expression specifying the first quarter of 2009. + Note, hledger always treats start dates as inclusive and end dates as exclusive: + -p "from 2009/1/1 to 2009/4/1" - Keywords like "from" and "to" are optional, and so are the spaces, as - long as you don't run two dates together. "to" can also be written as + Keywords like "from" and "to" are optional, and so are the spaces, as + long as you don't run two dates together. "to" can also be written as ".." or "-". These are equivalent to the above: @@ -4252,7 +4362,7 @@ Time periods -p2009/1/1to2009/4/1 -p2009/1/1..2009/4/1 - Dates are smart dates, so if the current year is 2009, the above can + Dates are smart dates, so if the current year is 2009, the above can also be written as: @@ -4266,13 +4376,13 @@ Time periods -p "from 2009/1/1" everything after january 1, 2009 - -p "since 2009/1" the same, since is a syn- + -p "since 2009/1" the same, since is a syn- onym -p "from 2009" the same - -p "to 2009" everything before january + -p "to 2009" everything before january 1, 2009 - A single date with no "from" or "to" defines both the start and end + A single date with no "from" or "to" defines both the start and end date like so: @@ -4283,15 +4393,15 @@ Time periods Or you can specify a single quarter like so: - -p "2009Q1" first quarter of 2009, equivalent to "2009/1/1 to + -p "2009Q1" first quarter of 2009, equivalent to "2009/1/1 to 2009/4/1" -p "q4" fourth quarter of the current year Period expressions with a report interval - -p/--period's argument can also begin with, or entirely consist of, a + -p/--period's argument can also begin with, or entirely consist of, a report interval. This should be separated from the start/end dates (if - any) by a space, or the word in. The basic intervals (which can also - be written as command line flags) are daily, weekly, monthly, quar- + any) by a space, or the word in. The basic intervals (which can also + be written as command line flags) are daily, weekly, monthly, quar- terly, and yearly. Some examples: @@ -4300,24 +4410,24 @@ Time periods -p "quarterly" As mentioned above, the weekly, monthly, quarterly and yearly intervals - require a report start date that is the first day of a week, month, - quarter or year. And, report start/end dates will be expanded if + require a report start date that is the first day of a week, month, + quarter or year. And, report start/end dates will be expanded if needed to span a whole number of intervals. For example: - -p "weekly from 2009/1/1 starts on 2008/12/29, closest preceding Mon- + -p "weekly from 2009/1/1 starts on 2008/12/29, closest preceding Mon- to 2009/4/1" day - -p "monthly in starts on 2018/11/01 + -p "monthly in starts on 2018/11/01 2008/11/25" - -p "quarterly from starts on 2009/04/01, ends on 2009/06/30, + -p "quarterly from starts on 2009/04/01, ends on 2009/06/30, 2009-05-05 to 2009-06-01" which are first and last days of Q2 2009 -p "yearly from starts on 2009/01/01, first day of 2009 2009-12-29" More complex report intervals - Some more complex kinds of interval are also supported in period + Some more complex kinds of interval are also supported in period expressions: o biweekly @@ -4330,26 +4440,26 @@ Time periods o every N days|weeks|months|quarters|years - These too will cause report start/end dates to be expanded, if needed, + These too will cause report start/end dates to be expanded, if needed, to span a whole number of intervals. Examples: - -p "bimonthly from 2008" periods will have boundaries on 2008/01/01, + -p "bimonthly from 2008" periods will have boundaries on 2008/01/01, 2008/03/01, ... -p "every 2 weeks" starts on closest preceding Monday - -p "every 5 months from periods will have boundaries on 2009/03/01, + -p "every 5 months from periods will have boundaries on 2009/03/01, 2009/03" 2009/08/01, ... Intervals with custom start date - All intervals mentioned above are required to start on their natural + All intervals mentioned above are required to start on their natural calendar boundaries, but the following intervals can start on any date: Weekly on custom day: - o every Nth day of week (th, nd, rd, or st are all accepted after the + o every Nth day of week (th, nd, rd, or st are all accepted after the number) - o every WEEKDAYNAME (full or three-letter english weekday name, case + o every WEEKDAYNAME (full or three-letter english weekday name, case insensitive) Monthly on custom day: @@ -4362,7 +4472,7 @@ Time periods o every MM/DD [of year] (month number and day of month number) - o every MONTHNAME DDth [of year] (full or three-letter english month + o every MONTHNAME DDth [of year] (full or three-letter english month name, case insensitive, and day of month number) o every DDth MONTHNAME [of year] (equivalent to the above) @@ -4373,62 +4483,62 @@ Time periods -p "every 2nd day of periods will go from Tue to Tue week" -p "every Tue" same - -p "every 15th day" period boundaries will be on 15th of each + -p "every 15th day" period boundaries will be on 15th of each month - -p "every 2nd Monday" period boundaries will be on second Monday of + -p "every 2nd Monday" period boundaries will be on second Monday of each month -p "every 11/05" yearly periods with boundaries on 5th of November -p "every 5th November" same -p "every Nov 5th" same - Show historical balances at end of the 15th day of each month (N is an + Show historical balances at end of the 15th day of each month (N is an end date, exclusive as always): $ hledger balance -H -p "every 16th day" - Group postings from the start of wednesday to end of the following + Group postings from the start of wednesday to end of the following tuesday (N is both (inclusive) start date and (exclusive) end date): $ hledger register checking -p "every 3rd day of week" Periods or dates ? - Report intervals like the above are most often used with -p|--period, - to divide reports into multiple subperiods - each generated date marks - a subperiod boundary. Here, the periods between the dates are what's + Report intervals like the above are most often used with -p|--period, + to divide reports into multiple subperiods - each generated date marks + a subperiod boundary. Here, the periods between the dates are what's important. - But report intervals can also be used with --forecast to generate - future transactions, or with balance --budget to generate budget goal- - setting transactions. For these, the dates themselves are what mat- + But report intervals can also be used with --forecast to generate + future transactions, or with balance --budget to generate budget goal- + setting transactions. For these, the dates themselves are what mat- ters. Events on multiple weekdays - The every WEEKDAYNAME form has a special variant with multiple day - names, comma-separated. Eg: every mon,thu,sat. Also, weekday and - weekendday are shorthand for mon,tue,wed,thu,fri and sat,sun respec- + The every WEEKDAYNAME form has a special variant with multiple day + names, comma-separated. Eg: every mon,thu,sat. Also, weekday and + weekendday are shorthand for mon,tue,wed,thu,fri and sat,sun respec- tively. This form is mainly intended for use with --forecast, to generate peri- odic transactions on arbitrary days of the week. It may be less useful - with -p, since it divides each week into subperiods of unequal length. - (Because gaps between periods are not allowed; if you'd like to change + with -p, since it divides each week into subperiods of unequal length. + (Because gaps between periods are not allowed; if you'd like to change this, see #1632.) Examples: - -p "every dates will be Mon, Wed, Fri; periods will be Mon- + -p "every dates will be Mon, Wed, Fri; periods will be Mon- mon,wed,fri" Tue, Wed-Thu, Fri-Sun - -p "every weekday" dates will be Mon, Tue, Wed, Thu, Fri; periods will + -p "every weekday" dates will be Mon, Tue, Wed, Thu, Fri; periods will be Mon, Tue, Wed, Thu, Fri-Sun -p "every weekend- dates will be Sat, Sun; periods will be Sat, Sun-Fri day" Depth - With the --depth NUM option (short form: -NUM), reports will show - accounts only to the specified depth, hiding deeper subaccounts. Use - this when you want a summary with less detail. This flag has the same + With the --depth NUM option (short form: -NUM), reports will show + accounts only to the specified depth, hiding deeper subaccounts. Use + this when you want a summary with less detail. This flag has the same effect as a depth: query argument: depth:2, --depth=2 or -2 are equiva- lent. @@ -4437,12 +4547,12 @@ Queries subset of your data. Most hledger commands accept optional query argu- ments to restrict their scope. The syntax is as follows: - o Zero or more space-separated query terms. These are most often + o Zero or more space-separated query terms. These are most often account name substrings: utilities food:groceries - o Terms with spaces or other special characters should be enclosed in + o Terms with spaces or other special characters should be enclosed in quotes: "personal care" @@ -4464,16 +4574,16 @@ Queries prefixed with not: to convert them into a negative match. acct:REGEX, REGEX - Match account names containing this (case insensitive) regular expres- + Match account names containing this (case insensitive) regular expres- sion. This is the default query type when there is no prefix, and reg- - ular expression syntax is typically not needed, so usually we just + ular expression syntax is typically not needed, so usually we just write an account name substring, like expenses or food. amt:N, amt:N, amt:>=N - Match postings with a single-commodity amount equal to, less than, or - greater than N. (Postings with multi-commodity amounts are not tested + Match postings with a single-commodity amount equal to, less than, or + greater than N. (Postings with multi-commodity amounts are not tested and will always match.) The comparison has two modes: if N is preceded - by a + or - sign (or is 0), the two signed numbers are compared. Oth- + by a + or - sign (or is 0), the two signed numbers are compared. Oth- erwise, the absolute magnitudes are compared, ignoring sign. code:REGEX @@ -4481,10 +4591,10 @@ Queries cur:REGEX Match postings or transactions including any amounts whose cur- - rency/commodity symbol is fully matched by REGEX. (For a partial - match, use .*REGEX.*). Note, to match special characters which are - regex-significant, you need to escape them with \. And for characters - which are significant to your shell you may need one more level of + rency/commodity symbol is fully matched by REGEX. (For a partial + match, use .*REGEX.*). Note, to match special characters which are + regex-significant, you need to escape them with \. And for characters + which are significant to your shell you may need one more level of escaping. So eg to match the dollar sign: hledger print cur:\\$. @@ -4492,17 +4602,17 @@ Queries Match transaction descriptions. date:PERIODEXPR - Match dates (or with the --date2 flag, secondary dates) within the - specified period. PERIODEXPR is a period expression with no report + Match dates (or with the --date2 flag, secondary dates) within the + specified period. PERIODEXPR is a period expression with no report interval. Examples: date:2016, date:thismonth, date:2/1-2/15, date:2021-07-27..nextquarter. date2:PERIODEXPR - Match secondary dates within the specified period (independent of the + Match secondary dates within the specified period (independent of the --date2 flag). depth:N - Match (or display, depending on command) accounts at or above this + Match (or display, depending on command) accounts at or above this depth. note:REGEX @@ -4510,7 +4620,7 @@ Queries whole description if there's no |). payee:REGEX - Match transaction payee/payer names (the part of the description left + Match transaction payee/payer names (the part of the description left of |, or the whole description if there's no |). real:, real:0 @@ -4520,11 +4630,11 @@ Queries Match unmarked, pending, or cleared transactions respectively. type:TYPECODES - Match by account type (see Declaring accounts > Account types). TYPE- - CODES is one or more of the single-letter account type codes ALERXCV, + Match by account type (see Declaring accounts > Account types). TYPE- + CODES is one or more of the single-letter account type codes ALERXCV, case insensitive. Note type:A and type:E will also match their respec- - tive subtypes C (Cash) and V (Conversion). Certain kinds of account - alias can disrupt account types, see Rewriting accounts > Aliases and + tive subtypes C (Cash) and V (Conversion). Certain kinds of account + alias can disrupt account types, see Rewriting accounts > Aliases and account types. tag:REGEX[=REGEX] @@ -4540,11 +4650,11 @@ Queries o Transactions also acquire the tags of their postings. (inacct:ACCTNAME - A special query term used automatically in hledger-web only: tells + A special query term used automatically in hledger-web only: tells hledger-web to show the transaction register for an account.) Combining query terms - When given multiple query terms, most commands select things which + When given multiple query terms, most commands select things which match: o any of the description terms AND @@ -4567,7 +4677,7 @@ Queries Although these fixed rules are enough for many needs, we do not support full boolean expressions (#203), (and you should not write AND or OR in - your queries). This makes certain queries hard to express, but here + your queries). This makes certain queries hard to express, but here are some tricks that can help: 1. Use a doubled not: prefix. Eg, to print only the food expenses paid @@ -4575,25 +4685,25 @@ Queries $ hledger print food not:not:cash - 2. Or pre-filter the transactions with print, piping the result into a + 2. Or pre-filter the transactions with print, piping the result into a second hledger command (with balance assertions disabled): $ hledger print cash | hledger -f- -I balance food Queries and command options - Some queries can also be expressed as command-line options: depth:2 is + Some queries can also be expressed as command-line options: depth:2 is equivalent to --depth 2, date:2020 is equivalent to -p 2020, etc. When - you mix command options and query arguments, generally the resulting + you mix command options and query arguments, generally the resulting query is their intersection. Queries and account aliases - When account names are rewritten with --alias or alias, acct: will + When account names are rewritten with --alias or alias, acct: will match either the old or the new account name. Queries and valuation - When amounts are converted to other commodities in cost or value - reports, cur: and amt: match the old commodity symbol and the old - amount quantity, not the new ones (except in hledger 1.22.0 where it's + When amounts are converted to other commodities in cost or value + reports, cur: and amt: match the old commodity symbol and the old + amount quantity, not the new ones (except in hledger 1.22.0 where it's reversed, see #1625). Querying with account aliases @@ -4601,20 +4711,20 @@ Queries will match either the old or the new account name. Querying with cost or value - When amounts are converted to other commodities in cost or value - reports, note that cur: matches the new commodity symbol, and not the + When amounts are converted to other commodities in cost or value + reports, note that cur: matches the new commodity symbol, and not the old one, and amt: matches the new quantity, and not the old one. Note: - this changed in hledger 1.22, previously it was the reverse, see the + this changed in hledger 1.22, previously it was the reverse, see the discussion at #1625. Pivoting - Normally, hledger groups and sums amounts within each account. The - --pivot FIELD option substitutes some other transaction field for + Normally, hledger groups and sums amounts within each account. The + --pivot FIELD option substitutes some other transaction field for account names, causing amounts to be grouped and summed by that field's - value instead. FIELD can be any of the transaction fields status, - code, description, payee, note, or a tag name. When pivoting on a tag - and a posting has multiple values of that tag, only the first value is - displayed. Values containing colon:separated:parts will be displayed + value instead. FIELD can be any of the transaction fields status, + code, description, payee, note, or a tag name. When pivoting on a tag + and a posting has multiple values of that tag, only the first value is + displayed. Values containing colon:separated:parts will be displayed hierarchically, like account names. Some examples: @@ -4646,7 +4756,7 @@ Pivoting -------------------- -2 EUR - Another way (the acct: query matches against the pivoted "account + Another way (the acct: query matches against the pivoted "account name"): $ hledger balance --pivot member acct:. @@ -4655,39 +4765,39 @@ Pivoting -2 EUR Generating data - Two features for generating transient data (visible only at report + Two features for generating transient data (visible only at report time) are built in to hledger's journal format: - o Auto posting rules can generate extra postings on certain transac- + o Auto posting rules can generate extra postings on certain transac- tions. They are activated by the --auto flag. - o Periodic transaction rules can generate repeating transactions, usu- - ally dated in the future, to help with forecasting or budgeting. - They are activated by the --forecast or balance --budget options, + o Periodic transaction rules can generate repeating transactions, usu- + ally dated in the future, to help with forecasting or budgeting. + They are activated by the --forecast or balance --budget options, described next. Forecasting - The --forecast flag activates any periodic transaction rules in the - journal. These will generate temporary additional transactions, usu- - ally recurring and in the future, which will appear in all reports. + The --forecast flag activates any periodic transaction rules in the + journal. These will generate temporary additional transactions, usu- + ally recurring and in the future, which will appear in all reports. hledger print --forecast is a good way to see them. - This can be useful for estimating balances into the future, perhaps + This can be useful for estimating balances into the future, perhaps experimenting with different scenarios. - It could also be useful for scripted data entry: you could describe - recurring transactions, and every so often copy the output of print + It could also be useful for scripted data entry: you could describe + recurring transactions, and every so often copy the output of print --forecast into the journal. - The generated transactions will have an extra tag, like generated- - transaction:~ PERIODICEXPR, indicating which periodic rule generated - them. There is also a similar, hidden tag, named _generated-transac- + The generated transactions will have an extra tag, like generated- + transaction:~ PERIODICEXPR, indicating which periodic rule generated + them. There is also a similar, hidden tag, named _generated-transac- tion:, which you can use to reliably match transactions generated "just now" (rather than printed in the past). The forecast transactions are generated within a forecast period, which - is independent of the report period. (Forecast period sets the bounds - for generated transactions, report period controls which transactions + is independent of the report period. (Forecast period sets the bounds + for generated transactions, report period controls which transactions are reported.) The forecast period begins on: o the start date provided within --forecast's argument, if any @@ -4696,7 +4806,7 @@ Forecasting o the report start date, if specified (with -b/-p/date:) - o the day after the latest ordinary transaction in the journal, if + o the day after the latest ordinary transaction in the journal, if any o otherwise today. @@ -4709,17 +4819,17 @@ Forecasting o otherwise 180 days (6 months) from today. - Note, this means that ordinary transactions will suppress periodic - transactions, by default; the periodic transactions will not start + Note, this means that ordinary transactions will suppress periodic + transactions, by default; the periodic transactions will not start until after the last ordinary transaction. This is usually convenient, but you can get around it in two ways: - o If you need to record some transactions in the future, make them - periodic transactions (with a single occurrence, eg: ~ YYYY-MM-DD) - rather than ordinary transactions. That way they won't suppress + o If you need to record some transactions in the future, make them + periodic transactions (with a single occurrence, eg: ~ YYYY-MM-DD) + rather than ordinary transactions. That way they won't suppress other periodic transactions. - o Or give --forecast a period expression argument. A forecast period + o Or give --forecast a period expression argument. A forecast period specified this way can overlap ordinary transactions, and need not be in the future. Some things to note: @@ -4728,44 +4838,44 @@ Forecasting o The period expression can specify the forecast period's start date, end date, or both. See also Report start & end date. - o The period expression should not specify a report interval. (Each + o The period expression should not specify a report interval. (Each periodic transaction rule specifies its own interval.) - Some examples: --forecast=202001-202004, --forecast=jan-, --fore- + Some examples: --forecast=202001-202004, --forecast=jan-, --fore- cast=2021. Budgeting - With the balance command's --budget report, each periodic transaction - rule generates recurring budget goals in specified accounts, and goals - and actual performance can be compared. See the balance command's doc + With the balance command's --budget report, each periodic transaction + rule generates recurring budget goals in specified accounts, and goals + and actual performance can be compared. See the balance command's doc below. See also: Budgeting and Forecasting. Cost reporting - This section is about recording the cost of things, in transactions - where one commodity is exchanged for another. Eg an exchange of cur- + This section is about recording the cost of things, in transactions + where one commodity is exchanged for another. Eg an exchange of cur- rency, or a stock purchase or sale. First, a quick glossary: - o Conversion - an exchange of one currency or commodity for another. - Eg a foreign currency exchange, or a purchase or sale of stock or + o Conversion - an exchange of one currency or commodity for another. + Eg a foreign currency exchange, or a purchase or sale of stock or cryptocurrency. - o Conversion transaction - a transaction involving one or more conver- + o Conversion transaction - a transaction involving one or more conver- sions. o Conversion rate - the cost per unit of one commodity in the other, ie the exchange rate. - o Cost - how much of one commodity was paid to acquire the other. And - more generally, in hledger docs: the amount exchanged in the "sec- + o Cost - how much of one commodity was paid to acquire the other. And + more generally, in hledger docs: the amount exchanged in the "sec- ondary" commodity (usually your base currency), whether in a purchase - or a sale, and whether expressed per unit or in total. Also, the + or a sale, and whether expressed per unit or in total. Also, the "@/@@ PRICE" notation used to represent this. -B: Convert to cost - As discussed in JOURNAL > Costs, when recording a transaction you can - also record the amount's cost in another commodity, by adding @ UNIT- + As discussed in JOURNAL > Costs, when recording a transaction you can + also record the amount's cost in another commodity, by adding @ UNIT- PRICE or @@ TOTALPRICE. Then you can see a report with amounts converted to cost, by adding the @@ -4784,8 +4894,8 @@ Cost reporting Notes: - -B is sensitive to the order of postings when a cost is inferred: the - inferred price will be in the commodity of the last amount. So if + -B is sensitive to the order of postings when a cost is inferred: the + inferred price will be in the commodity of the last amount. So if example 3's postings are reversed, while the transaction is equivalent, -B shows something different: @@ -4797,13 +4907,13 @@ Cost reporting EUR-100 assets:dollars # <- the dollars' selling price EUR100 assets:euros - The @/@@ cost notation is convenient, but has some drawbacks: it does - not truly balance the transaction, so it disrupts the accounting equa- + The @/@@ cost notation is convenient, but has some drawbacks: it does + not truly balance the transaction, so it disrupts the accounting equa- tion and tends to causes a non-zero total in balance reports. Equity conversion postings By contrast, conventional double entry bookkeeping (DEB) uses a differ- - ent notation: an extra pair of equity postings to balance conversion + ent notation: an extra pair of equity postings to balance conversion transactions. In this style, the above entry might be written: 2022-01-01 one hundred euros purchased at $1.35 each @@ -4812,16 +4922,16 @@ Cost reporting equity:conversion EUR-100 assets:euros EUR100 - This style is more correct, but it's also more verbose and makes cost + This style is more correct, but it's also more verbose and makes cost reporting more difficult for PTA tools. - Happily, current hledger can read either notation, or convert one to + Happily, current hledger can read either notation, or convert one to the other when needed, so you can use the one you prefer. Inferring equity postings from cost With --infer-equity, hledger detects transactions written with PTA cost - notation and adds equity conversion postings to them (and temporarily - permits the coexistence of equity conversion postings and cost nota- + notation and adds equity conversion postings to them (and temporarily + permits the coexistence of equity conversion postings and cost nota- tion, which normally would cause an unbalanced transaction error). Eg: 2022-01-01 @@ -4838,15 +4948,15 @@ Cost reporting The conversion account names can be changed with the conversion account type declaration. - --infer-equity is useful when when transactions have been recorded - using cost notation, to help preserve the accounting equation and bal- - ance reports' zero total, or to produce more conventional journal + --infer-equity is useful when when transactions have been recorded + using cost notation, to help preserve the accounting equation and bal- + ance reports' zero total, or to produce more conventional journal entries for sharing with non-PTA-users. Inferring cost from equity postings - The reverse operation is possible using --infer-costs, which detects - transactions written with equity conversion postings and adds PTA cost - notation to them (and temporarily permits the coexistence of equity + The reverse operation is possible using --infer-costs, which detects + transactions written with equity conversion postings and adds PTA cost + notation to them (and temporarily permits the coexistence of equity conversion postings and cost notation). Eg: 2022-01-01 @@ -4862,8 +4972,8 @@ Cost reporting equity:conversion EUR-100 assets:euros EUR100 - --infer-costs is useful when combined with -B/--cost, allowing cost - reporting even when transactions have been recorded using equity post- + --infer-costs is useful when combined with -B/--cost, allowing cost + reporting even when transactions have been recorded using equity post- ings: $ hledger print --infer-costs -B @@ -4880,52 +4990,54 @@ Cost reporting 2. two equity postings, next to one another 3. the equity accounts must be declared, with account type V/Conversion - (or if they are not declared, they must be named equity:conversion, + (or if they are not declared, they must be named equity:conversion, equity:trade, equity:trading or subaccounts of these) 4. the equity postings' amounts must exactly match the non-equity post- ings' amounts + 5. all of the amounts must be explicit, with none missing + Multiple such exchanges can coexist within a single transaction, should you need that. - When inferring cost, the order of postings matters: the cost is added - to the first of the non-equity postings involved in the exchange, in - the commodity of the last non-equity posting involved in the exchange. - If you don't want to write your postings in the required order, the - alternative is not to infer cost; instead, use explicit cost notation, - omitting the equity postings, inferring them later with --infer-equity + When inferring cost, the order of postings matters: the cost is added + to the first of the non-equity postings involved in the exchange, in + the commodity of the last non-equity posting involved in the exchange. + If you don't want to write your postings in the required order, the + alternative is not to infer cost; instead, use explicit cost notation, + omitting the equity postings, inferring them later with --infer-equity if needed. - --infer-equity and --infer-costs can be used together, if you have a + --infer-equity and --infer-costs can be used together, if you have a mixture of both notations in your journal. When to infer cost/equity - Inferring equity postings or costs is still fairly new, so not enabled - by default. We're not sure yet if that should change. Here are two + Inferring equity postings or costs is still fairly new, so not enabled + by default. We're not sure yet if that should change. Here are two suggestions to try, experience reports welcome: - 1. When you use -B, always use --infer-costs as well. Eg: hledger bal + 1. When you use -B, always use --infer-costs as well. Eg: hledger bal -B --infer-costs - 2. Always run hledger with both flags enabled. Eg: alias hl="hledger + 2. Always run hledger with both flags enabled. Eg: alias hl="hledger --infer-equity --infer-costs" How to record conversions - Essentially there are four ways to record a conversion transaction in + Essentially there are four ways to record a conversion transaction in hledger. Here are all of them, with pros and cons. Conversion with implicit cost - Let's assume 100 EUR is converted to 120 USD. You can just record the - outflow (100 EUR) and inflow (120 USD) in the appropriate asset + Let's assume 100 EUR is converted to 120 USD. You can just record the + outflow (100 EUR) and inflow (120 USD) in the appropriate asset account: 2021-01-01 assets:cash -100 EUR assets:cash 120 USD - hledger will assume this transaction is balanced, inferring that the - conversion rate must be 1 EUR = 1.20 USD. You can see the inferred + hledger will assume this transaction is balanced, inferring that the + conversion rate must be 1 EUR = 1.20 USD. You can see the inferred rate by using hledger print -x. Pro: @@ -4934,18 +5046,18 @@ Cost reporting Con: - o Less error checking - typos in amounts or commodity symbols may not + o Less error checking - typos in amounts or commodity symbols may not be detected o Conversion rate is not clear - o Disturbs the accounting equation, unless you add the --infer-equity + o Disturbs the accounting equation, unless you add the --infer-equity flag - You can prevent accidental implicit conversions due to a mistyped com- + You can prevent accidental implicit conversions due to a mistyped com- modity symbol, by using hledger check commodities. - You can prevent implicit conversions entirely, by using hledger check + You can prevent implicit conversions entirely, by using hledger check balancednoautoconversion, or -s/--strict. Conversion with explicit cost @@ -4968,16 +5080,16 @@ Cost reporting Con: - o Disturbs the accounting equation, unless you add the --infer-equity + o Disturbs the accounting equation, unless you add the --infer-equity flag Conversion with equity postings - In strict double entry bookkeeping, the above transaction is not bal- - anced in EUR or in USD, since some EUR disappears, and some USD + In strict double entry bookkeeping, the above transaction is not bal- + anced in EUR or in USD, since some EUR disappears, and some USD appears. This violates the accounting equation (A+L+E=0), and prevents reports like balancesheetequity from showing a zero total. - The proper way to make it balance is to add a balancing posting for + The proper way to make it balance is to add a balancing posting for each commodity, using an equity account: 2021-01-01 @@ -5003,8 +5115,8 @@ Cost reporting o Cost reporting requires adding the --infer-costs flag Conversion with equity postings and explicit cost - Here both equity postings and @ notation are used together. hledger - will usually complain about this redundancy, but when using the + Here both equity postings and @ notation are used together. hledger + will usually complain about this redundancy, but when using the --infer-costs flag it is accepted. 2021-01-01 @@ -5032,96 +5144,96 @@ Cost reporting o Not compatible with ledger Cost tips - o Recording the conversion rate explicitly is good because it makes + o Recording the conversion rate explicitly is good because it makes that clear and helps detect errors. - o Recording equity postings is good because it is correct bookkeeping + o Recording equity postings is good because it is correct bookkeeping and preserves the accounting equation. - o Combining these is possible by using the --infer-costs flag (which + o Combining these is possible by using the --infer-costs flag (which requires well-ordered postings). - o When you want to see the cost (or sale proceeds) of things, use -B - (or --cost). If you use equity conversion postings notation, use -B + o When you want to see the cost (or sale proceeds) of things, use -B + (or --cost). If you use equity conversion postings notation, use -B --infer-costs. - o If you use PTA cost notation, and you want to see a balanced balance + o If you use PTA cost notation, and you want to see a balanced balance sheet or print correct journal entries, use --infer-equity. o Conversion to cost is performed before valuation (described next). Valuation - Instead of reporting amounts in their original commodity, hledger can + Instead of reporting amounts in their original commodity, hledger can convert them to cost/sale amount (using the conversion rate recorded in - the transaction), and/or to market value (using some market price on a - certain date). This is controlled by the --value=TYPE[,COMMODITY] - option, which will be described below. We also provide the simpler -V + the transaction), and/or to market value (using some market price on a + certain date). This is controlled by the --value=TYPE[,COMMODITY] + option, which will be described below. We also provide the simpler -V and -X COMMODITY options, and often one of these is all you need: -V: Value - The -V/--market flag converts amounts to market value in their default + The -V/--market flag converts amounts to market value in their default valuation commodity, using the market prices in effect on the valuation date(s), if any. More on these in a minute. -X: Value in specified commodity The -X/--exchange=COMM option is like -V, except you tell it which cur- - rency you want to convert to, and it tries to convert everything to + rency you want to convert to, and it tries to convert everything to that. Valuation date - Since market prices can change from day to day, market value reports + Since market prices can change from day to day, market value reports have a valuation date (or more than one), which determines which market prices will be used. For single period reports, if an explicit report end date is specified, - that will be used as the valuation date; otherwise the valuation date + that will be used as the valuation date; otherwise the valuation date is the journal's end date. - For multiperiod reports, each column/period is valued on the last day + For multiperiod reports, each column/period is valued on the last day of the period, by default. Finding market price - To convert a commodity A to its market value in another commodity B, - hledger looks for a suitable market price (exchange rate) as follows, + To convert a commodity A to its market value in another commodity B, + hledger looks for a suitable market price (exchange rate) as follows, in this order of preference : - 1. A declared market price or inferred market price: A's latest market + 1. A declared market price or inferred market price: A's latest market price in B on or before the valuation date as declared by a P direc- - tive, or (with the --infer-market-prices flag) inferred from costs. + tive, or (with the --infer-market-prices flag) inferred from costs. 2. A reverse market price: the inverse of a declared or inferred market price from B to A. - 3. A forward chain of market prices: a synthetic price formed by com- + 3. A forward chain of market prices: a synthetic price formed by com- bining the shortest chain of "forward" (only 1 above) market prices, leading from A to B. - 4. Any chain of market prices: a chain of any market prices, including - both forward and reverse prices (1 and 2 above), leading from A to + 4. Any chain of market prices: a chain of any market prices, including + both forward and reverse prices (1 and 2 above), leading from A to B. - There is a limit to the length of these price chains; if hledger - reaches that length without finding a complete chain or exhausting all - possibilities, it will give up (with a "gave up" message visible in + There is a limit to the length of these price chains; if hledger + reaches that length without finding a complete chain or exhausting all + possibilities, it will give up (with a "gave up" message visible in --debug=2 output). That limit is currently 1000. - Amounts for which no suitable market price can be found, are not con- + Amounts for which no suitable market price can be found, are not con- verted. --infer-market-prices: market prices from transactions Normally, market value in hledger is fully controlled by, and requires, P directives in your journal. Since adding and updating those can be a - chore, and since transactions usually take place at close to market - value, why not use the recorded costs as additional market prices (as - Ledger does) ? Adding the --infer-market-prices flag to -V, -X or + chore, and since transactions usually take place at close to market + value, why not use the recorded costs as additional market prices (as + Ledger does) ? Adding the --infer-market-prices flag to -V, -X or --value enables this. - So for example, hledger bs -V --infer-market-prices will get market - prices both from P directives and from transactions. If both occur on + So for example, hledger bs -V --infer-market-prices will get market + prices both from P directives and from transactions. If both occur on the same day, the P directive takes precedence. There is a downside: value reports can sometimes be affected in confus- - ing/undesired ways by your journal entries. If this happens to you, + ing/undesired ways by your journal entries. If this happens to you, read all of this Valuation section carefully, and try adding --debug or --debug=2 to troubleshoot. @@ -5129,15 +5241,15 @@ Valuation o multicommodity transactions with explicit prices (@/@@) - o multicommodity transactions with implicit prices (no @, two commodi- - ties, unbalanced). (With these, the order of postings matters. + o multicommodity transactions with implicit prices (no @, two commodi- + ties, unbalanced). (With these, the order of postings matters. hledger print -x can be useful for troubleshooting.) o multicommodity transactions with equity postings, if cost is inferred with --infer-costs. - There is a limitation (bug) currently: when a valuation commodity is - not specified, prices inferred with --infer-market-prices do not help + There is a limitation (bug) currently: when a valuation commodity is + not specified, prices inferred with --infer-market-prices do not help select a default valuation commodity, as P prices would. So conversion might not happen because no valuation commodity was detected (--debug=2 will show this). To be safe, specify the valuation commmodity, eg: @@ -5147,8 +5259,8 @@ Valuation o --value=then,EUR --infer-market-prices, not --value=then --infer-mar- ket-prices - Signed costs and market prices can be confusing. For reference, here - is the current behaviour, since hledger 1.25. (If you think it should + Signed costs and market prices can be confusing. For reference, here + is the current behaviour, since hledger 1.25. (If you think it should work differently, see #1870.) 2022-01-01 Positive Unit prices @@ -5178,7 +5290,7 @@ Valuation b B -1 @@ A -1 All of the transactions above are considered balanced (and on each day, - the two transactions are considered equivalent). Here are the market + the two transactions are considered equivalent). Here are the market prices inferred for B: $ hledger -f- --infer-market-prices prices @@ -5191,34 +5303,34 @@ Valuation Valuation commodity When you specify a valuation commodity (-X COMM or --value TYPE,COMM): - hledger will convert all amounts to COMM, wherever it can find a suit- + hledger will convert all amounts to COMM, wherever it can find a suit- able market price (including by reversing or chaining prices). - When you leave the valuation commodity unspecified (-V or --value + When you leave the valuation commodity unspecified (-V or --value TYPE): - For each commodity A, hledger picks a default valuation commodity as + For each commodity A, hledger picks a default valuation commodity as follows, in this order of preference: 1. The price commodity from the latest P-declared market price for A on or before valuation date. 2. The price commodity from the latest P-declared market price for A on - any date. (Allows conversion to proceed when there are inferred + any date. (Allows conversion to proceed when there are inferred prices before the valuation date.) - 3. If there are no P directives at all (any commodity or date) and the - --infer-market-prices flag is used: the price commodity from the + 3. If there are no P directives at all (any commodity or date) and the + --infer-market-prices flag is used: the price commodity from the latest transaction-inferred price for A on or before valuation date. This means: - o If you have P directives, they determine which commodities -V will + o If you have P directives, they determine which commodities -V will convert, and to what. - o If you have no P directives, and use the --infer-market-prices flag, + o If you have no P directives, and use the --infer-market-prices flag, costs determine it. - Amounts for which no valuation commodity can be found are not con- + Amounts for which no valuation commodity can be found are not con- verted. Simple valuation examples @@ -5245,7 +5357,7 @@ Valuation $ hledger -f t.j bal -N euros -V -e 2016/11/4 $110.00 assets:euros - What are they worth after 2016/12/21 ? (no report end date specified, + What are they worth after 2016/12/21 ? (no report end date specified, defaults to today) $ hledger -f t.j bal -N euros -V @@ -5265,31 +5377,31 @@ Valuation The TYPE part selects cost or value and valuation date: --value=then - Convert amounts to their value in the default valuation commod- + Convert amounts to their value in the default valuation commod- ity, using market prices on each posting's date. --value=end - Convert amounts to their value in the default valuation commod- - ity, using market prices on the last day of the report period - (or if unspecified, the journal's end date); or in multiperiod + Convert amounts to their value in the default valuation commod- + ity, using market prices on the last day of the report period + (or if unspecified, the journal's end date); or in multiperiod reports, market prices on the last day of each subperiod. --value=now - Convert amounts to their value in the default valuation commod- - ity using current market prices (as of when report is gener- + Convert amounts to their value in the default valuation commod- + ity using current market prices (as of when report is gener- ated). --value=YYYY-MM-DD - Convert amounts to their value in the default valuation commod- + Convert amounts to their value in the default valuation commod- ity using market prices on this date. To select a different valuation commodity, add the optional ,COMM part: - a comma, then the target commodity's symbol. Eg: --value=now,EUR. + a comma, then the target commodity's symbol. Eg: --value=now,EUR. hledger will do its best to convert amounts to this commodity, deducing market prices as described above. More valuation examples - Here are some examples showing the effect of --value, as seen with + Here are some examples showing the effect of --value, as seen with print: P 2000-01-01 A 1 B @@ -5327,7 +5439,7 @@ Valuation 2000-02-01 (a) 2 B - With no report period specified, that shows the value as of the last + With no report period specified, that shows the value as of the last day of the journal (2000-03-01): $ hledger -f- print --value=end @@ -5364,7 +5476,7 @@ Valuation 2000-03-01 (a) 1 B - You may need to explicitly set a commodity's display style, when + You may need to explicitly set a commodity's display style, when reverse prices are used. Eg this output might be surprising: P 2000-01-01 A 2B @@ -5378,10 +5490,10 @@ Valuation a 0 b 0 - Explanation: because there's no amount or commodity directive specify- - ing a display style for A, 0.5A gets the default style, which shows no + Explanation: because there's no amount or commodity directive specify- + ing a display style for A, 0.5A gets the default style, which shows no decimal digits. Because the displayed amount looks like zero, the com- - modity symbol and minus sign are not displayed either. Adding a com- + modity symbol and minus sign are not displayed either. Adding a com- modity directive sets a more useful display style for A: P 2000-01-01 A 2B @@ -5397,7 +5509,7 @@ Valuation b -0.50A Interaction of valuation and queries - When matching postings based on queries in the presence of valuation, + When matching postings based on queries in the presence of valuation, the following happens. 1. The query is separated into two parts: @@ -5411,16 +5523,16 @@ Valuation 3. Valuation is applied to the postings. - 4. The postings are matched to the other parts of the query based on + 4. The postings are matched to the other parts of the query based on post-valued amounts. See: 1625 Effect of valuation on reports - Here is a reference for how valuation is supposed to affect each part - of hledger's reports (and a glossary). (It's wide, you'll have to - scroll sideways.) It may be useful when troubleshooting. If you find - problems, please report them, ideally with a reproducible example. + Here is a reference for how valuation is supposed to affect each part + of hledger's reports (and a glossary). (It's wide, you'll have to + scroll sideways.) It may be useful when troubleshooting. If you find + problems, please report them, ideally with a reproducible example. Related: #329, #1083. @@ -5428,7 +5540,7 @@ Valuation type --value=now ----------------------------------------------------------------------------------------------- print - posting cost value at value at posting value at value at + posting cost value at value at posting value at value at amounts report end date report or DATE/today or today journal end balance unchanged unchanged unchanged unchanged unchanged @@ -5445,7 +5557,7 @@ Valuation with report report or posting was made report or interval journal journal start start - posting cost value at value at posting value at value at + posting cost value at value at posting value at value at amounts report or date report or DATE/today journal end journal end summary post- summarised value at sum of postings value at value at @@ -5458,21 +5570,20 @@ Valuation balance (bs, bse, cf, is) - balance sums of value at value at posting value at value at + balance sums of value at value at posting value at value at changes costs report end date report or DATE/today of - or today of journal end sums of post- + or today of journal end sums of post- sums of of sums of ings postings postings budget like balance like balance like balance like bal- like balance amounts changes changes changes ances changes (--budget) - grand total sum of dis- sum of dis- sum of displayed sum of dis- sum of dis- + grand total sum of dis- sum of dis- sum of displayed sum of dis- sum of dis- played val- played val- valued played val- played values ues ues ues - balance (bs, - bse, cf, is) + bse, cf, is) with report interval starting bal- sums of value at sums of values of value at sums of post- @@ -5493,14 +5604,15 @@ Valuation report start end at respective to period posting dates end + budget like balance like balance like balance like bal- like balance amounts changes/end changes/end changes/end bal- ances changes/end (--budget) balances balances ances balances row totals, sums, aver- sums, aver- sums, averages of sums, aver- sums, aver- - row averages ages of dis- ages of dis- displayed values ages of dis- ages of dis- + row averages ages of dis- ages of dis- displayed values ages of dis- ages of dis- (-T, -A) played val- played val- played val- played values ues ues ues - column totals sums of dis- sums of dis- sums of displayed sums of dis- sums of dis- + column totals sums of dis- sums of dis- sums of displayed sums of dis- sums of dis- played val- played val- values played val- played values ues ues ues grand total, sum, average sum, average sum, average of sum, average sum, average @@ -5515,29 +5627,29 @@ Valuation cost calculated using price(s) recorded in the transaction(s). - value market value using available market price declarations, or the + value market value using available market price declarations, or the unchanged amount if no conversion rate can be found. report start - the first day of the report period specified with -b or -p or + the first day of the report period specified with -b or -p or date:, otherwise today. report or journal start - the first day of the report period specified with -b or -p or - date:, otherwise the earliest transaction date in the journal, + the first day of the report period specified with -b or -p or + date:, otherwise the earliest transaction date in the journal, otherwise today. report end - the last day of the report period specified with -e or -p or + the last day of the report period specified with -e or -p or date:, otherwise today. report or journal end - the last day of the report period specified with -e or -p or - date:, otherwise the latest transaction date in the journal, + the last day of the report period specified with -e or -p or + date:, otherwise the latest transaction date in the journal, otherwise today. report interval - a flag (-D/-W/-M/-Q/-Y) or period expression that activates the + a flag (-D/-W/-M/-Q/-Y) or period expression that activates the report's multi-period mode (whether showing one or many subperi- ods). @@ -5583,7 +5695,7 @@ PART 4: COMMANDS o activity - show postings-per-interval bar charts - o balance (bal) - show balance changes/end balances/budgets in any + o balance (bal) - show balance changes/end balances/budgets in any accounts o codes - show transaction codes @@ -5604,14 +5716,9 @@ PART 4: COMMANDS o print - show transactions (journal entries) - o print-unique - show only transactions with unique descriptions - - o register (reg) - show postings in one or more accounts & running + o register (reg) - show postings in one or more accounts & running total - o register-match - show a recent posting that best matches a descrip- - tion - o stats - show journal statistics o tags - show tag names @@ -5798,74 +5905,78 @@ PART 4: COMMANDS aregister requires one argument: the account to report on. You can write either the full account name, or a case-insensitive regular expression which will select the alphabetically first matched account. - (Eg if you have assets:aaa:checking and assets:bbb:checking accounts, - hledger areg checking would select assets:aaa:checking.) - Transactions involving subaccounts of this account will also be shown. - aregister ignores depth limits, so its final total will always match a + When there are multiple matches, the alphabetically-first choice can be + surprising; eg if you have assets:per:checking 1 and assets:biz:check- + ing 2 accounts, hledger areg checking would select assets:biz:checking + 2. It's just a convenience to save typing, so if in doubt, write the + full account name, or a distinctive substring that matches uniquely. + + Transactions involving subaccounts of this account will also be shown. + aregister ignores depth limits, so its final total will always match a balance report with similar arguments. - Any additional arguments form a query which will filter the transac- + Any additional arguments form a query which will filter the transac- tions shown. Note some queries will disturb the running balance, caus- ing it to be different from the account's real-world running balance. - An example: this shows the transactions and historical running balance + An example: this shows the transactions and historical running balance during july, in the first account whose name contains "checking": $ hledger areg checking date:jul Each aregister line item shows: - o the transaction's date (or the relevant posting's date if different, + o the transaction's date (or the relevant posting's date if different, see below) - o the names of all the other account(s) involved in this transaction + o the names of all the other account(s) involved in this transaction (probably abbreviated) o the total change to this account's balance from this transaction o the account's historical running balance after this transaction. - Transactions making a net change of zero are not shown by default; add + Transactions making a net change of zero are not shown by default; add the -E/--empty flag to show them. - For performance reasons, column widths are chosen based on the first - 1000 lines; this means unusually wide values in later lines can cause - visual discontinuities as column widths are adjusted. If you want to - ensure perfect alignment, at the cost of more time and memory, use the + For performance reasons, column widths are chosen based on the first + 1000 lines; this means unusually wide values in later lines can cause + visual discontinuities as column widths are adjusted. If you want to + ensure perfect alignment, at the cost of more time and memory, use the --align-all flag. - This command also supports the output destination and output format + This command also supports the output destination and output format options. The output formats supported are txt, csv, and json. aregister and custom posting dates - Transactions whose date is outside the report period can still be - shown, if they have a posting to this account dated inside the report - period. (And in this case it's the posting date that is shown.) This + Transactions whose date is outside the report period can still be + shown, if they have a posting to this account dated inside the report + period. (And in this case it's the posting date that is shown.) This ensures that aregister can show an accurate historical running balance, matching the one shown by register -H with the same arguments. - To filter strictly by transaction date instead, add the --txn-dates - flag. If you use this flag and some of your postings have custom + To filter strictly by transaction date instead, add the --txn-dates + flag. If you use this flag and some of your postings have custom dates, it's probably best to assume the running balance is wrong. balance balance, bal Show accounts and their balances. - balance is one of hledger's oldest and most versatile commands, for - listing account balances, balance changes, values, value changes and + balance is one of hledger's oldest and most versatile commands, for + listing account balances, balance changes, values, value changes and more, during one time period or many. Generally it shows a table, with rows representing accounts, and columns representing periods. - Note there are some higher-level variants of the balance command with - convenient defaults, which can be simpler to use: balancesheet, bal- + Note there are some higher-level variants of the balance command with + convenient defaults, which can be simpler to use: balancesheet, bal- ancesheetequity, cashflow and incomestatement. When you need more con- trol, then use balance. balance features - Here's a quick overview of the balance command's features, followed by - more detailed descriptions and examples. Many of these work with the + Here's a quick overview of the balance command's features, followed by + more detailed descriptions and examples. Many of these work with the higher-level commands as well. balance can show.. @@ -5916,7 +6027,7 @@ PART 4: COMMANDS ..with.. - o totals (-T), averages (-A), percentages (-%), inverted sign + o totals (-T), averages (-A), percentages (-%), inverted sign (--invert) o rows and columns swapped (--transpose) @@ -5928,24 +6039,24 @@ PART 4: COMMANDS o commodities displayed on the same line or multiple lines (--layout) This command supports the output destination and output format options, - with output formats txt, csv, json, and (multi-period reports only:) - html. In txt output in a colour-supporting terminal, negative amounts + with output formats txt, csv, json, and (multi-period reports only:) + html. In txt output in a colour-supporting terminal, negative amounts are shown in red. - The --related/-r flag shows the balance of the other postings in the + The --related/-r flag shows the balance of the other postings in the transactions of the postings which would normally be shown. Simple balance report - With no arguments, balance shows a list of all accounts and their - change of balance - ie, the sum of posting amounts, both inflows and - outflows - during the entire period of the journal. ("Simple" here - means just one column of numbers, covering a single period. You can + With no arguments, balance shows a list of all accounts and their + change of balance - ie, the sum of posting amounts, both inflows and + outflows - during the entire period of the journal. ("Simple" here + means just one column of numbers, covering a single period. You can also have multi-period reports, described later.) - For real-world accounts, these numbers will normally be their end bal- + For real-world accounts, these numbers will normally be their end bal- ance at the end of the journal period; more on this below. - Accounts are sorted by declaration order if any, and then alphabeti- + Accounts are sorted by declaration order if any, and then alphabeti- cally by account name. For instance (using examples/sample.journal): $ hledger -f examples/sample.journal bal @@ -5960,7 +6071,7 @@ PART 4: COMMANDS 0 Accounts with a zero balance (and no non-zero subaccounts, in tree mode - - see below) are hidden by default. Use -E/--empty to show them + - see below) are hidden by default. Use -E/--empty to show them (revealing assets:bank:checking here): $ hledger -f examples/sample.journal bal -E @@ -5975,12 +6086,12 @@ PART 4: COMMANDS -------------------- 0 - The total of the amounts displayed is shown as the last line, unless + The total of the amounts displayed is shown as the last line, unless -N/--no-total is used. Balance report line format For single-period balance reports displayed in the terminal (only), you - can use --format FMT to customise the format and content of each line. + can use --format FMT to customise the format and content of each line. Eg: $ hledger -f examples/sample.journal balance --format "%20(account) %12(total)" @@ -5998,7 +6109,7 @@ PART 4: COMMANDS 0 The FMT format string specifies the formatting applied to each - account/balance pair. It may contain any suitable text, with data + account/balance pair. It may contain any suitable text, with data fields interpolated like so: %[MIN][.MAX](FIELDNAME) @@ -6009,14 +6120,14 @@ PART 4: COMMANDS o FIELDNAME must be enclosed in parentheses, and can be one of: - o depth_spacer - a number of spaces equal to the account's depth, or + o depth_spacer - a number of spaces equal to the account's depth, or if MIN is specified, MIN * depth spaces. o account - the account's name o total - the account's balance/posted total, right justified - Also, FMT can begin with an optional prefix to control how multi-com- + Also, FMT can begin with an optional prefix to control how multi-com- modity amounts are rendered: o %_ - render on multiple lines, bottom-aligned (the default) @@ -6025,26 +6136,26 @@ PART 4: COMMANDS o %, - render on one line, comma-separated - There are some quirks. Eg in one-line mode, %(depth_spacer) has no - effect, instead %(account) has indentation built in. Experimentation + There are some quirks. Eg in one-line mode, %(depth_spacer) has no + effect, instead %(account) has indentation built in. Experimentation may be needed to get pleasing results. Some example formats: o %(total) - the account's total - o %-20.20(account) - the account's name, left justified, padded to 20 + o %-20.20(account) - the account's name, left justified, padded to 20 characters and clipped at 20 characters - o %,%-50(account) %25(total) - account name padded to 50 characters, - total padded to 20 characters, with multiple commodities rendered on + o %,%-50(account) %25(total) - account name padded to 50 characters, + total padded to 20 characters, with multiple commodities rendered on one line - o %20(total) %2(depth_spacer)%-(account) - the default format for the + o %20(total) %2(depth_spacer)%-(account) - the default format for the single-column balance report Filtered balance report - You can show fewer accounts, a different time period, totals from + You can show fewer accounts, a different time period, totals from cleared transactions only, etc. by using query arguments or options to limit the postings being matched. Eg: @@ -6054,10 +6165,10 @@ PART 4: COMMANDS $-2 List or tree mode - By default, or with -l/--flat, accounts are shown as a flat list with + By default, or with -l/--flat, accounts are shown as a flat list with their full names visible, as in the examples above. - With -t/--tree, the account hierarchy is shown, with subaccounts' + With -t/--tree, the account hierarchy is shown, with subaccounts' "leaf" names indented below their parent: $ hledger -f examples/sample.journal balance @@ -6077,26 +6188,26 @@ PART 4: COMMANDS Notes: o "Boring" accounts are combined with their subaccount for more compact - output, unless --no-elide is used. Boring accounts have no balance - of their own and just one subaccount (eg assets:bank and liabilities + output, unless --no-elide is used. Boring accounts have no balance + of their own and just one subaccount (eg assets:bank and liabilities above). - o All balances shown are "inclusive", ie including the balances from - all subaccounts. Note this means some repetition in the output, + o All balances shown are "inclusive", ie including the balances from + all subaccounts. Note this means some repetition in the output, which requires explanation when sharing reports with non-plaintextac- - counting-users. A tree mode report's final total is the sum of the + counting-users. A tree mode report's final total is the sum of the top-level balances shown, not of all the balances shown. - o Each group of sibling accounts (ie, under a common parent) is sorted + o Each group of sibling accounts (ie, under a common parent) is sorted separately. Depth limiting - With a depth:NUM query, or --depth NUM option, or just -NUM (eg: -3) - balance reports will show accounts only to the specified depth, hiding - the deeper subaccounts. This can be useful for getting an overview + With a depth:NUM query, or --depth NUM option, or just -NUM (eg: -3) + balance reports will show accounts only to the specified depth, hiding + the deeper subaccounts. This can be useful for getting an overview without too much detail. - Account balances at the depth limit always include the balances from + Account balances at the depth limit always include the balances from any deeper subaccounts (even in list mode). Eg, limiting to depth 1: $ hledger -f examples/sample.journal balance -1 @@ -6108,7 +6219,7 @@ PART 4: COMMANDS 0 Dropping top-level accounts - You can also hide one or more top-level account name parts, using + You can also hide one or more top-level account name parts, using --drop NUM. This can be useful for hiding repetitive top-level account names: @@ -6120,54 +6231,54 @@ PART 4: COMMANDS Showing declared accounts - With --declared, accounts which have been declared with an account - directive will be included in the balance report, even if they have no + With --declared, accounts which have been declared with an account + directive will be included in the balance report, even if they have no transactions. (Since they will have a zero balance, you will also need -E/--empty to see them.) - More precisely, leaf declared accounts (with no subaccounts) will be + More precisely, leaf declared accounts (with no subaccounts) will be included, since those are usually the more useful in reports. - The idea of this is to be able to see a useful "complete" balance - report, even when you don't have transactions in all of your declared + The idea of this is to be able to see a useful "complete" balance + report, even when you don't have transactions in all of your declared accounts yet. Sorting by amount - With -S/--sort-amount, accounts with the largest (most positive) bal- - ances are shown first. Eg: hledger bal expenses -MAS shows your big- - gest averaged monthly expenses first. When more than one commodity is - present, they will be sorted by the alphabetically earliest commodity - first, and then by subsequent commodities (if an amount is missing a + With -S/--sort-amount, accounts with the largest (most positive) bal- + ances are shown first. Eg: hledger bal expenses -MAS shows your big- + gest averaged monthly expenses first. When more than one commodity is + present, they will be sorted by the alphabetically earliest commodity + first, and then by subsequent commodities (if an amount is missing a commodity, it is treated as 0). - Revenues and liability balances are typically negative, however, so -S - shows these in reverse order. To work around this, you can add - --invert to flip the signs. (Or, use one of the higher-level reports, - which flip the sign automatically. Eg: hledger incomestatement -MAS). + Revenues and liability balances are typically negative, however, so -S + shows these in reverse order. To work around this, you can add + --invert to flip the signs. (Or, use one of the higher-level reports, + which flip the sign automatically. Eg: hledger incomestatement -MAS). Percentages - With -%/--percent, balance reports show each account's value expressed + With -%/--percent, balance reports show each account's value expressed as a percentage of the (column) total. Note it is not useful to calculate percentages if the amounts in a col- - umn have mixed signs. In this case, make a separate report for each + umn have mixed signs. In this case, make a separate report for each sign, eg: $ hledger bal -% amt:`>0` $ hledger bal -% amt:`<0` - Similarly, if the amounts in a column have mixed commodities, convert - them to one commodity with -B, -V, -X or --value, or make a separate + Similarly, if the amounts in a column have mixed commodities, convert + them to one commodity with -B, -V, -X or --value, or make a separate report for each commodity: $ hledger bal -% cur:\\$ $ hledger bal -% cur:EUR Multi-period balance report - With a report interval (set by the -D/--daily, -W/--weekly, - -M/--monthly, -Q/--quarterly, -Y/--yearly, or -p/--period flag), bal- - ance shows a tabular report, with columns representing successive time + With a report interval (set by the -D/--daily, -W/--weekly, + -M/--monthly, -Q/--quarterly, -Y/--yearly, or -p/--period flag), bal- + ance shows a tabular report, with columns representing successive time periods (and a title): $ hledger -f examples/sample.journal bal --quarterly income expenses -E @@ -6188,21 +6299,21 @@ PART 4: COMMANDS encompass the displayed subperiods (so that the first and last subpe- riods have the same duration as the others). - o Leading and trailing periods (columns) containing all zeroes are not + o Leading and trailing periods (columns) containing all zeroes are not shown, unless -E/--empty is used. - o Accounts (rows) containing all zeroes are not shown, unless + o Accounts (rows) containing all zeroes are not shown, unless -E/--empty is used. - o Amounts with many commodities are shown in abbreviated form, unless + o Amounts with many commodities are shown in abbreviated form, unless --no-elide is used. (experimental) - o Average and/or total columns can be added with the -A/--average and + o Average and/or total columns can be added with the -A/--average and -T/--row-total flags. o The --transpose flag can be used to exchange rows and columns. - o The --pivot FIELD option causes a different transaction field to be + o The --pivot FIELD option causes a different transaction field to be used as "account name". See PIVOTING. Multi-period reports with many periods can be too wide for easy viewing @@ -6216,57 +6327,57 @@ PART 4: COMMANDS o Reduce the terminal's font size - o View with a pager like less, eg: hledger bal -D --color=yes | less + o View with a pager like less, eg: hledger bal -D --color=yes | less -RS - o Output as CSV and use a CSV viewer like visidata (hledger bal -D -O - csv | vd -f csv), Emacs' csv-mode (M-x csv-mode, C-c C-a), or a + o Output as CSV and use a CSV viewer like visidata (hledger bal -D -O + csv | vd -f csv), Emacs' csv-mode (M-x csv-mode, C-c C-a), or a spreadsheet (hledger bal -D -o a.csv && open a.csv) - o Output as HTML and view with a browser: hledger bal -D -o a.html && + o Output as HTML and view with a browser: hledger bal -D -o a.html && open a.html Balance change, end balance - It's important to be clear on the meaning of the numbers shown in bal- + It's important to be clear on the meaning of the numbers shown in bal- ance reports. Here is some terminology we use: - A balance change is the net amount added to, or removed from, an + A balance change is the net amount added to, or removed from, an account during some period. - An end balance is the amount accumulated in an account as of some date - (and some time, but hledger doesn't store that; assume end of day in + An end balance is the amount accumulated in an account as of some date + (and some time, but hledger doesn't store that; assume end of day in your timezone). It is the sum of previous balance changes. - We call it a historical end balance if it includes all balance changes + We call it a historical end balance if it includes all balance changes since the account was created. For a real world account, this means it - will match the "historical record", eg the balances reported in your + will match the "historical record", eg the balances reported in your bank statements or bank web UI. (If they are correct!) - In general, balance changes are what you want to see when reviewing + In general, balance changes are what you want to see when reviewing revenues and expenses, and historical end balances are what you want to see when reviewing or reconciling asset, liability and equity accounts. - balance shows balance changes by default. To see accurate historical + balance shows balance changes by default. To see accurate historical end balances: - 1. Initialise account starting balances with an "opening balances" - transaction (a transfer from equity to the account), unless the + 1. Initialise account starting balances with an "opening balances" + transaction (a transfer from equity to the account), unless the journal covers the account's full lifetime. 2. Include all of of the account's prior postings in the report, by not - specifying a report start date, or by using the -H/--historical + specifying a report start date, or by using the -H/--historical flag. (-H causes report start date to be ignored when summing post- ings.) Balance report types - The balance command is quite flexible; here is the full detail on how - to control what it reports. If the following seems complicated, don't + The balance command is quite flexible; here is the full detail on how + to control what it reports. If the following seems complicated, don't worry - this is for advanced reporting, and it does typically take some time and experimentation to get clear on all these report modes. There are three important option groups: - hledger balance [CALCULATIONTYPE] [ACCUMULATIONTYPE] [VALUATIONTYPE] + hledger balance [CALCULATIONTYPE] [ACCUMULATIONTYPE] [VALUATIONTYPE] ... Calculation type @@ -6278,44 +6389,44 @@ PART 4: COMMANDS each account/period) o --valuechange : show the change in period-end historical balance val- - ues (caused by deposits, withdrawals, and/or market price fluctua- + ues (caused by deposits, withdrawals, and/or market price fluctua- tions) - o --gain : show the unrealised capital gain/loss, (the current valued + o --gain : show the unrealised capital gain/loss, (the current valued balance minus each amount's original cost) Accumulation type - How amounts should accumulate across report periods. Another way to - say it: which time period's postings should contribute to each cell's + How amounts should accumulate across report periods. Another way to + say it: which time period's postings should contribute to each cell's calculation. It is one of: - o --change : calculate with postings from column start to column end, - ie "just this column". Typically used to see revenues/expenses. + o --change : calculate with postings from column start to column end, + ie "just this column". Typically used to see revenues/expenses. (default for balance, incomestatement) - o --cumulative : calculate with postings from report start to column - end, ie "previous columns plus this column". Typically used to show + o --cumulative : calculate with postings from report start to column + end, ie "previous columns plus this column". Typically used to show changes accumulated since the report's start date. Not often used. - o --historical/-H : calculate with postings from journal start to col- - umn end, ie "all postings from before report start date until this - column's end". Typically used to see historical end balances of + o --historical/-H : calculate with postings from journal start to col- + umn end, ie "all postings from before report start date until this + column's end". Typically used to see historical end balances of assets/liabilities/equity. (default for balancesheet, balancesheete- quity, cashflow) Valuation type - Which kind of value or cost conversion should be applied, if any, + Which kind of value or cost conversion should be applied, if any, before displaying the report. It is one of: o no valuation type : don't convert to cost or value (default) - o --value=cost[,COMM] : convert amounts to cost (then optionally to + o --value=cost[,COMM] : convert amounts to cost (then optionally to some other commodity) - o --value=then[,COMM] : convert amounts to market value on transaction + o --value=then[,COMM] : convert amounts to market value on transaction dates - o --value=end[,COMM] : convert amounts to market value on period end + o --value=end[,COMM] : convert amounts to market value on period end date(s) (default with --valuechange, --gain) @@ -6326,7 +6437,7 @@ PART 4: COMMANDS or one of the equivalent simpler flags: - o -B/--cost : like --value=cost (though, note --cost and --value are + o -B/--cost : like --value=cost (though, note --cost and --value are independent options which can both be used at once) o -V/--market : like --value=end @@ -6336,13 +6447,13 @@ PART 4: COMMANDS See Cost reporting and Valuation for more about these. Combining balance report types - Most combinations of these options should produce reasonable reports, - but if you find any that seem wrong or misleading, let us know. The + Most combinations of these options should produce reasonable reports, + but if you find any that seem wrong or misleading, let us know. The following restrictions are applied: o --valuechange implies --value=end - o --valuechange makes --change the default when used with the bal- + o --valuechange makes --change the default when used with the bal- ancesheet/balancesheetequity commands o --cumulative or --historical disables --row-total/-T @@ -6357,26 +6468,26 @@ PART 4: COMMANDS lation:v ----------------------------------------------------------------------------------- --change change in period sum of posting- period-end DATE-value of - date market val- value of change change in + date market val- value of change change in ues in period in period period --cumu- change from sum of posting- period-end DATE-value of - lative report start to date market val- value of change change from + lative report start to date market val- value of change change from period end ues from report from report report start start to period start to period to period end end end --his- change from sum of posting- period-end DATE-value of - torical journal start to date market val- value of change change from - /-H period end (his- ues from journal from journal journal start + torical journal start to date market val- value of change change from + /-H period end (his- ues from journal from journal journal start torical end bal- start to period start to period to period end ance) end end Budget report - The --budget report type activates extra columns showing any budget - goals for each account and period. The budget goals are defined by + The --budget report type activates extra columns showing any budget + goals for each account and period. The budget goals are defined by periodic transactions. This is useful for comparing planned and actual income, expenses, time usage, etc. - For example, you can take average monthly expenses in the common + For example, you can take average monthly expenses in the common expense categories to construct a minimal monthly budget: ;; Budget @@ -6423,26 +6534,26 @@ PART 4: COMMANDS This is different from a normal balance report in several ways: - o Only accounts with budget goals during the report period are shown, + o Only accounts with budget goals during the report period are shown, by default. - o In each column, in square brackets after the actual amount, budget - goal amounts are shown, and the actual/goal percentage. (Note: bud- + o In each column, in square brackets after the actual amount, budget + goal amounts are shown, and the actual/goal percentage. (Note: bud- get goals should be in the same commodity as the actual amount.) - o All parent accounts are always shown, even in list mode. Eg assets, + o All parent accounts are always shown, even in list mode. Eg assets, assets:bank, and expenses above. - o Amounts always include all subaccounts, budgeted or unbudgeted, even + o Amounts always include all subaccounts, budgeted or unbudgeted, even in list mode. - This means that the numbers displayed will not always add up! Eg - above, the expenses actual amount includes the gifts and supplies + This means that the numbers displayed will not always add up! Eg + above, the expenses actual amount includes the gifts and supplies transactions, but the expenses:gifts and expenses:supplies accounts are not shown, as they have no budget amounts declared. - This can be confusing. When you need to make things clearer, use the - -E/--empty flag, which will reveal all accounts including unbudgeted + This can be confusing. When you need to make things clearer, use the + -E/--empty flag, which will reveal all accounts including unbudgeted ones, giving the full picture. Eg: $ hledger balance -M --budget --empty @@ -6484,12 +6595,12 @@ PART 4: COMMANDS For more examples and notes, see Budgeting. Budget report start date - This might be a bug, but for now: when making budget reports, it's a + This might be a bug, but for now: when making budget reports, it's a good idea to explicitly set the report's start date to the first day of - a reporting period, because a periodic rule like ~ monthly generates - its transactions on the 1st of each month, and if your journal has no - regular transactions on the 1st, the default report start date could - exclude that budget goal, which can be a little surprising. Eg here + a reporting period, because a periodic rule like ~ monthly generates + its transactions on the 1st of each month, and if your journal has no + regular transactions on the 1st, the default report start date could + exclude that budget goal, which can be a little surprising. Eg here the default report period is just the day of 2020-01-15: ~ monthly in 2020 @@ -6508,9 +6619,9 @@ PART 4: COMMANDS --------------++------------ || $400 - To avoid this, specify the budget report's period, or at least the - start date, with -b/-e/-p/date:, to ensure it includes the budget goal - transactions (periodic transactions) that you want. Eg, adding -b + To avoid this, specify the budget report's period, or at least the + start date, with -b/-e/-p/date:, to ensure it includes the budget goal + transactions (periodic transactions) that you want. Eg, adding -b 2020/1/1 to the above: $ hledger bal expenses --budget -b 2020/1/1 @@ -6523,12 +6634,12 @@ PART 4: COMMANDS || $400 [80% of $500] Budgets and subaccounts - You can add budgets to any account in your account hierarchy. If you + You can add budgets to any account in your account hierarchy. If you have budgets on both parent account and some of its children, then bud- - get(s) of the child account(s) would be added to the budget of their + get(s) of the child account(s) would be added to the budget of their parent, much like account balances behave. - In the most simple case this means that once you add a budget to any + In the most simple case this means that once you add a budget to any account, all its parents would have budget as well. To illustrate this, consider the following budget: @@ -6538,13 +6649,13 @@ PART 4: COMMANDS expenses:personal:electronics $100.00 liabilities - With this, monthly budget for electronics is defined to be $100 and - budget for personal expenses is an additional $1000, which implicitly + With this, monthly budget for electronics is defined to be $100 and + budget for personal expenses is an additional $1000, which implicitly means that budget for both expenses:personal and expenses is $1100. - Transactions in expenses:personal:electronics will be counted both - towards its $100 budget and $1100 of expenses:personal , and transac- - tions in any other subaccount of expenses:personal would be counted + Transactions in expenses:personal:electronics will be counted both + towards its $100 budget and $1100 of expenses:personal , and transac- + tions in any other subaccount of expenses:personal would be counted towards only towards the budget of expenses:personal. For example, let's consider these transactions: @@ -6570,9 +6681,9 @@ PART 4: COMMANDS expenses:personal $30.00 liabilities - As you can see, we have transactions in expenses:personal:electron- - ics:upgrades and expenses:personal:train tickets, and since both of - these accounts are without explicitly defined budget, these transac- + As you can see, we have transactions in expenses:personal:electron- + ics:upgrades and expenses:personal:train tickets, and since both of + these accounts are without explicitly defined budget, these transac- tions would be counted towards budgets of expenses:personal:electronics and expenses:personal accordingly: @@ -6588,7 +6699,7 @@ PART 4: COMMANDS -------------------------------++------------------------------- || 0 [ 0] - And with --empty, we can get a better picture of budget allocation and + And with --empty, we can get a better picture of budget allocation and consumption: $ hledger balance --budget -M --empty @@ -6607,28 +6718,28 @@ PART 4: COMMANDS Selecting budget goals The budget report evaluates periodic transaction rules to generate spe- - cial "goal transactions", which generate the goal amounts for each - account in each report subperiod. When troubleshooting, you can use + cial "goal transactions", which generate the goal amounts for each + account in each report subperiod. When troubleshooting, you can use the print command to show these as forecasted transactions: $ hledger print --forecast=BUDGETREPORTPERIOD tag:generated - By default, the budget report uses all available periodic transaction - rules to generate goals. This includes rules with a different report - interval from your report. Eg if you have daily, weekly and monthly - periodic rules, all of these will contribute to the goals in a monthly + By default, the budget report uses all available periodic transaction + rules to generate goals. This includes rules with a different report + interval from your report. Eg if you have daily, weekly and monthly + periodic rules, all of these will contribute to the goals in a monthly budget report. - You can select a subset of periodic rules by providing an argument to - the --budget flag. --budget=DESCPAT will match all periodic rules + You can select a subset of periodic rules by providing an argument to + the --budget flag. --budget=DESCPAT will match all periodic rules whose description contains DESCPAT, a case-insensitive substring (not a - regular expression or query). This means you can give your periodic - rules descriptions (remember that two spaces are needed), and then + regular expression or query). This means you can give your periodic + rules descriptions (remember that two spaces are needed), and then select from multiple budgets defined in your journal. Data layout - The --layout option affects how balance reports show multi-commodity - amounts and commodity symbols, which can improve readability. It can + The --layout option affects how balance reports show multi-commodity + amounts and commodity symbols, which can improve readability. It can also normalise the data for easy consumption by other programs. It has four possible values: @@ -6640,10 +6751,10 @@ PART 4: COMMANDS o --layout=bare: commodity symbols are in their own column, amounts are bare numbers - o --layout=tidy: data is normalised to easily-consumed "tidy" form, + o --layout=tidy: data is normalised to easily-consumed "tidy" form, with one row per data value - Here are the --layout modes supported by each output format; note only + Here are the --layout modes supported by each output format; note only CSV output supports all of them: @@ -6667,7 +6778,7 @@ PART 4: COMMANDS ------------------++-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- || 10.00 ITOT, 337.18 USD, 12.00 VEA, 106.00 VHT 70.00 GLD, 18.00 ITOT, -98.12 USD, 10.00 VEA, 18.00 VHT -11.00 ITOT, 4881.44 USD, 14.00 VEA, 170.00 VHT 70.00 GLD, 17.00 ITOT, 5120.50 USD, 36.00 VEA, 294.00 VHT - o Limited wide layout. A width limit reduces the width, but some com- + o Limited wide layout. A width limit reduces the width, but some com- modities will be hidden: $ hledger -f examples/bcexample.hledger bal assets:us:etrade -3 -T -Y --layout=wide,32 @@ -6679,7 +6790,7 @@ PART 4: COMMANDS ------------------++--------------------------------------------------------------------------------------------------------------------------- || 10.00 ITOT, 337.18 USD, 2 more.. 70.00 GLD, 18.00 ITOT, 3 more.. -11.00 ITOT, 3 more.. 70.00 GLD, 17.00 ITOT, 3 more.. - o Tall layout. Each commodity gets a new line (may be different in + o Tall layout. Each commodity gets a new line (may be different in each column), and account names are repeated: $ hledger -f examples/bcexample.hledger bal assets:us:etrade -3 -T -Y --layout=tall @@ -6699,7 +6810,7 @@ PART 4: COMMANDS || 106.00 VHT 10.00 VEA 170.00 VHT 36.00 VEA || 18.00 VHT 294.00 VHT - o Bare layout. Commodity symbols are kept in one column, each commod- + o Bare layout. Commodity symbols are kept in one column, each commod- ity gets its own report row, account names are repeated: $ hledger -f examples/bcexample.hledger bal assets:us:etrade -3 -T -Y --layout=bare @@ -6719,7 +6830,7 @@ PART 4: COMMANDS || VEA 12.00 10.00 14.00 36.00 || VHT 106.00 18.00 170.00 294.00 - o Bare layout also affects CSV output, which is useful for producing + o Bare layout also affects CSV output, which is useful for producing data that is easier to consume, eg for making charts: $ hledger -f examples/bcexample.hledger bal assets:us:etrade -3 -O csv --layout=bare @@ -6736,7 +6847,7 @@ PART 4: COMMANDS "total","VHT","294.00" o Tidy layout produces normalised "tidy data", where every variable has - its own column and each row represents a single data point. See + its own column and each row represents a single data point. See https://cran.r-project.org/web/packages/tidyr/vignettes/tidy- data.html for more. This is the easiest kind of data for other soft- ware to consume. Here's how it looks: @@ -6763,25 +6874,25 @@ PART 4: COMMANDS Some frequently used balance options/reports are: o bal -M revenues expenses - Show revenues/expenses in each month. Also available as the incomes- + Show revenues/expenses in each month. Also available as the incomes- tatement command. o bal -M -H assets liabilities - Show historical asset/liability balances at each month end. Also + Show historical asset/liability balances at each month end. Also available as the balancesheet command. o bal -M -H assets liabilities equity - Show historical asset/liability/equity balances at each month end. + Show historical asset/liability/equity balances at each month end. Also available as the balancesheetequity command. o bal -M assets not:receivable - Show changes to liquid assets in each month. Also available as the + Show changes to liquid assets in each month. Also available as the cashflow command. Also: o bal -M expenses -2 -SA - Show monthly expenses summarised to depth 2 and sorted by average + Show monthly expenses summarised to depth 2 and sorted by average amount. o bal -M --budget expenses @@ -6796,14 +6907,14 @@ PART 4: COMMANDS balancesheet balancesheet, bs - This command displays a balance sheet, showing historical ending bal- + This command displays a balance sheet, showing historical ending bal- ances of asset and liability accounts. (To see equity as well, use the - balancesheetequity command.) Amounts are shown with normal positive + balancesheetequity command.) Amounts are shown with normal positive sign, as in conventional financial statements. - This report shows accounts declared with the Asset, Cash or Liability - type (see account types). Or if no such accounts are declared, it - shows top-level accounts named asset or liability (case insensitive, + This report shows accounts declared with the Asset, Cash or Liability + type (see account types). Or if no such accounts are declared, it + shows top-level accounts named asset or liability (case insensitive, plurals allowed) and their subaccounts. Example: @@ -6828,24 +6939,24 @@ PART 4: COMMANDS 0 This command is a higher-level variant of the balance command, and sup- - ports many of that command's features, such as multi-period reports. - It is similar to hledger balance -H assets liabilities, but with - smarter account detection, and liabilities displayed with their sign + ports many of that command's features, such as multi-period reports. + It is similar to hledger balance -H assets liabilities, but with + smarter account detection, and liabilities displayed with their sign flipped. - This command also supports the output destination and output format - options The output formats supported are txt, csv, html, and (experi- + This command also supports the output destination and output format + options The output formats supported are txt, csv, html, and (experi- mental) json. balancesheetequity balancesheetequity, bse - This command displays a balance sheet, showing historical ending bal- - ances of asset, liability and equity accounts. Amounts are shown with + This command displays a balance sheet, showing historical ending bal- + ances of asset, liability and equity accounts. Amounts are shown with normal positive sign, as in conventional financial statements. - This report shows accounts declared with the Asset, Cash, Liability or - Equity type (see account types). Or if no such accounts are declared, - it shows top-level accounts named asset, liability or equity (case + This report shows accounts declared with the Asset, Cash, Liability or + Equity type (see account types). Or if no such accounts are declared, + it shows top-level accounts named asset, liability or equity (case insensitive, plurals allowed) and their subaccounts. Example: @@ -6875,31 +6986,31 @@ PART 4: COMMANDS 0 This command is a higher-level variant of the balance command, and sup- - ports many of that command's features, such as multi-period reports. + ports many of that command's features, such as multi-period reports. It is similar to hledger balance -H assets liabilities equity, but with - smarter account detection, and liabilities/equity displayed with their + smarter account detection, and liabilities/equity displayed with their sign flipped. - This command also supports the output destination and output format - options The output formats supported are txt, csv, html, and (experi- + This command also supports the output destination and output format + options The output formats supported are txt, csv, html, and (experi- mental) json. cashflow cashflow, cf - This command displays a cashflow statement, showing the inflows and - outflows affecting "cash" (ie, liquid, easily convertible) assets. - Amounts are shown with normal positive sign, as in conventional finan- + This command displays a cashflow statement, showing the inflows and + outflows affecting "cash" (ie, liquid, easily convertible) assets. + Amounts are shown with normal positive sign, as in conventional finan- cial statements. - This report shows accounts declared with the Cash type (see account + This report shows accounts declared with the Cash type (see account types). Or if no such accounts are declared, it shows accounts - o under a top-level account named asset (case insensitive, plural + o under a top-level account named asset (case insensitive, plural allowed) o whose name contains some variation of cash, bank, checking or saving. - More precisely: all accounts matching this case insensitive regular + More precisely: all accounts matching this case insensitive regular expression: ^assets?(:.+)?:(cash|bank|che(ck|que?)(ing)?|savings?|currentcash)(:|$) @@ -6923,22 +7034,22 @@ PART 4: COMMANDS $-1 This command is a higher-level variant of the balance command, and sup- - ports many of that command's features, such as multi-period reports. - It is similar to hledger balance assets not:fixed not:investment + ports many of that command's features, such as multi-period reports. + It is similar to hledger balance assets not:fixed not:investment not:receivable, but with smarter account detection. - This command also supports the output destination and output format - options The output formats supported are txt, csv, html, and (experi- + This command also supports the output destination and output format + options The output formats supported are txt, csv, html, and (experi- mental) json. check check Check for various kinds of errors in your data. - hledger provides a number of built-in error checks to help prevent - problems in your data. Some of these are run automatically; or, you - can use this check command to run them on demand, with no output and a - zero exit code if all is well. Specify their names (or a prefix) as + hledger provides a number of built-in error checks to help prevent + problems in your data. Some of these are run automatically; or, you + can use this check command to run them on demand, with no output and a + zero exit code if all is well. Specify their names (or a prefix) as argument(s). Some examples: @@ -6947,7 +7058,7 @@ PART 4: COMMANDS hledger check -s # basic + strict checks hledger check ordereddates payees # basic + two other checks - If you are an Emacs user, you can also configure flycheck-hledger to + If you are an Emacs user, you can also configure flycheck-hledger to run these checks, providing instant feedback as you edit the journal. Here are the checks currently available: @@ -6959,98 +7070,98 @@ PART 4: COMMANDS o parseable - data files are well-formed and can be successfully parsed o balancedwithautoconversion - all transactions are balanced, inferring - missing amounts where necessary, and possibly converting commodities + missing amounts where necessary, and possibly converting commodities using costs or automatically-inferred costs - o assertions - all balance assertions in the journal are passing. + o assertions - all balance assertions in the journal are passing. (This check can be disabled with -I/--ignore-assertions.) Strict checks These additional checks are run when the -s/--strict (strict mode) flag - is used. Or, they can be run by giving their names as arguments to + is used. Or, they can be run by giving their names as arguments to check: o accounts - all account names used by transactions have been declared o commodities - all commodity symbols used have been declared - o balancednoautoconversion - transactions are balanced, possibly using + o balancednoautoconversion - transactions are balanced, possibly using explicit costs but not inferred ones Other checks - These checks can be run only by giving their names as arguments to - check. They are more specialised and not desirable for everyone, + These checks can be run only by giving their names as arguments to + check. They are more specialised and not desirable for everyone, therefore optional: o ordereddates - transactions are ordered by date within each file o payees - all payees used by transactions have been declared - o recentassertions - all accounts with balance assertions have a bal- + o recentassertions - all accounts with balance assertions have a bal- ance assertion no more than 7 days before their latest posting o uniqueleafnames - all account leaf names are unique Custom checks - A few more checks are are available as separate add-on commands, in + A few more checks are are available as separate add-on commands, in https://github.com/simonmichael/hledger/tree/master/bin: - o hledger-check-tagfiles - all tag values containing / (a forward + o hledger-check-tagfiles - all tag values containing / (a forward slash) exist as file paths - o hledger-check-fancyassertions - more complex balance assertions are + o hledger-check-fancyassertions - more complex balance assertions are passing You could make similar scripts to perform your own custom checks. See: Cookbook -> Scripting. More about specific checks - hledger check recentassertions will complain if any balance-asserted + hledger check recentassertions will complain if any balance-asserted account does not have a balance assertion within 7 days before its lat- - est posting. This aims to prevent the situation where you are regu- - larly updating your journal, but forgetting to check your balances - against the real world, then one day must dig back through months of - data to find an error. It assumes that adding a balance assertion - requires/reminds you to check the real-world balance. That may not be - true if you auto-generate balance assertions from bank data; in that - case, I recommend to import transactions uncleared, then use the man- - ual-review-and-mark-cleared phase as a reminder to check the latest + est posting. This aims to prevent the situation where you are regu- + larly updating your journal, but forgetting to check your balances + against the real world, then one day must dig back through months of + data to find an error. It assumes that adding a balance assertion + requires/reminds you to check the real-world balance. That may not be + true if you auto-generate balance assertions from bank data; in that + case, I recommend to import transactions uncleared, then use the man- + ual-review-and-mark-cleared phase as a reminder to check the latest assertions against real-world balances. close close, equity - Prints a sample "closing" transaction bringing specified account bal- - ances to zero, and an inverse "opening" transaction restoring the same + Prints a sample "closing" transaction bringing specified account bal- + ances to zero, and an inverse "opening" transaction restoring the same account balances. - If like most people you split your journal files by time, eg by year: - at the end of the year you can use this command to "close out" your - asset and liability (and perhaps equity) balances in the old file, and - reinitialise them in the new file. This helps ensure that report bal- - ances remain correct whether you are including old files or not. - (Because all closing/opening transactions except the very first will + If like most people you split your journal files by time, eg by year: + at the end of the year you can use this command to "close out" your + asset and liability (and perhaps equity) balances in the old file, and + reinitialise them in the new file. This helps ensure that report bal- + ances remain correct whether you are including old files or not. + (Because all closing/opening transactions except the very first will cancel out - see example below.) Some people also use this command to close out revenue and expense bal- - ances at the end of an accounting period. This properly records the - period's profit/loss as "retained earnings" (part of equity), and + ances at the end of an accounting period. This properly records the + period's profit/loss as "retained earnings" (part of equity), and allows the accounting equation (A-L=E) to balance, which you could then check by the bse report's zero total. - You can print just the closing transaction by using the --close flag, + You can print just the closing transaction by using the --close flag, or just the opening transaction with the --open flag. Their descriptions are closing balances and opening balances by - default; you can customise these with the --close-desc and --open-desc + default; you can customise these with the --close-desc and --open-desc options. - Just one balancing equity posting is used by default, with the amount + Just one balancing equity posting is used by default, with the amount left implicit. The default account name is equity:opening/closing bal- - ances. You can customise the account name(s) with --close-acct and - --open-acct. (If you specify only one of these, it will be used for + ances. You can customise the account name(s) with --close-acct and + --open-acct. (If you specify only one of these, it will be used for both.) - With --x/--explicit, the equity posting's amount will be shown explic- + With --x/--explicit, the equity posting's amount will be shown explic- itly, and if it involves multiple commodities, there will be a separate equity posting for each commodity (as in the print command). @@ -7058,26 +7169,26 @@ PART 4: COMMANDS balances (good for troubleshooting). close and costs - Costs are ignored (and discarded) by closing/opening transactions, by - default. With --show-costs, they are preserved; there will be a sepa- - rate equity posting for each cost in each commodity. This means bal- - ance -B reports will look the same after the transition. Note if you + Costs are ignored (and discarded) by closing/opening transactions, by + default. With --show-costs, they are preserved; there will be a sepa- + rate equity posting for each cost in each commodity. This means bal- + ance -B reports will look the same after the transition. Note if you have many foreign currency or investment transactions, this will gener- ate very large journal entries. close date - The default closing date is yesterday, or the journal's end date, + The default closing date is yesterday, or the journal's end date, whichever is later. - Unless you are running close on exactly the first day of the new - period, you'll want to override the closing date. This is done by - specifying a report end date, where "last day of the report period" - will be the closing date. The opening date is always the following - day. So to close on (end of) 2020-12-31 and open on (start of) + Unless you are running close on exactly the first day of the new + period, you'll want to override the closing date. This is done by + specifying a report end date, where "last day of the report period" + will be the closing date. The opening date is always the following + day. So to close on (end of) 2020-12-31 and open on (start of) 2021-01-01, any of these will work: - end date argu- explanation + end date argu- explanation ment -------------------------------------------------------------------------- -e 2021-01-01 end dates are exclusive @@ -7107,17 +7218,17 @@ PART 4: COMMANDS Hiding opening/closing transactions Although the closing/opening transactions cancel out, they will be vis- - ible in reports like print and register, creating some visual clutter. + ible in reports like print and register, creating some visual clutter. You can exclude them all with a query, like: $ hledger print not:desc:'opening|closing' # less typing $ hledger print not:'equity:opening/closing balances' # more precise - But when reporting on multiple files, this can get a bit tricky; you + But when reporting on multiple files, this can get a bit tricky; you may need to keep the earliest opening balances, for a historical regis- - ter report; or you may need to suppress a closing transaction, to see - year-end balances. If you find yourself needing more precise queries, - here's one solution: add more easily-matched tags to opening/closing + ter report; or you may need to suppress a closing transaction, to see + year-end balances. If you find yourself needing more precise queries, + here's one solution: add more easily-matched tags to opening/closing transactions, like this: ; 2019.journal @@ -7152,18 +7263,18 @@ PART 4: COMMANDS # 2020 year end balances, suppressing 2020 closing txn close and balance assertions - The closing and opening transactions will include balance assertions, - verifying that the accounts have first been reset to zero and then - restored to their previous balance. These provide valuable error - checking, alerting you when things get out of line, but you can ignore + The closing and opening transactions will include balance assertions, + verifying that the accounts have first been reset to zero and then + restored to their previous balance. These provide valuable error + checking, alerting you when things get out of line, but you can ignore them temporarily with -I or just remove them if you prefer. You probably shouldn't use status or realness filters (like -C or -R or status:) with close, or the generated balance assertions will depend on - these flags. Likewise, if you run this command with --auto, the bal- + these flags. Likewise, if you run this command with --auto, the bal- ance assertions would probably always require --auto. - Multi-day transactions (where some postings have a different date) + Multi-day transactions (where some postings have a different date) break the balance assertions, because the money is temporarily "invisi- ble" while in transit: @@ -7171,8 +7282,8 @@ PART 4: COMMANDS expenses:food 5 assets:bank:checking -5 ; date: 2021/1/2 - To fix the assertions, you can add a temporary account to track such - in-transit money (splitting the multi-day transaction into two single- + To fix the assertions, you can add a temporary account to track such + in-transit money (splitting the multi-day transaction into two single- day transactions): ; in 2020.journal: @@ -7186,8 +7297,8 @@ PART 4: COMMANDS assets:bank:checking Example: close revenue/expense accounts to retained earnings - For this, use --close to suppress the opening transaction, as it's not - needed. Also you'll want to change the equity account name to your + For this, use --close to suppress the opening transaction, as it's not + needed. Also you'll want to change the equity account name to your equivalent of "equity:retained earnings". Closing 2021's first quarter revenues/expenses: @@ -7200,13 +7311,13 @@ PART 4: COMMANDS $ hledger close --close revenues expenses -p Q1 \ --close-acct='equity:retained earnings' >> $LEDGER_FILE - Now, the first quarter's balance sheet should show a zero (unless you + Now, the first quarter's balance sheet should show a zero (unless you are using @/@@ notation without equity postings): $ hledger bse -p Q1 And we must suppress the closing transaction to see the first quarter's - income statement (using the description; not:'retained earnings' won't + income statement (using the description; not:'retained earnings' won't work here): $ hledger is -p Q1 not:desc:'closing balances' @@ -7215,13 +7326,13 @@ PART 4: COMMANDS codes List the codes seen in transactions, in the order parsed. - This command prints the value of each transaction's code field, in the - order transactions were parsed. The transaction code is an optional - value written in parentheses between the date and description, often + This command prints the value of each transaction's code field, in the + order transactions were parsed. The transaction code is an optional + value written in parentheses between the date and description, often used to store a cheque number, order number or similar. Transactions aren't required to have a code, and missing or empty codes - will not be shown by default. With the -E/--empty flag, they will be + will not be shown by default. With the -E/--empty flag, they will be printed as blank lines. You can add a query to select a subset of transactions. @@ -7264,7 +7375,7 @@ PART 4: COMMANDS List the unique descriptions that appear in transactions. This command lists the unique descriptions that appear in transactions, - in alphabetic order. You can add a query to select a subset of trans- + in alphabetic order. You can add a query to select a subset of trans- actions. Example: @@ -7276,18 +7387,18 @@ PART 4: COMMANDS diff diff - Compares a particular account's transactions in two input files. It + Compares a particular account's transactions in two input files. It shows any transactions to this account which are in one file but not in the other. More precisely, for each posting affecting this account in either file, - it looks for a corresponding posting in the other file which posts the - same amount to the same account (ignoring date, description, etc.) + it looks for a corresponding posting in the other file which posts the + same amount to the same account (ignoring date, description, etc.) Since postings not transactions are compared, this also works when mul- tiple bank transactions have been combined into a single journal entry. This is useful eg if you have downloaded an account's transactions from - your bank (eg as CSV data). When hledger and your bank disagree about + your bank (eg as CSV data). When hledger and your bank disagree about the account balance, you can compare the bank data with your journal to find out the cause. @@ -7305,23 +7416,23 @@ PART 4: COMMANDS files files - List all files included in the journal. With a REGEX argument, only - file names matching the regular expression (case sensitive) are shown. + List all files included in the journal. With a REGEX argument, only + file names matching the regular expression (case sensitive) are shown. help help - Show the hledger user manual in the terminal, with info, man, or a - pager. With a TOPIC argument, open it at that topic if possible. - TOPIC can be any heading in the manual, or a heading prefix, case - insensitive. Eg: commands, print, forecast, journal, amount, "auto + Show the hledger user manual in the terminal, with info, man, or a + pager. With a TOPIC argument, open it at that topic if possible. + TOPIC can be any heading in the manual, or a heading prefix, case + insensitive. Eg: commands, print, forecast, journal, amount, "auto postings". This command shows the hledger manual built in to your hledger version. It can be useful when offline, or when you prefer the terminal to a web - browser, or when the appropriate hledger manual or viewing tools are + browser, or when the appropriate hledger manual or viewing tools are not installed on your system. - By default it chooses the best viewer found in $PATH (preferring info + By default it chooses the best viewer found in $PATH (preferring info since the hledger manual is large). You can select a particular viewer with the -i, -m, or -p flags. @@ -7333,71 +7444,71 @@ PART 4: COMMANDS import import - Read new transactions added to each FILE since last run, and add them - to the journal. Or with --dry-run, just print the transactions that - would be added. Or with --catchup, just mark all of the FILEs' trans- + Read new transactions added to each FILE since last run, and add them + to the journal. Or with --dry-run, just print the transactions that + would be added. Or with --catchup, just mark all of the FILEs' trans- actions as imported, without actually importing any. - This command may append new transactions to the main journal file - (which should be in journal format). Existing transactions are not - changed. This is one of the few hledger commands that writes to the + This command may append new transactions to the main journal file + (which should be in journal format). Existing transactions are not + changed. This is one of the few hledger commands that writes to the journal file (see also add). - Unlike other hledger commands, with import the journal file is an out- + Unlike other hledger commands, with import the journal file is an out- put file, and will be modified, though only by appending (existing data - will not be changed). The input files are specified as arguments, so - to import one or more CSV files to your main journal, you will run + will not be changed). The input files are specified as arguments, so + to import one or more CSV files to your main journal, you will run hledger import bank.csv or perhaps hledger import *.csv. Note you can import from any file format, though CSV files are the most common import source, and these docs focus on that case. Deduplication - As a convenience import does deduplication while reading transactions. + As a convenience import does deduplication while reading transactions. This does not mean "ignore transactions that look the same", but rather "ignore transactions that have been seen before". This is intended for - when you are periodically importing foreign data which may contain - already-imported transactions. So eg, if every day you download bank - CSV files containing redundant data, you can safely run hledger import - bank.csv and only new transactions will be imported. (import is idem- + when you are periodically importing foreign data which may contain + already-imported transactions. So eg, if every day you download bank + CSV files containing redundant data, you can safely run hledger import + bank.csv and only new transactions will be imported. (import is idem- potent.) - Since the items being read (CSV records, eg) often do not come with - unique identifiers, hledger detects new transactions by date, assuming + Since the items being read (CSV records, eg) often do not come with + unique identifiers, hledger detects new transactions by date, assuming that: 1. new items always have the newest dates 2. item dates do not change across reads - 3. and items with the same date remain in the same relative order + 3. and items with the same date remain in the same relative order across reads. - These are often true of CSV files representing transactions, or true - enough so that it works pretty well in practice. 1 is important, but + These are often true of CSV files representing transactions, or true + enough so that it works pretty well in practice. 1 is important, but violations of 2 and 3 amongst the old transactions won't matter (and if - you import often, the new transactions will be few, so less likely to + you import often, the new transactions will be few, so less likely to be the ones affected). - hledger remembers the latest date processed in each input file by sav- + hledger remembers the latest date processed in each input file by sav- ing a hidden ".latest" state file in the same directory. Eg when read- - ing finance/bank.csv, it will look for and update the finance/.lat- - est.bank.csv state file. The format is simple: one or more lines con- - taining the same ISO-format date (YYYY-MM-DD), meaning "I have pro- - cessed transactions up to this date, and this many of them on that + ing finance/bank.csv, it will look for and update the finance/.lat- + est.bank.csv state file. The format is simple: one or more lines con- + taining the same ISO-format date (YYYY-MM-DD), meaning "I have pro- + cessed transactions up to this date, and this many of them on that date." Normally you won't see or manipulate these state files yourself. - But if needed, you can delete them to reset the state (making all - transactions "new"), or you can construct them to "catch up" to a cer- + But if needed, you can delete them to reset the state (making all + transactions "new"), or you can construct them to "catch up" to a cer- tain date. - Note deduplication (and updating of state files) can also be done by + Note deduplication (and updating of state files) can also be done by print --new, but this is less often used. Import testing - With --dry-run, the transactions that will be imported are printed to + With --dry-run, the transactions that will be imported are printed to the terminal, without updating your journal or state files. The output - is valid journal format, like the print command, so you can re-parse - it. Eg, to see any importable transactions which CSV rules have not + is valid journal format, like the print command, so you can re-parse + it. Eg, to see any importable transactions which CSV rules have not categorised: $ hledger import --dry bank.csv | hledger -f- -I print unknown @@ -7407,17 +7518,17 @@ PART 4: COMMANDS $ ls bank.csv* | entr bash -c 'echo ====; hledger import --dry bank.csv | hledger -f- -I print unknown' Importing balance assignments - Entries added by import will have their posting amounts made explicit - (like hledger print -x). This means that any balance assignments in - imported files must be evaluated; but, imported files don't get to see - the main file's account balances. As a result, importing entries with + Entries added by import will have their posting amounts made explicit + (like hledger print -x). This means that any balance assignments in + imported files must be evaluated; but, imported files don't get to see + the main file's account balances. As a result, importing entries with balance assignments (eg from an institution that provides only balances - and not posting amounts) will probably generate incorrect posting + and not posting amounts) will probably generate incorrect posting amounts. To avoid this problem, use print instead of import: $ hledger print IMPORTFILE [--new] >> $LEDGER_FILE - (If you think import should leave amounts implicit like print does, + (If you think import should leave amounts implicit like print does, please test it and send a pull request.) Commodity display styles @@ -7427,12 +7538,12 @@ PART 4: COMMANDS incomestatement incomestatement, is This command displays an income statement, showing revenues and - expenses during one or more periods. Amounts are shown with normal + expenses during one or more periods. Amounts are shown with normal positive sign, as in conventional financial statements. - This report shows accounts declared with the Revenue or Expense type - (see account types). Or if no such accounts are declared, it shows - top-level accounts named revenue or income or expense (case insensi- + This report shows accounts declared with the Revenue or Expense type + (see account types). Or if no such accounts are declared, it shows + top-level accounts named revenue or income or expense (case insensi- tive, plurals allowed) and their subaccounts. Example: @@ -7459,22 +7570,22 @@ PART 4: COMMANDS 0 This command is a higher-level variant of the balance command, and sup- - ports many of that command's features, such as multi-period reports. + ports many of that command's features, such as multi-period reports. It is similar to hledger balance '(revenues|income)' expenses, but with - smarter account detection, and revenues/income displayed with their + smarter account detection, and revenues/income displayed with their sign flipped. - This command also supports the output destination and output format - options The output formats supported are txt, csv, html, and (experi- + This command also supports the output destination and output format + options The output formats supported are txt, csv, html, and (experi- mental) json. notes notes List the unique notes that appear in transactions. - This command lists the unique notes that appear in transactions, in - alphabetic order. You can add a query to select a subset of transac- - tions. The note is the part of the transaction description after a | + This command lists the unique notes that appear in transactions, in + alphabetic order. You can add a query to select a subset of transac- + tions. The note is the part of the transaction description after a | character (or if there is no |, the whole description). Example: @@ -7487,14 +7598,14 @@ PART 4: COMMANDS payees List the unique payee/payer names that appear in transactions. - This command lists unique payee/payer names which have been declared - with payee directives (--declared), used in transaction descriptions + This command lists unique payee/payer names which have been declared + with payee directives (--declared), used in transaction descriptions (--used), or both (the default). - The payee/payer is the part of the transaction description before a | + The payee/payer is the part of the transaction description before a | character (or if there is no |, the whole description). - You can add query arguments to select a subset of transactions. This + You can add query arguments to select a subset of transactions. This implies --used. Example: @@ -7506,10 +7617,10 @@ PART 4: COMMANDS prices prices - Print market price directives from the journal. With --infer-market- - prices, generate additional market prices from costs. With --infer- - reverse-prices, also generate market prices by inverting known prices. - Prices can be filtered by a query. Price amounts are displayed with + Print market price directives from the journal. With --infer-market- + prices, generate additional market prices from costs. With --infer- + reverse-prices, also generate market prices by inverting known prices. + Prices can be filtered by a query. Price amounts are displayed with their full precision. print @@ -7519,17 +7630,17 @@ PART 4: COMMANDS The print command displays full journal entries (transactions) from the journal file, sorted by date (or with --date2, by secondary date). - Amounts are shown mostly normalised to commodity display style, eg the - placement of commodity symbols will be consistent. All of their deci- + Amounts are shown mostly normalised to commodity display style, eg the + placement of commodity symbols will be consistent. All of their deci- mal places are shown, as in the original journal entry (with one alter- ation: in some cases trailing zeroes are added.) Amounts are shown right-aligned within each transaction (but not across all transactions). - Directives and inter-transaction comments are not shown, currently. + Directives and inter-transaction comments are not shown, currently. This means the print command is somewhat lossy, and if you are using it - to reformat your journal you should take care to also copy over the + to reformat your journal you should take care to also copy over the directives and file-level comments. Eg: @@ -7556,7 +7667,7 @@ PART 4: COMMANDS liabilities:debts $1 assets:bank:checking $-1 - print's output is usually a valid hledger journal, and you can process + print's output is usually a valid hledger journal, and you can process it again with a second hledger command. This can be useful for certain kinds of search, eg: @@ -7566,7 +7677,7 @@ PART 4: COMMANDS There are some situations where print's output can become unparseable: - o Valuation affects posting amounts but not balance assertion or bal- + o Valuation affects posting amounts but not balance assertion or bal- ance assignment amounts, potentially causing those to fail. o Auto postings can generate postings with too many missing amounts. @@ -7575,33 +7686,33 @@ PART 4: COMMANDS Normally, the journal entry's explicit or implicit amount style is pre- served. For example, when an amount is omitted in the journal, it will - not appear in the output. Similarly, when a cost is implied but not - written, it will not appear in the output. You can use the + not appear in the output. Similarly, when a cost is implied but not + written, it will not appear in the output. You can use the -x/--explicit flag to make all amounts and costs explicit, which can be useful for troubleshooting or for making your journal more readable and - robust against data entry errors. -x is also implied by using any of + robust against data entry errors. -x is also implied by using any of -B,-V,-X,--value. - Note, -x/--explicit will cause postings with a multi-commodity amount - (these can arise when a multi-commodity transaction has an implicit - amount) to be split into multiple single-commodity postings, keeping + Note, -x/--explicit will cause postings with a multi-commodity amount + (these can arise when a multi-commodity transaction has an implicit + amount) to be split into multiple single-commodity postings, keeping the output parseable. - With -B/--cost, amounts with costs are converted to cost using that + With -B/--cost, amounts with costs are converted to cost using that price. This can be used for troubleshooting. - With -m DESC/--match=DESC, print does a fuzzy search for the one trans- - action whose description is most similar to DESC, also preferring - recent tranactions. DESC should contain at least two characters. If - there is no similar-enough match, no transaction will be shown and the - program exit code will be non-zero. + With -m DESC/--match=DESC, print does a fuzzy search for one recent + transaction whose description is most similar to DESC. DESC should + contain at least two characters. If there is no similar-enough match, + no transaction will be shown and the program exit code will be non- + zero. - With --new, hledger prints only transactions it has not seen on a pre- - vious run. This uses the same deduplication system as the import com- + With --new, hledger prints only transactions it has not seen on a pre- + vious run. This uses the same deduplication system as the import com- mand. (See import's docs for details.) - This command also supports the output destination and output format - options The output formats supported are txt, csv, and (experimental) + This command also supports the output destination and output format + options The output formats supported are txt, csv, and (experimental) json and sql. Here's an example of print's CSV output: @@ -7620,51 +7731,35 @@ PART 4: COMMANDS "5","2008/12/31","","*","","pay off","","liabilities:debts","1","$","","1","","" "5","2008/12/31","","*","","pay off","","assets:bank:checking","-1","$","1","","","" - o There is one CSV record per posting, with the parent transaction's + o There is one CSV record per posting, with the parent transaction's fields repeated. o The "txnidx" (transaction index) field shows which postings belong to - the same transaction. (This number might change if transactions are - reordered within the file, files are parsed/included in a different + the same transaction. (This number might change if transactions are + reordered within the file, files are parsed/included in a different order, etc.) - o The amount is separated into "commodity" (the symbol) and "amount" + o The amount is separated into "commodity" (the symbol) and "amount" (numeric quantity) fields. o The numeric amount is repeated in either the "credit" or "debit" col- - umn, for convenience. (Those names are not accurate in the account- - ing sense; it just puts negative amounts under credit and zero or + umn, for convenience. (Those names are not accurate in the account- + ing sense; it just puts negative amounts under credit and zero or greater amounts under debit.) - print-unique - print-unique - Print transactions which do not reuse an already-seen description. - - Example: - - $ cat unique.journal - 1/1 test - (acct:one) 1 - 2/2 test - (acct:two) 2 - $ LEDGER_FILE=unique.journal hledger print-unique - (-f option not supported) - 2015/01/01 test - (acct:one) 1 - register register, reg Show postings and their running total. The register command displays matched postings, across all accounts, in - date order, with their running total or running historical balance. - (See also the aregister command, which shows matched transactions in a + date order, with their running total or running historical balance. + (See also the aregister command, which shows matched transactions in a specific account.) register normally shows line per posting, but note that multi-commodity amounts will occupy multiple lines (one line per commodity). - It is typically used with a query selecting a particular account, to + It is typically used with a query selecting a particular account, to see that account's activity: $ hledger register checking @@ -7675,14 +7770,14 @@ PART 4: COMMANDS With --date2, it shows and sorts by secondary date instead. - For performance reasons, column widths are chosen based on the first - 1000 lines; this means unusually wide values in later lines can cause - visual discontinuities as column widths are adjusted. If you want to - ensure perfect alignment, at the cost of more time and memory, use the + For performance reasons, column widths are chosen based on the first + 1000 lines; this means unusually wide values in later lines can cause + visual discontinuities as column widths are adjusted. If you want to + ensure perfect alignment, at the cost of more time and memory, use the --align-all flag. - The --historical/-H flag adds the balance from any undisplayed prior - postings to the running total. This is useful when you want to see + The --historical/-H flag adds the balance from any undisplayed prior + postings to the running total. This is useful when you want to see only recent activity, with a historically accurate running balance: $ hledger register checking -b 2008/6 --historical @@ -7692,30 +7787,30 @@ PART 4: COMMANDS The --depth option limits the amount of sub-account detail displayed. - The --average/-A flag shows the running average posting amount instead + The --average/-A flag shows the running average posting amount instead of the running total (so, the final number displayed is the average for - the whole report period). This flag implies --empty (see below). It - is affected by --historical. It works best when showing just one + the whole report period). This flag implies --empty (see below). It + is affected by --historical. It works best when showing just one account and one commodity. - The --related/-r flag shows the other postings in the transactions of + The --related/-r flag shows the other postings in the transactions of the postings which would normally be shown. - The --invert flag negates all amounts. For example, it can be used on + The --invert flag negates all amounts. For example, it can be used on an income account where amounts are normally displayed as negative num- - bers. It's also useful to show postings on the checking account + bers. It's also useful to show postings on the checking account together with the related account: $ hledger register --related --invert assets:checking - With a reporting interval, register shows summary postings, one per + With a reporting interval, register shows summary postings, one per interval, aggregating the postings to each account: $ hledger register --monthly income 2008/01 income:salary $-1 $-1 2008/06 income:gifts $-1 $-2 - Periods with no activity, and summary postings with a zero amount, are + Periods with no activity, and summary postings with a zero amount, are not shown by default; use the --empty/-E flag to see them: $ hledger register --monthly income -E @@ -7732,7 +7827,7 @@ PART 4: COMMANDS 2008/11 0 $-2 2008/12 0 $-2 - Often, you'll want to see just one line per interval. The --depth + Often, you'll want to see just one line per interval. The --depth option helps with this, causing subaccounts to be aggregated: $ hledger register --monthly assets --depth 1h @@ -7740,11 +7835,16 @@ PART 4: COMMANDS 2008/06 assets $-1 0 2008/12 assets $-1 $-1 - Note when using report intervals, if you specify start/end dates these - will be adjusted outward if necessary to contain a whole number of - intervals. This ensures that the first and last intervals are full + Note when using report intervals, if you specify start/end dates these + will be adjusted outward if necessary to contain a whole number of + intervals. This ensures that the first and last intervals are full length and comparable to the others in the report. + With -m DESC/--match=DESC, register does a fuzzy search for one recent + posting whose description is most similar to DESC. DESC should contain + at least two characters. If there is no similar-enough match, no post- + ing will be shown and the program exit code will be non-zero. + Custom register output register uses the full terminal width by default, except on windows. You can override this by setting the COLUMNS environment variable (not @@ -7772,14 +7872,6 @@ PART 4: COMMANDS options The output formats supported are txt, csv, and (experimental) json. - register-match - register-match - Print the one posting whose transaction description is closest to DESC, - in the style of the register command. If there are multiple equally - good matches, it shows the most recent. Query options (options, not - arguments) can be used to restrict the search space. Helps ledger- - autosync detect already-seen transactions when importing. - rewrite rewrite Print all transactions, rewriting the postings of matched transactions.