slaesvuo/hledger
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

;doc: csv: more cleanup, reorder

Browse Source
This commit is contained in:
Simon Michael 2023-01-11 13:53:47 -10:00
parent 43466dbe03
commit 214cb35e02
1 changed files with 302 additions and 307 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

609
hledger/hledger.m4.md
View File

@ -2764,6 +2764,8 @@ value EXPR
See also <https://hledger.org/ledger.html> for a detailed hledger/Ledger syntax comparison. See also <https://hledger.org/ledger.html> for a detailed hledger/Ledger syntax comparison.
<a name="csv-format"></a>
# CSV # CSV
hledger can read [CSV](http://en.wikipedia.org/wiki/Comma-separated_values) files hledger can read [CSV](http://en.wikipedia.org/wiki/Comma-separated_values) files
@ -2772,7 +2774,7 @@ automatically converting each record into a transaction.
(To learn about *writing* CSV, see [CSV output](#csv-output).) (To learn about *writing* CSV, see [CSV output](#csv-output).)
Note, for best error messages when reading CSV/TSV/SSV files, For best error messages when reading CSV/TSV/SSV files,
make sure they have a corresponding `.csv`, `.tsv` or `.ssv` file extension make sure they have a corresponding `.csv`, `.tsv` or `.ssv` file extension
or use a hledger file prefix (see [File Extension](#file-extension) below). or use a hledger file prefix (see [File Extension](#file-extension) below).
@ -2789,15 +2791,6 @@ You can specify a different rules file with the `--rules-file` option.
If no rules file is found, hledger will create a sample rules file, If no rules file is found, hledger will create a sample rules file,
which you'll need to adjust. which you'll need to adjust.
There's an introductory [Importing CSV data](/import-csv.html) tutorial on hledger.org.
## Examples
Here are some sample hledger CSV rules files. See also the full collection at:\
<https://github.com/simonmichael/hledger/tree/master/examples/csv>
### Basic
At minimum, the rules file must identify the date and amount fields, At minimum, the rules file must identify the date and amount fields,
and often it also specifies the date format and how many header lines and often it also specifies the date format and how many header lines
there are. Here's a simple CSV file and a rules file for it: there are. Here's a simple CSV file and a rules file for it:
@ -2808,7 +2801,7 @@ Date, Description, Id, Amount
```rules ```rules
# basic.csv.rules # basic.csv.rules
skip 1 skip 1
fields date, description, _, amount fields date, description, , amount
date-format %d/%m/%Y date-format %d/%m/%Y
``` ```
```shell ```shell
@ -2818,285 +2811,12 @@ $ hledger print -f basic.csv
income:unknown -10.23 income:unknown -10.23
``` ```
Default account names are chosen, since we didn't set them.
### Bank of Ireland There's an introductory [Importing CSV data](/import-csv.html) tutorial on hledger.org,
and more [CSV rules examples](#csv-rules-examples) below,
and a larger collection at <https://github.com/simonmichael/hledger/tree/master/examples/csv>.
Here's a CSV with two amount fields (Debit and Credit), and a balance field, ## CSV rules cheatsheet
which we can use to add balance assertions, which is not necessary but
provides extra error checking:
```csv
Date,Details,Debit,Credit,Balance
07/12/2012,LODGMENT 529898,,10.0,131.21
07/12/2012,PAYMENT,5,,126
```
```rules
# bankofireland-checking.csv.rules
# skip the header line
skip
# name the csv fields, and assign some of them as journal entry fields
fields date, description, amount-out, amount-in, balance
# We generate balance assertions by assigning to "balance"
# above, but you may sometimes need to remove these because:
#
# - the CSV balance differs from the true balance,
# by up to 0.0000000000005 in my experience
#
# - it is sometimes calculated based on non-chronological ordering,
# eg when multiple transactions clear on the same day
# date is in UK/Ireland format
date-format %d/%m/%Y
# set the currency
currency EUR
# set the base account for all txns
account1 assets:bank:boi:checking
```
```shell
$ hledger -f bankofireland-checking.csv print
2012-12-07 LODGMENT 529898
assets:bank:boi:checking EUR10.0 = EUR131.2
income:unknown EUR-10.0
2012-12-07 PAYMENT
assets:bank:boi:checking EUR-5.0 = EUR126.0
expenses:unknown EUR5.0
```
The balance assertions don't raise an error above, because we're
reading directly from CSV, but they will be checked if these entries
are imported into a journal file.
### Amazon
Here we convert amazon.com order history, and use an if block to
generate a third posting if there's a fee.
(In practice you'd probably get this data from your bank instead,
but it's an example.)
```csv
"Date","Type","To/From","Name","Status","Amount","Fees","Transaction ID"
"Jul 29, 2012","Payment","To","Foo.","Completed","$20.00","$0.00","16000000000000DGLNJPI1P9B8DKPVHL"
"Jul 30, 2012","Payment","To","Adapteva, Inc.","Completed","$25.00","$1.00","17LA58JSKRD4HDGLNJPI1P9B8DKPVHL"
```
```rules
# amazon-orders.csv.rules
# skip one header line
skip 1
# name the csv fields, and assign the transaction's date, amount and code.
# Avoided the "status" and "amount" hledger field names to prevent confusion.
fields date, _, toorfrom, name, amzstatus, amzamount, fees, code
# how to parse the date
date-format %b %-d, %Y
# combine two fields to make the description
description %toorfrom %name
# save the status as a tag
comment status:%amzstatus
# set the base account for all transactions
account1 assets:amazon
# leave amount1 blank so it can balance the other(s).
# I'm assuming amzamount excludes the fees, don't remember
# set a generic account2
account2 expenses:misc
amount2 %amzamount
# and maybe refine it further:
#include categorisation.rules
# add a third posting for fees, but only if they are non-zero.
if %fees [1-9]
account3 expenses:fees
amount3 %fees
```
```shell
$ hledger -f amazon-orders.csv print
2012-07-29 (16000000000000DGLNJPI1P9B8DKPVHL) To Foo. ; status:Completed
assets:amazon
expenses:misc $20.00
2012-07-30 (17LA58JSKRD4HDGLNJPI1P9B8DKPVHL) To Adapteva, Inc. ; status:Completed
assets:amazon
expenses:misc $25.00
expenses:fees $1.00
```
### Paypal
Here's a real-world rules file for (customised) Paypal CSV,
with some Paypal-specific rules, and a second rules file included:
```csv
"Date","Time","TimeZone","Name","Type","Status","Currency","Gross","Fee","Net","From Email Address","To Email Address","Transaction ID","Item Title","Item ID","Reference Txn ID","Receipt ID","Balance","Note"
"10/01/2019","03:46:20","PDT","Calm Radio","Subscription Payment","Completed","USD","-6.99","0.00","-6.99","simon@joyful.com","memberships@calmradio.com","60P57143A8206782E","MONTHLY - $1 for the first 2 Months: Me - Order 99309. Item total: $1.00 USD first 2 months, then $6.99 / Month","","I-R8YLY094FJYR","","-6.99",""
"10/01/2019","03:46:20","PDT","","Bank Deposit to PP Account ","Pending","USD","6.99","0.00","6.99","","simon@joyful.com","0TU1544T080463733","","","60P57143A8206782E","","0.00",""
"10/01/2019","08:57:01","PDT","Patreon","PreApproved Payment Bill User Payment","Completed","USD","-7.00","0.00","-7.00","simon@joyful.com","support@patreon.com","2722394R5F586712G","Patreon* Membership","","B-0PG93074E7M86381M","","-7.00",""
"10/01/2019","08:57:01","PDT","","Bank Deposit to PP Account ","Pending","USD","7.00","0.00","7.00","","simon@joyful.com","71854087RG994194F","Patreon* Membership","","2722394R5F586712G","","0.00",""
"10/19/2019","03:02:12","PDT","Wikimedia Foundation, Inc.","Subscription Payment","Completed","USD","-2.00","0.00","-2.00","simon@joyful.com","tle@wikimedia.org","K9U43044RY432050M","Monthly donation to the Wikimedia Foundation","","I-R5C3YUS3285L","","-2.00",""
"10/19/2019","03:02:12","PDT","","Bank Deposit to PP Account ","Pending","USD","2.00","0.00","2.00","","simon@joyful.com","3XJ107139A851061F","","","K9U43044RY432050M","","0.00",""
"10/22/2019","05:07:06","PDT","Noble Benefactor","Subscription Payment","Completed","USD","10.00","-0.59","9.41","noble@bene.fac.tor","simon@joyful.com","6L8L1662YP1334033","Joyful Systems","","I-KC9VBGY2GWDB","","9.41",""
```
```rules
# paypal-custom.csv.rules
# Tips:
# Export from Activity -> Statements -> Custom -> Activity download
# Suggested transaction type: "Balance affecting"
# Paypal's default fields in 2018 were:
# "Date","Time","TimeZone","Name","Type","Status","Currency","Gross","Fee","Net","From Email Address","To Email Address","Transaction ID","Shipping Address","Address Status","Item Title","Item ID","Shipping and Handling Amount","Insurance Amount","Sales Tax","Option 1 Name","Option 1 Value","Option 2 Name","Option 2 Value","Reference Txn ID","Invoice Number","Custom Number","Quantity","Receipt ID","Balance","Address Line 1","Address Line 2/District/Neighborhood","Town/City","State/Province/Region/County/Territory/Prefecture/Republic","Zip/Postal Code","Country","Contact Phone Number","Subject","Note","Country Code","Balance Impact"
# This rules file assumes the following more detailed fields, configured in "Customize report fields":
# "Date","Time","TimeZone","Name","Type","Status","Currency","Gross","Fee","Net","From Email Address","To Email Address","Transaction ID","Item Title","Item ID","Reference Txn ID","Receipt ID","Balance","Note"
fields date, time, timezone, description_, type, status_, currency, grossamount, feeamount, netamount, fromemail, toemail, code, itemtitle, itemid, referencetxnid, receiptid, balance, note
skip 1
date-format %-m/%-d/%Y
# ignore some paypal events
if
In Progress
Temporary Hold
Update to
skip
# add more fields to the description
description %description_ %itemtitle
# save some other fields as tags
comment itemid:%itemid, fromemail:%fromemail, toemail:%toemail, time:%time, type:%type, status:%status_
# convert to short currency symbols
if %currency USD
currency $
if %currency EUR
currency E
if %currency GBP
currency P
# generate postings
# the first posting will be the money leaving/entering my paypal account
# (negative means leaving my account, in all amount fields)
account1 assets:online:paypal
amount1 %netamount
# the second posting will be money sent to/received from other party
# (account2 is set below)
amount2 -%grossamount
# if there's a fee, add a third posting for the money taken by paypal.
if %feeamount [1-9]
account3 expenses:banking:paypal
amount3 -%feeamount
comment3 business:
# choose an account for the second posting
# override the default account names:
# if the amount is positive, it's income (a debit)
if %grossamount ^[^-]
account2 income:unknown
# if negative, it's an expense (a credit)
if %grossamount ^-
account2 expenses:unknown
# apply common rules for setting account2 & other tweaks
include common.rules
# apply some overrides specific to this csv
# Transfers from/to bank. These are usually marked Pending,
# which can be disregarded in this case.
if
Bank Account
Bank Deposit to PP Account
description %type for %referencetxnid %itemtitle
account2 assets:bank:wf:pchecking
account1 assets:online:paypal
# Currency conversions
if Currency Conversion
account2 equity:currency conversion
```
```rules
# common.rules
if
darcs
noble benefactor
account2 revenues:foss donations:darcshub
comment2 business:
if
Calm Radio
account2 expenses:online:apps
if
electronic frontier foundation
Patreon
wikimedia
Advent of Code
account2 expenses:dues
if Google
account2 expenses:online:apps
description google | music
```
```shell
$ hledger -f paypal-custom.csv print
2019-10-01 (60P57143A8206782E) Calm Radio MONTHLY - $1 for the first 2 Months: Me - Order 99309. Item total: $1.00 USD first 2 months, then $6.99 / Month ; itemid:, fromemail:simon@joyful.com, toemail:memberships@calmradio.com, time:03:46:20, type:Subscription Payment, status:Completed
assets:online:paypal $-6.99 = $-6.99
expenses:online:apps $6.99
2019-10-01 (0TU1544T080463733) Bank Deposit to PP Account for 60P57143A8206782E ; itemid:, fromemail:, toemail:simon@joyful.com, time:03:46:20, type:Bank Deposit to PP Account, status:Pending
assets:online:paypal $6.99 = $0.00
assets:bank:wf:pchecking $-6.99
2019-10-01 (2722394R5F586712G) Patreon Patreon* Membership ; itemid:, fromemail:simon@joyful.com, toemail:support@patreon.com, time:08:57:01, type:PreApproved Payment Bill User Payment, status:Completed
assets:online:paypal $-7.00 = $-7.00
expenses:dues $7.00
2019-10-01 (71854087RG994194F) Bank Deposit to PP Account for 2722394R5F586712G Patreon* Membership ; itemid:, fromemail:, toemail:simon@joyful.com, time:08:57:01, type:Bank Deposit to PP Account, status:Pending
assets:online:paypal $7.00 = $0.00
assets:bank:wf:pchecking $-7.00
2019-10-19 (K9U43044RY432050M) Wikimedia Foundation, Inc. Monthly donation to the Wikimedia Foundation ; itemid:, fromemail:simon@joyful.com, toemail:tle@wikimedia.org, time:03:02:12, type:Subscription Payment, status:Completed
assets:online:paypal $-2.00 = $-2.00
expenses:dues $2.00
expenses:banking:paypal ; business:
2019-10-19 (3XJ107139A851061F) Bank Deposit to PP Account for K9U43044RY432050M ; itemid:, fromemail:, toemail:simon@joyful.com, time:03:02:12, type:Bank Deposit to PP Account, status:Pending
assets:online:paypal $2.00 = $0.00
assets:bank:wf:pchecking $-2.00
2019-10-22 (6L8L1662YP1334033) Noble Benefactor Joyful Systems ; itemid:, fromemail:noble@bene.fac.tor, toemail:simon@joyful.com, time:05:07:06, type:Subscription Payment, status:Completed
assets:online:paypal $9.41 = $9.41
revenues:foss donations:darcshub $-10.00 ; business:
expenses:banking:paypal $0.59 ; business:
```
<a name="csv-format"></a>
## CSV rules
The following kinds of rule can appear in the rules file, in any order. The following kinds of rule can appear in the rules file, in any order.
Blank lines and lines beginning with `#` or `;` or `*` are ignored. Blank lines and lines beginning with `#` or `;` or `*` are ignored.
@ -3111,7 +2831,7 @@ Blank lines and lines beginning with `#` or `;` or `*` are ignored.
| [**`newest-first`**](#newest-first) | improve txn order when: there are multiple records, newest first, all with the same date | | [**`newest-first`**](#newest-first) | improve txn order when: there are multiple records, newest first, all with the same date |
| [**`intra-day-reversed`**](#intra-day-reversed) | improve txn order when: same-day txns are in opposite order to the overall file | | [**`intra-day-reversed`**](#intra-day-reversed) | improve txn order when: same-day txns are in opposite order to the overall file |
| [**`balance-type`**](#balance-type) | select which type of balance assignments to use | | [**`balance-type`**](#balance-type) | select which type of balance assignments to use |
| [**`fields`**](#fields) | name CSV fields for easy reference, and optionally assign their values to hledger fields | | [**`fields`**](#fields-list) | name CSV fields for easy reference, and optionally assign their values to hledger fields |
| [**Field assignment**](#field-assignment) | assign a CSV value or interpolated text value to a hledger field, constructing the txn | | [**Field assignment**](#field-assignment) | assign a CSV value or interpolated text value to a hledger field, constructing the txn |
| [**`if` block**](#if-block) | conditionally assign values to hledger fields, or `skip` a record or `end` (skip rest of file) | | [**`if` block**](#if-block) | conditionally assign values to hledger fields, or `skip` a record or `end` (skip rest of file) |
| [**`if` table**](#if-table) | conditionally assign values to hledger fields, using compact syntax | | [**`if` table**](#if-table) | conditionally assign values to hledger fields, using compact syntax |
@ -3270,12 +2990,12 @@ intra-day-reversed
## `fields` ## `fields` list
```rules ```rules
fields FIELDNAME1, FIELDNAME2, ... fields FIELDNAME1, FIELDNAME2, ...
``` ```
A fields list (the word "fields" followed by comma-separated field names) is optional, but convenient. A fields list (the word `fields` followed by comma-separated field names) is optional, but convenient.
It does two things: It does two things:
1. It names the CSV field in each column. 1. It names the CSV field in each column.
@ -3299,7 +3019,7 @@ Also:
- There must be least two items in the list (at least one comma). - There must be least two items in the list (at least one comma).
- Field names may not contain spaces. Spaces before/after field names are optional. - Field names may not contain spaces. Spaces before/after field names are optional.
- Field names may contain `_` (underscore) or `-` (hyphen). - Field names may contain `_` (underscore) or `-` (hyphen).
- Fields you don't care about can be given a dummy name (eg just `_`) or an empty name. - Fields you don't care about can be given a dummy name or an empty name.
If the CSV contains column headings, it's convenient to use these for your field names, If the CSV contains column headings, it's convenient to use these for your field names,
suitably modified (eg lower-cased with spaces replaced by underscores). suitably modified (eg lower-cased with spaces replaced by underscores).
@ -3314,7 +3034,7 @@ HLEDGERFIELD FIELDVALUE
``` ```
Field assignments are the more flexible way to assign CSV values to hledger fields. Field assignments are the more flexible way to assign CSV values to hledger fields.
They can be used instead of or in addition to a [fields](#fields) list (see above). They can be used instead of or in addition to a [fields list](#fields-list) (see above).
To assign a value to a hledger field, write the [field name](#field-names) To assign a value to a hledger field, write the [field name](#field-names)
(any of the standard hledger field/pseudo-field names, defined below), (any of the standard hledger field/pseudo-field names, defined below),
@ -3639,9 +3359,9 @@ Here are the balance assertion types for quick reference:
==* multi commodity, include subaccounts ==* multi commodity, include subaccounts
``` ```
## Tips ## CSV rules tips
## Rapid feedback ### Rapid feedback
It's a good idea to get rapid feedback while creating/troubleshooting CSV rules. It's a good idea to get rapid feedback while creating/troubleshooting CSV rules.
Here's a good way, using entr from [eradman.com/entrproject](https://eradman.com/entrproject): Here's a good way, using entr from [eradman.com/entrproject](https://eradman.com/entrproject):
@ -3652,7 +3372,7 @@ A desc: query (eg) is used to select just one, or a few, transactions of interes
"bash -c" is used to run multiple commands, so we can echo a separator each time "bash -c" is used to run multiple commands, so we can echo a separator each time
the command re-runs, making it easier to read the output. the command re-runs, making it easier to read the output.
## Valid CSV ### Valid CSV
Note that hledger will only accept valid CSV conforming to [RFC 4180](https://tools.ietf.org/html/rfc4180), Note that hledger will only accept valid CSV conforming to [RFC 4180](https://tools.ietf.org/html/rfc4180),
and equivalent SSV and TSV formats (like RFC 4180 but with semicolon or tab as separators). and equivalent SSV and TSV formats (like RFC 4180 but with semicolon or tab as separators).
@ -3666,7 +3386,7 @@ This means, eg:
If your CSV/SSV/TSV is not valid in this sense, you'll need to transform it before reading with hledger. If your CSV/SSV/TSV is not valid in this sense, you'll need to transform it before reading with hledger.
Try using sed, or a more permissive CSV parser like [python's csv lib](https://docs.python.org/3/library/csv.html). Try using sed, or a more permissive CSV parser like [python's csv lib](https://docs.python.org/3/library/csv.html).
## File Extension ### File Extension
To help hledger identify the format and show the right error messages, To help hledger identify the format and show the right error messages,
CSV/SSV/TSV files should normally be named with a `.csv`, `.ssv` or `.tsv` CSV/SSV/TSV files should normally be named with a `.csv`, `.ssv` or `.tsv`
@ -3683,14 +3403,14 @@ $ cat foo | hledger -f ssv:- foo
You can override the file extension with a [separator](#separator) rule if needed. You can override the file extension with a [separator](#separator) rule if needed.
See also: [Input files](#input-files) in the hledger manual. See also: [Input files](#input-files) in the hledger manual.
## Reading multiple CSV files ### Reading multiple CSV files
If you use multiple `-f` options to read multiple CSV files at once, If you use multiple `-f` options to read multiple CSV files at once,
hledger will look for a correspondingly-named rules file for each CSV hledger will look for a correspondingly-named rules file for each CSV
file. But if you use the `--rules-file` option, that rules file will file. But if you use the `--rules-file` option, that rules file will
be used for all the CSV files. be used for all the CSV files.
## Valid transactions ### Valid transactions
After reading a CSV file, hledger post-processes and validates the After reading a CSV file, hledger post-processes and validates the
generated journal entries as it would for a journal file - balancing generated journal entries as it would for a journal file - balancing
@ -3706,7 +3426,7 @@ balance assertions generated from CSV right away, pipe into another hledger:
$ hledger -f file.csv print | hledger -f- print $ hledger -f file.csv print | hledger -f- print
``` ```
## Deduplicating, importing ### Deduplicating, importing
When you download a CSV file periodically, eg to get your latest bank When you download a CSV file periodically, eg to get your latest bank
transactions, the new file may overlap with the old one, containing transactions, the new file may overlap with the old one, containing
@ -3733,14 +3453,14 @@ data. See:
- <https://hledger.org/cookbook.html#setups-and-workflows> - <https://hledger.org/cookbook.html#setups-and-workflows>
- <https://plaintextaccounting.org> -> data import/conversion - <https://plaintextaccounting.org> -> data import/conversion
## Setting amounts ### Setting amounts
Some tips on using the [amount-setting rules](#amount) discussed above. Some tips on using the [amount-setting rules](#amount) discussed above.
Here are the ways to set a posting's amount: Here are the ways to set a posting's amount:
1. **If the CSV has a single amount field:**\ 1. **If the CSV has a single amount field:**\
Assign (via a [fields list](#fields) or a [field assignment](#field-assignment)) to `amountN`. Assign (via a [fields list](#fields-list) or a [field assignment](#field-assignment)) to `amountN`.
This sets the Nth posting's amount. N is usually 1 or 2 but can go up to 99. This sets the Nth posting's amount. N is usually 1 or 2 but can go up to 99.
2. **If the CSV has separate amount fields for debit & credit (in & out):**\ 2. **If the CSV has separate amount fields for debit & credit (in & out):**\
@ -3801,7 +3521,7 @@ Here are the ways to set a posting's amount:
account1 assets:checking account1 assets:checking
``` ```
## Amount signs ### Amount signs
There is some special handling for amount signs, to simplify parsing and sign-flipping: There is some special handling for amount signs, to simplify parsing and sign-flipping:
@ -3817,7 +3537,7 @@ There is some special handling for amount signs, to simplify parsing and sign-fl
- **If an amount value contains just a sign (or just a set of parentheses):**\ - **If an amount value contains just a sign (or just a set of parentheses):**\
that is removed, making it an empty value. `"+"` or `"-"` or `"()"` becomes `""`. that is removed, making it an empty value. `"+"` or `"-"` or `"()"` becomes `""`.
## Setting currency/commodity ### Setting currency/commodity
If the currency/commodity symbol is included in the CSV's amount field(s): If the currency/commodity symbol is included in the CSV's amount field(s):
@ -3874,7 +3594,7 @@ amount %amt %cur
Note we used a temporary field name (`cur`) that is not `currency` - Note we used a temporary field name (`cur`) that is not `currency` -
that would trigger the prepending effect, which we don't want here. that would trigger the prepending effect, which we don't want here.
## Amount decimal places ### Amount decimal places
Like amounts in a journal file, Like amounts in a journal file,
the amounts generated by CSV rules like `amount1` influence the amounts generated by CSV rules like `amount1` influence
@ -3884,7 +3604,7 @@ the number of decimal places displayed in reports.
The original amounts as written in the CSV file do not affect display The original amounts as written in the CSV file do not affect display
style (because we don't yet reliably know their commodity). style (because we don't yet reliably know their commodity).
## Referencing other fields ### Referencing other fields
In field assignments, you can interpolate only CSV fields, not hledger In field assignments, you can interpolate only CSV fields, not hledger
fields. In the example below, there's both a CSV field and a hledger fields. In the example below, there's both a CSV field and a hledger
@ -3920,7 +3640,7 @@ if something
comment C comment C
``` ```
## How CSV rules are evaluated ### How CSV rules are evaluated
Here's how to think of CSV rules being evaluated (if you really need to). Here's how to think of CSV rules being evaluated (if you really need to).
First, First,
@ -3956,6 +3676,281 @@ user specified.
<a name="timeclock-format"></a> <a name="timeclock-format"></a>
## CSV rules examples
### Bank of Ireland
Here's a CSV with two amount fields (Debit and Credit), and a balance field,
which we can use to add balance assertions, which is not necessary but
provides extra error checking:
```csv
Date,Details,Debit,Credit,Balance
07/12/2012,LODGMENT 529898,,10.0,131.21
07/12/2012,PAYMENT,5,,126
```
```rules
# bankofireland-checking.csv.rules
# skip the header line
skip
# name the csv fields, and assign some of them as journal entry fields
fields date, description, amount-out, amount-in, balance
# We generate balance assertions by assigning to "balance"
# above, but you may sometimes need to remove these because:
#
# - the CSV balance differs from the true balance,
# by up to 0.0000000000005 in my experience
#
# - it is sometimes calculated based on non-chronological ordering,
# eg when multiple transactions clear on the same day
# date is in UK/Ireland format
date-format %d/%m/%Y
# set the currency
currency EUR
# set the base account for all txns
account1 assets:bank:boi:checking
```
```shell
$ hledger -f bankofireland-checking.csv print
2012-12-07 LODGMENT 529898
assets:bank:boi:checking EUR10.0 = EUR131.2
income:unknown EUR-10.0
2012-12-07 PAYMENT
assets:bank:boi:checking EUR-5.0 = EUR126.0
expenses:unknown EUR5.0
```
The balance assertions don't raise an error above, because we're
reading directly from CSV, but they will be checked if these entries
are imported into a journal file.
### Amazon
Here we convert amazon.com order history, and use an if block to
generate a third posting if there's a fee.
(In practice you'd probably get this data from your bank instead,
but it's an example.)
```csv
"Date","Type","To/From","Name","Status","Amount","Fees","Transaction ID"
"Jul 29, 2012","Payment","To","Foo.","Completed","$20.00","$0.00","16000000000000DGLNJPI1P9B8DKPVHL"
"Jul 30, 2012","Payment","To","Adapteva, Inc.","Completed","$25.00","$1.00","17LA58JSKRD4HDGLNJPI1P9B8DKPVHL"
```
```rules
# amazon-orders.csv.rules
# skip one header line
skip 1
# name the csv fields, and assign the transaction's date, amount and code.
# Avoided the "status" and "amount" hledger field names to prevent confusion.
fields date, _, toorfrom, name, amzstatus, amzamount, fees, code
# how to parse the date
date-format %b %-d, %Y
# combine two fields to make the description
description %toorfrom %name
# save the status as a tag
comment status:%amzstatus
# set the base account for all transactions
account1 assets:amazon
# leave amount1 blank so it can balance the other(s).
# I'm assuming amzamount excludes the fees, don't remember
# set a generic account2
account2 expenses:misc
amount2 %amzamount
# and maybe refine it further:
#include categorisation.rules
# add a third posting for fees, but only if they are non-zero.
if %fees [1-9]
account3 expenses:fees
amount3 %fees
```
```shell
$ hledger -f amazon-orders.csv print
2012-07-29 (16000000000000DGLNJPI1P9B8DKPVHL) To Foo. ; status:Completed
assets:amazon
expenses:misc $20.00
2012-07-30 (17LA58JSKRD4HDGLNJPI1P9B8DKPVHL) To Adapteva, Inc. ; status:Completed
assets:amazon
expenses:misc $25.00
expenses:fees $1.00
```
### Paypal
Here's a real-world rules file for (customised) Paypal CSV,
with some Paypal-specific rules, and a second rules file included:
```csv
"Date","Time","TimeZone","Name","Type","Status","Currency","Gross","Fee","Net","From Email Address","To Email Address","Transaction ID","Item Title","Item ID","Reference Txn ID","Receipt ID","Balance","Note"
"10/01/2019","03:46:20","PDT","Calm Radio","Subscription Payment","Completed","USD","-6.99","0.00","-6.99","simon@joyful.com","memberships@calmradio.com","60P57143A8206782E","MONTHLY - $1 for the first 2 Months: Me - Order 99309. Item total: $1.00 USD first 2 months, then $6.99 / Month","","I-R8YLY094FJYR","","-6.99",""
"10/01/2019","03:46:20","PDT","","Bank Deposit to PP Account ","Pending","USD","6.99","0.00","6.99","","simon@joyful.com","0TU1544T080463733","","","60P57143A8206782E","","0.00",""
"10/01/2019","08:57:01","PDT","Patreon","PreApproved Payment Bill User Payment","Completed","USD","-7.00","0.00","-7.00","simon@joyful.com","support@patreon.com","2722394R5F586712G","Patreon* Membership","","B-0PG93074E7M86381M","","-7.00",""
"10/01/2019","08:57:01","PDT","","Bank Deposit to PP Account ","Pending","USD","7.00","0.00","7.00","","simon@joyful.com","71854087RG994194F","Patreon* Membership","","2722394R5F586712G","","0.00",""
"10/19/2019","03:02:12","PDT","Wikimedia Foundation, Inc.","Subscription Payment","Completed","USD","-2.00","0.00","-2.00","simon@joyful.com","tle@wikimedia.org","K9U43044RY432050M","Monthly donation to the Wikimedia Foundation","","I-R5C3YUS3285L","","-2.00",""
"10/19/2019","03:02:12","PDT","","Bank Deposit to PP Account ","Pending","USD","2.00","0.00","2.00","","simon@joyful.com","3XJ107139A851061F","","","K9U43044RY432050M","","0.00",""
"10/22/2019","05:07:06","PDT","Noble Benefactor","Subscription Payment","Completed","USD","10.00","-0.59","9.41","noble@bene.fac.tor","simon@joyful.com","6L8L1662YP1334033","Joyful Systems","","I-KC9VBGY2GWDB","","9.41",""
```
```rules
# paypal-custom.csv.rules
# Tips:
# Export from Activity -> Statements -> Custom -> Activity download
# Suggested transaction type: "Balance affecting"
# Paypal's default fields in 2018 were:
# "Date","Time","TimeZone","Name","Type","Status","Currency","Gross","Fee","Net","From Email Address","To Email Address","Transaction ID","Shipping Address","Address Status","Item Title","Item ID","Shipping and Handling Amount","Insurance Amount","Sales Tax","Option 1 Name","Option 1 Value","Option 2 Name","Option 2 Value","Reference Txn ID","Invoice Number","Custom Number","Quantity","Receipt ID","Balance","Address Line 1","Address Line 2/District/Neighborhood","Town/City","State/Province/Region/County/Territory/Prefecture/Republic","Zip/Postal Code","Country","Contact Phone Number","Subject","Note","Country Code","Balance Impact"
# This rules file assumes the following more detailed fields, configured in "Customize report fields":
# "Date","Time","TimeZone","Name","Type","Status","Currency","Gross","Fee","Net","From Email Address","To Email Address","Transaction ID","Item Title","Item ID","Reference Txn ID","Receipt ID","Balance","Note"
fields date, time, timezone, description_, type, status_, currency, grossamount, feeamount, netamount, fromemail, toemail, code, itemtitle, itemid, referencetxnid, receiptid, balance, note
skip 1
date-format %-m/%-d/%Y
# ignore some paypal events
if
In Progress
Temporary Hold
Update to
skip
# add more fields to the description
description %description_ %itemtitle
# save some other fields as tags
comment itemid:%itemid, fromemail:%fromemail, toemail:%toemail, time:%time, type:%type, status:%status_
# convert to short currency symbols
if %currency USD
currency $
if %currency EUR
currency E
if %currency GBP
currency P
# generate postings
# the first posting will be the money leaving/entering my paypal account
# (negative means leaving my account, in all amount fields)
account1 assets:online:paypal
amount1 %netamount
# the second posting will be money sent to/received from other party
# (account2 is set below)
amount2 -%grossamount
# if there's a fee, add a third posting for the money taken by paypal.
if %feeamount [1-9]
account3 expenses:banking:paypal
amount3 -%feeamount
comment3 business:
# choose an account for the second posting
# override the default account names:
# if the amount is positive, it's income (a debit)
if %grossamount ^[^-]
account2 income:unknown
# if negative, it's an expense (a credit)
if %grossamount ^-
account2 expenses:unknown
# apply common rules for setting account2 & other tweaks
include common.rules
# apply some overrides specific to this csv
# Transfers from/to bank. These are usually marked Pending,
# which can be disregarded in this case.
if
Bank Account
Bank Deposit to PP Account
description %type for %referencetxnid %itemtitle
account2 assets:bank:wf:pchecking
account1 assets:online:paypal
# Currency conversions
if Currency Conversion
account2 equity:currency conversion
```
```rules
# common.rules
if
darcs
noble benefactor
account2 revenues:foss donations:darcshub
comment2 business:
if
Calm Radio
account2 expenses:online:apps
if
electronic frontier foundation
Patreon
wikimedia
Advent of Code
account2 expenses:dues
if Google
account2 expenses:online:apps
description google | music
```
```shell
$ hledger -f paypal-custom.csv print
2019-10-01 (60P57143A8206782E) Calm Radio MONTHLY - $1 for the first 2 Months: Me - Order 99309. Item total: $1.00 USD first 2 months, then $6.99 / Month ; itemid:, fromemail:simon@joyful.com, toemail:memberships@calmradio.com, time:03:46:20, type:Subscription Payment, status:Completed
assets:online:paypal $-6.99 = $-6.99
expenses:online:apps $6.99
2019-10-01 (0TU1544T080463733) Bank Deposit to PP Account for 60P57143A8206782E ; itemid:, fromemail:, toemail:simon@joyful.com, time:03:46:20, type:Bank Deposit to PP Account, status:Pending
assets:online:paypal $6.99 = $0.00
assets:bank:wf:pchecking $-6.99
2019-10-01 (2722394R5F586712G) Patreon Patreon* Membership ; itemid:, fromemail:simon@joyful.com, toemail:support@patreon.com, time:08:57:01, type:PreApproved Payment Bill User Payment, status:Completed
assets:online:paypal $-7.00 = $-7.00
expenses:dues $7.00
2019-10-01 (71854087RG994194F) Bank Deposit to PP Account for 2722394R5F586712G Patreon* Membership ; itemid:, fromemail:, toemail:simon@joyful.com, time:08:57:01, type:Bank Deposit to PP Account, status:Pending
assets:online:paypal $7.00 = $0.00
assets:bank:wf:pchecking $-7.00
2019-10-19 (K9U43044RY432050M) Wikimedia Foundation, Inc. Monthly donation to the Wikimedia Foundation ; itemid:, fromemail:simon@joyful.com, toemail:tle@wikimedia.org, time:03:02:12, type:Subscription Payment, status:Completed
assets:online:paypal $-2.00 = $-2.00
expenses:dues $2.00
expenses:banking:paypal ; business:
2019-10-19 (3XJ107139A851061F) Bank Deposit to PP Account for K9U43044RY432050M ; itemid:, fromemail:, toemail:simon@joyful.com, time:03:02:12, type:Bank Deposit to PP Account, status:Pending
assets:online:paypal $2.00 = $0.00
assets:bank:wf:pchecking $-2.00
2019-10-22 (6L8L1662YP1334033) Noble Benefactor Joyful Systems ; itemid:, fromemail:noble@bene.fac.tor, toemail:simon@joyful.com, time:05:07:06, type:Subscription Payment, status:Completed
assets:online:paypal $9.41 = $9.41
revenues:foss donations:darcshub $-10.00 ; business:
expenses:banking:paypal $0.59 ; business:
```
# Timeclock # Timeclock
The time logging format of timeclock.el, as read by hledger. The time logging format of timeclock.el, as read by hledger.