;examples:csv: Open Collective
This commit is contained in:
Simon Michael
parent
3d5704fe22
commit
2dd2ad5cf9
85
examples/csv/other/opencollective/README.md
Normal file
85
examples/csv/other/opencollective/README.md
Normal file
@ -0,0 +1,85 @@
|
||||
Here are notes on Open Collective's complicated CSV story (as of 2025-12, based on hledger's open collective).
|
||||
|
||||
On the Transactions screen, first remove any filters so that all transactions will be exported, including:
|
||||
|
||||
- Contribution (incoming or outgoing)
|
||||
- Expense
|
||||
- Host fee
|
||||
- Payment processor cover
|
||||
- Payment processor fee
|
||||
|
||||
Eg: <https://opencollective.com/hledger/transactions?kind=ALL>
|
||||
|
||||
When you click the Export CSV button, two CSV formats are offered:
|
||||
|
||||
- Legacy Platform Default (Pre-2024) (we'll call it Legacy)
|
||||
- Platform Default (we'll call it Default)
|
||||
|
||||
Each has three options:
|
||||
|
||||
- Selected fields for export - about 25 reorderable fields; we'll assume the default order.
|
||||
(It says "out of 85", suggesting there might be more available, but if so they're not accessible to a normal collective admin.)
|
||||
- Use field IDs as column headers instead of field names - normally on for Legacy and off for Default.
|
||||
- Export taxes and payment processor fees as columns - ditto.
|
||||
|
||||
Additionally, the data available varies by era:
|
||||
|
||||
- Host fee records start in 2021-06.
|
||||
- With the "as columns" option on, payment processor fees are available for all eras. Otherwise, processor fee records start in 2024-01.
|
||||
- Payment processor cover records start in 2024-01.
|
||||
|
||||
In eras where fees are not available, contribution amounts are under-reported.
|
||||
Eg, a $10 contribution with a $1 host fee and $0.59 payment processor fee might be reported as just a $8.41 contribution.
|
||||
|
||||
Here are the formats we provide rules for, and their limitations:
|
||||
|
||||
| format | accurate contribution amounts? | host fees? | processor fees? | processor covers? | balances? | notes | rules file |
|
||||
|---------------------------|-------------------------------------|---------------------------------------|-----------------|-------------------|---------------------|---------------------------------------------|------------------------------|
|
||||
| Legacy | yes | from 2021/06 (inferrable before then) | yes | from 2024 | yes; some are wrong | provides the most data across all eras | opencollective-legacy.rules |
|
||||
| Default + fees as columns | from 2024 (inferrable from 2021/06) | from 2021/06 | yes | from 2024 | no | simplest if you have no data before 2021-06 | opencollective-columns.rules |
|
||||
| Default | from 2024 | from 2021/06 | from 2024 | from 2024 | no | simplest if you have no data before 2024 | opencollective-default.rules |
|
||||
|
||||
opencollective-legacy.rules is used by the hledger project and is the best-tested.
|
||||
|
||||
## How to test your conversion
|
||||
|
||||
### CSV balance column
|
||||
|
||||
If you are using the legacy format, you can use the balance column to generate hledger balance assertions.
|
||||
Though in some record types (eg contributions after 2021-06, and expenses) the balance values may be out of sequence; those should be ignored.
|
||||
|
||||
Then, hledger should agree with Open Collective's balances; check with `hledger check assertions` or `hledger check -s`.
|
||||
|
||||
### Budget stats
|
||||
|
||||
In the Budget section on the collective's page (eg <https://opencollective.com/hledger#category-BUDGET>),
|
||||
you can see some calculated values (and a few more if you mouse over), listed below.
|
||||
Some of these can be reproduced by the hledger reports shown.
|
||||
(If you have used the legacy format and the provided rules; or if your data begins in 2024 or later.
|
||||
Tested with the hledger collective.)
|
||||
|
||||
#### Today's balance
|
||||
|
||||
`hledger bs`
|
||||
|
||||
#### Total consolidated including Projects and Events
|
||||
|
||||
#### Total raised
|
||||
|
||||
`hledger is not:disbursed` *? not reproduced yet*
|
||||
|
||||
#### Total contributed before fees
|
||||
|
||||
`hledger is revenues`
|
||||
|
||||
#### Total disbursed
|
||||
|
||||
`hledger is disbursed` *? not reproduced yet*
|
||||
|
||||
#### Estimated annual budget
|
||||
|
||||
#### Monthly recurring
|
||||
|
||||
#### Total received in the last 12 months
|
||||
|
||||
`hledger is revenues -b $(date -I -d '-12 months')`
|
||||
@ -0,0 +1,58 @@
|
||||
# hledger 1.51 conversion rules for Open Collective's "Platform Default" CSV format
|
||||
# with the "Export taxes and payment processor fees as columns" option enabled.
|
||||
# See README.md for more details.
|
||||
#
|
||||
# Examples:
|
||||
#
|
||||
# A contribution transaction before 2021-06:
|
||||
# "Effective Date & Time","Transaction ID","Description","Credit/Debit","Kind","Group ID","Amount Single Column","Currency","Is Reverse","Is Reversed","Reverse Transaction ID","Account Handle","Account Name","Opposite Account Handle","Opposite Account Name","Payment Processor","Payment Method","Contribution Memo","Expense Type","Expense Tags","Expense Payout Method Type","Accounting Category Code","Accounting Category Name","Merchant ID","Reverse Kind","Payment Processor Fee","Tax Amount"
|
||||
# "2017-01-20T19:21:45",5126,"Monthly contribution from Simon Michael (Bronze)","CREDIT","CONTRIBUTION","8b272eb0-d303-4cea-99db-3bcc40f54518",8.41,"USD","","","","hledger","hledger","simon","Simon Michael","STRIPE","CREDITCARD",,,"",,"","",,,-0.59,0
|
||||
#
|
||||
# A contribution transaction after 2021-06:
|
||||
# "2025-12-23T10:59:07",11299548,"Monthly contribution from Guest (Bronze)","CREDIT","CONTRIBUTION","d40ea8c3-f20f-4a88-8426-8c7405be447c",1.41,"USD","","","","hledger","hledger","guest-d120c27f","Guest","PAYPAL","SUBSCRIPTION",,,"",,"","",,,-0.59,0
|
||||
# "2025-12-23T10:59:07",11299543,"Host Fee to Open Source Collective","DEBIT","HOST_FEE","d40ea8c3-f20f-4a88-8426-8c7405be447c",-0.2,"USD","","","","hledger","hledger","opensource","Open Source Collective",,,,,"",,"","",,,0,0
|
||||
|
||||
source hledger-transactions.csv
|
||||
#archive
|
||||
fields date_time, transaction_id, description, credit_debit, kind, group_id, amount_single_column, currency_, is_reverse, is_reversed, reverse_transaction_id, account_handle, account_name, opposite_account_handle, opposite_account_name, payment_processor, payment_method, contribution_memo, expense_type, expense_tags, expense_payout_method_type, accounting_category_code, accounting_category_name, merchant_id, reverse_kind, payment_processor_fee, tax_amount
|
||||
skip 1
|
||||
newest-first
|
||||
date-format %Y-%m-%dT%H:%M:%S
|
||||
date %date_time
|
||||
comment \nid:%transaction_id, group:%group_id, dc:%type
|
||||
if %is_reverse .
|
||||
comment \nid:%transaction_id, reversing:%reverse_transaction_id, group:%group_id, dc:%type
|
||||
|
||||
# We will generate postings in FROM,TO order most of the time:
|
||||
# posting1 is the source, if any
|
||||
# posting2 is this open collective account
|
||||
# posting3 is the payment processor fee, if any
|
||||
# posting4 is the destination, if any
|
||||
|
||||
account2 assets:opencollective:%account_name
|
||||
amount2 %amount_single_column %currency_
|
||||
|
||||
|
||||
if %payment_processor_fee [1-9]
|
||||
account3 expenses:fees:processor:%payment_processor
|
||||
amount3 -%payment_processor_fee %currency_
|
||||
|
||||
if %payment_processor_fee [1-9] && ! %payment_processor .
|
||||
account3 expenses:fees:processor:%expense_payout_method_type
|
||||
|
||||
|
||||
if %kind CONTRIBUTION
|
||||
account1 revenues:contributions:%opposite_account_name
|
||||
|
||||
if %kind CONTRIBUTION && %credit_debit DEBIT && ! %is_reverse REVERSE
|
||||
account1 expenses:disbursed:%opposite_account_name
|
||||
|
||||
|
||||
if %kind EXPENSE
|
||||
account4 expenses:disbursed:%opposite_account_name
|
||||
|
||||
if %kind HOST_FEE
|
||||
account4 expenses:fees:host:%opposite_account_name
|
||||
|
||||
if %kind PAYMENT_PROCESSOR_COVER
|
||||
account4 expenses:fees:processor:%opposite_account_name
|
||||
@ -0,0 +1,61 @@
|
||||
# hledger 1.51 conversion rules for Open Collective's "Platform Default" CSV format.
|
||||
# See README.md for more details.
|
||||
#
|
||||
# Examples:
|
||||
#
|
||||
# A contribution transaction before 2021-06:
|
||||
# "Effective Date & Time","Transaction ID","Description","Credit/Debit","Kind","Group ID","Amount Single Column","Currency","Is Reverse","Is Reversed","Reverse Transaction ID","Account Handle","Account Name","Opposite Account Handle","Opposite Account Name","Payment Processor","Payment Method","Contribution Memo","Expense Type","Expense Tags","Expense Payout Method Type","Accounting Category Code","Accounting Category Name","Merchant ID","Reverse Kind"
|
||||
# "2017-01-20T19:21:45",5126,"Monthly contribution from Simon Michael (Bronze)","CREDIT","CONTRIBUTION","8b272eb0-d303-4cea-99db-3bcc40f54518",8.41,"USD","","","","hledger","hledger","simon","Simon Michael","STRIPE","CREDITCARD",,,"",,"","",,
|
||||
#
|
||||
# A contribution transaction after 2021-06:
|
||||
# "2025-12-23T10:59:07",11299548,"Monthly contribution from Guest (Bronze)","CREDIT","CONTRIBUTION","d40ea8c3-f20f-4a88-8426-8c7405be447c",2,"USD","","","","hledger","hledger","guest-d120c27f","Guest","PAYPAL","SUBSCRIPTION",,,"",,"","",,
|
||||
# "2025-12-23T10:59:07",11299545,"Transaction to PayPal","DEBIT","PAYMENT_PROCESSOR_FEE","d40ea8c3-f20f-4a88-8426-8c7405be447c",-0.59,"USD","","","","hledger","hledger","paypal-payment-processor-vendor","PayPal","PAYPAL","SUBSCRIPTION",,,"",,"","",,
|
||||
# "2025-12-23T10:59:07",11299543,"Host Fee to Open Source Collective","DEBIT","HOST_FEE","d40ea8c3-f20f-4a88-8426-8c7405be447c",-0.2,"USD","","","","hledger","hledger","opensource","Open Source Collective",,,,,"",,"","",,
|
||||
|
||||
source hledger-transactions.csv
|
||||
#archive
|
||||
fields date_time, transaction_id, description, credit_debit, kind, group_id, amount_single_column, currency_, is_reverse, is_reversed, reverse_transaction_id, account_handle, account_name, opposite_account_handle, opposite_account_name, payment_processor, payment_method, contribution_memo, expense_type, expense_tags, expense_payout_method_type, accounting_category_code, accounting_category_name, merchant_id, reverse_kind, payment_processor_fee, tax_amount
|
||||
skip 1
|
||||
newest-first
|
||||
date-format %Y-%m-%dT%H:%M:%S
|
||||
date %date_time
|
||||
comment \nid:%transaction_id, group:%group_id, dc:%type
|
||||
if %is_reverse .
|
||||
comment \nid:%transaction_id, reversing:%reverse_transaction_id, group:%group_id, dc:%type
|
||||
|
||||
# We will generate postings in FROM,TO order most of the time:
|
||||
# posting1 is the source, if any
|
||||
# posting2 is this open collective account
|
||||
# posting3 is the payment processor fee, if any
|
||||
# posting4 is the destination, if any
|
||||
|
||||
account2 assets:opencollective:%account_name
|
||||
amount2 %amount_single_column %currency_
|
||||
|
||||
|
||||
if %payment_processor_fee [1-9]
|
||||
account3 expenses:fees:processor:%payment_processor
|
||||
amount3 -%payment_processor_fee %currency_
|
||||
|
||||
if %payment_processor_fee [1-9] && ! %payment_processor .
|
||||
account3 expenses:fees:processor:%expense_payout_method_type
|
||||
|
||||
|
||||
if %kind CONTRIBUTION
|
||||
account1 revenues:contributions:%opposite_account_name
|
||||
|
||||
if %kind CONTRIBUTION && %credit_debit DEBIT && ! %is_reverse REVERSE
|
||||
account1 expenses:disbursed:%opposite_account_name
|
||||
|
||||
|
||||
if %kind EXPENSE
|
||||
account4 expenses:disbursed:%opposite_account_name
|
||||
|
||||
if %kind HOST_FEE
|
||||
account4 expenses:fees:host:%opposite_account_name
|
||||
|
||||
if %kind PAYMENT_PROCESSOR_FEE
|
||||
account4 expenses:fees:processor:%payment_processor
|
||||
|
||||
if %kind PAYMENT_PROCESSOR_COVER
|
||||
account4 expenses:fees:processor:%opposite_account_name
|
||||
@ -0,0 +1,76 @@
|
||||
# hledger 1.51 conversion rules for Open Collective's "Legacy Platform Default (Pre-2024)" CSV format.
|
||||
# See README.md for more details.
|
||||
#
|
||||
# Examples:
|
||||
#
|
||||
# A contribution transaction before 2021-06: (host fee is missing but can be inferred)
|
||||
# "datetime","shortId","shortGroup","description","type","kind","isRefund","isRefunded","shortRefundId","displayAmount","amount","paymentProcessorFee","netAmount","balance","currency","accountSlug","accountName","oppositeAccountSlug","oppositeAccountName","paymentMethodService","paymentMethodType","expenseType","expenseTags","payoutMethodType","merchantId","orderMemo","taxAmount"
|
||||
# "2017-01-20T19:21:45","f50dc2b7","8b272eb0","Monthly contribution from Simon Michael (Bronze)","CREDIT","CONTRIBUTION","","","","$10.00 USD",10,-0.59,8.41,8.41,"USD","hledger","hledger","simon","Simon Michael","STRIPE","CREDITCARD",,"",,,,0
|
||||
#
|
||||
# A contribution transaction after 2021-06:
|
||||
# "2025-12-23T10:59:20","e34843a9","d40ea8c3","Monthly contribution from Guest (Bronze)","CREDIT","CONTRIBUTION","","","","$2.00 USD",2,-0.59,1.41,7128.76,"USD","hledger","hledger","guest-d120c27f","Guest","PAYPAL","SUBSCRIPTION",,"",,,,0
|
||||
# "2025-12-23T10:59:20","94960bd6","d40ea8c3","Host Fee to Open Source Collective","DEBIT","HOST_FEE","","","","-$0.20 USD",-0.2,0,-0.2,7127.97,"USD","hledger","hledger","opensource","Open Source Collective",,,,"",,,,0
|
||||
|
||||
source hledger-transactions.csv
|
||||
# archive
|
||||
fields date, shortId, shortGroup, description, type, kind, isRefund, isRefunded, shortRefundId, displayAmount, amount_, paymentProcessorFee, netAmount, balance_, currency_, accountSlug, accountName, oppositeAccountSlug, oppositeAccountName, paymentMethodService, paymentMethodType, expenseType, expenseTags, payoutMethodType, merchantId, orderMemo, taxAmount
|
||||
skip 1
|
||||
newest-first
|
||||
date-format %Y-%m-%dT%H:%M:%S
|
||||
comment \nid:%shortId, group:%shortGroup, dc:%type, payment-service:%paymentMethodService, payment-type:%paymentMethodType
|
||||
if %isRefund .
|
||||
comment \nid:%shortId, refunding:%shortRefundId, group:%shortGroup, dc:%type, payment-service:%paymentMethodService, payment-type:%paymentMethodType
|
||||
|
||||
# We will generate postings in FROM,TO order most of the time:
|
||||
# posting1 is the source, if any
|
||||
# posting2 is the incoming payment processor fee, if any
|
||||
# posting3 is the incoming host fee, if any
|
||||
# posting4 is this open collective account
|
||||
# posting5 is the outgoing payment processor fee, if any
|
||||
# posting6 is the outgoing host fee, if any
|
||||
# posting7 is another destination, if any
|
||||
|
||||
account4 assets:opencollective:%accountName
|
||||
amount4 %amount_ %currency_
|
||||
|
||||
# if %date < 2021-06
|
||||
if %date ^(20[01]|2020|2021-0[1-5]) && %kind CONTRIBUTION
|
||||
account1 revenues:contributions:%oppositeAccountName
|
||||
amount1 -%amount_ %currency_
|
||||
account2 expenses:fees:processor:%paymentMethodService
|
||||
amount2 -%paymentProcessorFee %currency_
|
||||
account3 expenses:fees:host:Open Source Collective
|
||||
account4 assets:opencollective:%accountName
|
||||
amount4 %netAmount %currency_
|
||||
balance4 %balance_ %currency_
|
||||
# the fiscal host name is hard-coded here
|
||||
|
||||
# if %date >= 2021-06
|
||||
if %date ^(2021-(0[6-9]|1)|202[2-9]|20[3-9]|2[1-9]|[3-9]) && %kind CONTRIBUTION
|
||||
account1 revenues:contributions:%oppositeAccountName
|
||||
amount1 -%amount_ %currency_
|
||||
account2 expenses:fees:processor:%paymentMethodService
|
||||
amount2 -%paymentProcessorFee %currency_
|
||||
account4 assets:opencollective:%accountName
|
||||
amount4 %netAmount %currency_
|
||||
# some records have wrong balance values which can't be used
|
||||
# balance4 %balance_ %currency_ # wrong
|
||||
|
||||
if %date ^(2021-(0[6-9]|1)|202[2-9]|20[3-9]|2[1-9]|[3-9]) && %kind CONTRIBUTION && %type DEBIT
|
||||
account1 expenses:disbursed:%oppositeAccountName
|
||||
|
||||
if %date ^(2021-(0[6-9]|1)|202[2-9]|20[3-9]|2[1-9]|[3-9]) && %kind CONTRIBUTION && %isRefund REFUND
|
||||
account1 revenues:contributions:%oppositeAccountName
|
||||
|
||||
if %kind EXPENSE
|
||||
amount4 %netAmount %currency_
|
||||
account5 expenses:fees:processor:%payoutMethodType
|
||||
amount5 -%paymentProcessorFee %currency_
|
||||
account7 expenses:disbursed:%oppositeAccountName
|
||||
amount7 -%amount_ %currency_
|
||||
# balance4 %balance_ %currency_ # wrong
|
||||
|
||||
if %kind HOST_FEE|PAYMENT_PROCESSOR_COVER
|
||||
balance4 %balance_ %currency_
|
||||
account6 expenses:fees:host:%oppositeAccountName
|
||||
amount6 -%amount_ %currency_
|
||||
Loading…
Reference in New Issue
Block a user