From 33bedcbab07e4b5c702aee58f87bd66cc39c813e Mon Sep 17 00:00:00 2001 From: Simon Michael Date: Thu, 20 Jan 2011 00:18:54 +0000 Subject: [PATCH] parsing: infer a conversion price in unpriced two-commodity transactions --- MANUAL.markdown | 31 +++--- hledger-lib/Hledger/Data/Transaction.hs | 121 ++++++++++++++++++----- tests/prices.test | 123 ++++++++++++++++++++++++ tests/prices.tests | 66 ------------- 4 files changed, 238 insertions(+), 103 deletions(-) create mode 100644 tests/prices.test delete mode 100644 tests/prices.tests diff --git a/MANUAL.markdown b/MANUAL.markdown index 38f459ee1..967b4a48a 100644 --- a/MANUAL.markdown +++ b/MANUAL.markdown @@ -360,20 +360,28 @@ Or, you can write `@@ TOTALPRICE`, which is sometimes more convenient: assets:cash Or, you can set the price for this commodity as of a certain date, using a -historical price directive as shown here: +historical price directive (P) as shown: ; the exchange rate for euro is $1.35 on 2009/1/1 (and thereafter, until a newer price directive is found) ; four space-separated fields: P, date, commodity symbol, unit price in 2nd commodity P 2009/1/1 € $1.35 - 2009/1/2 x + 2009/1/2 expenses:foreign currency €100 assets -The print command shows any prices in effect. Either example above will show: +Or, you can write a transaction in two commodities, without prices but +with all amounts specified, and a conversion price will be inferred so as +to balance the transaction: + + 2009/1/2 + expenses:foreign currency €100 + assets $-135 + +The print command shows any prices in effect. So the first example above gives: $ hledger print - 2009/01/02 x + 2009/01/02 expenses:foreign currency €100 @ $1.35 assets €-100 @ $1.35 @@ -385,8 +393,8 @@ with any command: expenses:foreign currency $135.00 assets $-135.00 -The `--cost/-B` flag does only one lookup step, ie it will not look up the -price of a price's commodity. +In other words the `--cost/-B` flag converts amounts to their price's +commodity. (It will not look up the price of a price.) Note hledger handles prices differently from c++ ledger in this respect: we assume unit prices do not vary over time. This is good for simple @@ -1110,8 +1118,9 @@ entries, and the following c++ ledger options and commands: it does not print multi-commodity transactions in valid journal format.) - hledger's default commodity directive (D) sets the commodity for - subsequent commodityless amounts. ledger uses it only to set commodity - display settings and for the entry command. + subsequent commodityless amounts, and contributes to that commodity's + display settings. ledger uses D only for commodity display settings + and for the entry command. ## Troubleshooting @@ -1241,10 +1250,8 @@ Here are some issues you might encounter when you run hledger: the current list of things we know we don't parse (see also [file format compatibility](#file-format-compatibility): - - AMOUNT1 = AMOUNT2 (balance assertion ? price specification ?) - - specifying prices via postings in different commodities - - comma decimal point and period thousands separator, or any number - format other than the US standard + - AMOUNT1 = AMOUNT2 (balance assertion/price specification ?) + - ... AMOUNT {...} ## Examples and recipes diff --git a/hledger-lib/Hledger/Data/Transaction.hs b/hledger-lib/Hledger/Data/Transaction.hs index 392639e9d..4220cb359 100644 --- a/hledger-lib/Hledger/Data/Transaction.hs +++ b/hledger-lib/Hledger/Data/Transaction.hs @@ -15,7 +15,7 @@ import Hledger.Data.Types import Hledger.Data.Dates import Hledger.Data.Posting import Hledger.Data.Amount -import Hledger.Data.Commodity (dollars, dollar, unknown) +import Hledger.Data.Commodity instance Show Transaction where show = showTransactionUnelided @@ -133,29 +133,87 @@ isTransactionBalanced canonicalcommoditymap t = rsum' = canonicaliseMixedAmount canonicalcommoditymap $ costOfMixedAmount rsum bvsum' = canonicaliseMixedAmount canonicalcommoditymap $ costOfMixedAmount bvsum --- | Ensure that this entry is balanced, possibly auto-filling a missing --- amount first. We can auto-fill if there is just one non-virtual --- transaction without an amount. The auto-filled balance will be --- converted to cost basis if possible. If the entry can not be balanced, --- return an error message instead. +-- | Ensure this transaction is balanced, possibly inferring a missing +-- amount or a conversion price first, or return an error message. +-- +-- Balancing is affected by the provided commodities' display precisions. +-- +-- We can infer an amount when there are multiple real postings and +-- exactly one of them is amountless; likewise for balanced virtual +-- postings. Inferred amounts are converted to cost basis when possible. +-- +-- We can infer a price when all amounts were specified and the sum of +-- real postings' amounts is exactly two non-explicitly-priced amounts in +-- different commodities; likewise for balanced virtual postings. balanceTransaction :: Maybe (Map.Map String Commodity) -> Transaction -> Either String Transaction balanceTransaction canonicalcommoditymap t@Transaction{tpostings=ps} | length rwithoutamounts > 1 || length bvwithoutamounts > 1 = Left $ printerr "could not balance this transaction (too many missing amounts)" - | not $ isTransactionBalanced canonicalcommoditymap t' = Left $ printerr $ nonzerobalanceerror t' - | otherwise = Right t' + | not $ isTransactionBalanced canonicalcommoditymap t''' = Left $ printerr $ nonzerobalanceerror t''' + | otherwise = Right t''' where - rps = filter isReal ps - bvps = filter isBalancedVirtual ps - (rwithamounts, rwithoutamounts) = partition hasAmount rps - (bvwithamounts, bvwithoutamounts) = partition hasAmount bvps - t' = t{tpostings=map balance ps} + -- maybe infer missing amounts + (rwithamounts, rwithoutamounts) = partition hasAmount $ realPostings t + (bvwithamounts, bvwithoutamounts) = partition hasAmount $ balancedVirtualPostings t + ramounts = map pamount rwithamounts + bvamounts = map pamount bvwithamounts + t' = t{tpostings=map inferamount ps} where - balance p | not (hasAmount p) && isReal p - = p{pamount = (-(sum $ map pamount rwithamounts))} - | not (hasAmount p) && isBalancedVirtual p - = p{pamount = (-(sum $ map pamount bvwithamounts))} - | otherwise = p + inferamount p | not (hasAmount p) && isReal p = p{pamount = (- sum ramounts)} + | not (hasAmount p) && isBalancedVirtual p = p{pamount = (- sum bvamounts)} + | otherwise = p + + -- maybe infer conversion prices, for real postings + rmixedamountsinorder = map pamount $ realPostings t' + ramountsinorder = concatMap amounts rmixedamountsinorder + rcommoditiesinorder = map commodity ramountsinorder + rsumamounts = amounts $ sum rmixedamountsinorder + -- assumption: the sum of mixed amounts is normalised (one simple amount per commodity) + t'' = if length rsumamounts == 2 && all (isNothing.price) rsumamounts && t'==t + then t'{tpostings=map inferprice ps} + else t' + where + -- assumption: a posting's mixed amount contains one simple amount + inferprice p@Posting{pamount=Mixed [a@Amount{commodity=c,price=Nothing}], ptype=RegularPosting} + = p{pamount=Mixed [a{price=conversionprice c}]} + where + conversionprice c | c == unpricedcommodity + -- assign a balancing price. Use @@ for more exact output when possible. + = if length ramountsinunpricedcommodity == 1 + then Just $ TotalPrice $ Mixed [setAmountPrecision maxprecision $ negate $ targetcommodityamount] + else Just $ UnitPrice $ Mixed [setAmountPrecision maxprecision $ negate $ targetcommodityamount `divideAmount` (quantity unpricedamount)] + | otherwise = Nothing + where + unpricedcommodity = head $ filter (`elem` (map commodity rsumamounts)) rcommoditiesinorder + unpricedamount = head $ filter ((==unpricedcommodity).commodity) rsumamounts + targetcommodityamount = head $ filter ((/=unpricedcommodity).commodity) rsumamounts + ramountsinunpricedcommodity = filter ((==unpricedcommodity).commodity) ramountsinorder + inferprice p = p + + -- maybe infer prices for balanced virtual postings. Just duplicates the above for now. + bvmixedamountsinorder = map pamount $ balancedVirtualPostings t'' + bvamountsinorder = concatMap amounts bvmixedamountsinorder + bvcommoditiesinorder = map commodity bvamountsinorder + bvsumamounts = amounts $ sum bvmixedamountsinorder + t''' = if length bvsumamounts == 2 && all (isNothing.price) bvsumamounts && t'==t -- XXX could check specifically for bv amount inferring + then t''{tpostings=map inferprice ps} + else t'' + where + inferprice p@Posting{pamount=Mixed [a@Amount{commodity=c,price=Nothing}], ptype=BalancedVirtualPosting} + = p{pamount=Mixed [a{price=conversionprice c}]} + where + conversionprice c | c == unpricedcommodity + = if length bvamountsinunpricedcommodity == 1 + then Just $ TotalPrice $ Mixed [setAmountPrecision maxprecision $ negate $ targetcommodityamount] + else Just $ UnitPrice $ Mixed [setAmountPrecision maxprecision $ negate $ targetcommodityamount `divideAmount` (quantity unpricedamount)] + | otherwise = Nothing + where + unpricedcommodity = head $ filter (`elem` (map commodity bvsumamounts)) bvcommoditiesinorder + unpricedamount = head $ filter ((==unpricedcommodity).commodity) bvsumamounts + targetcommodityamount = head $ filter ((/=unpricedcommodity).commodity) bvsumamounts + bvamountsinunpricedcommodity = filter ((==unpricedcommodity).commodity) bvamountsinorder + inferprice p = p + printerr s = intercalate "\n" [s, showTransactionUnelided t] nonzerobalanceerror :: Transaction -> String @@ -172,7 +230,6 @@ nonzerobalanceerror t = printf "could not balance this transaction (%s%s%s)" rms journalTransactionWithDate :: WhichDate -> Transaction -> Transaction journalTransactionWithDate ActualDate t = t journalTransactionWithDate EffectiveDate t = txnTieKnot t{tdate=fromMaybe (tdate t) (teffectivedate t)} - -- | Ensure a transaction's postings refer back to it. txnTieKnot :: Transaction -> Transaction @@ -268,25 +325,39 @@ tests_Hledger_Data_Transaction = TestList [ assertBool "detect unbalanced entry, sign error" (isLeft $ balanceTransaction Nothing (Transaction (parsedate "2007/01/28") Nothing False "" "test" "" [] - [Posting False "a" (Mixed [dollars 1]) "" RegularPosting [] Nothing, + [Posting False "a" (Mixed [dollars 1]) "" RegularPosting [] Nothing, Posting False "b" (Mixed [dollars 1]) "" RegularPosting [] Nothing ] "")) assertBool "detect unbalanced entry, multiple missing amounts" (isLeft $ balanceTransaction Nothing (Transaction (parsedate "2007/01/28") Nothing False "" "test" "" [] - [Posting False "a" missingamt "" RegularPosting [] Nothing, + [Posting False "a" missingamt "" RegularPosting [] Nothing, Posting False "b" missingamt "" RegularPosting [] Nothing ] "")) - let e = balanceTransaction Nothing (Transaction (parsedate "2007/01/28") Nothing False "" "test" "" [] - [Posting False "a" (Mixed [dollars 1]) "" RegularPosting [] Nothing, + let e = balanceTransaction Nothing (Transaction (parsedate "2007/01/28") Nothing False "" "" "" [] + [Posting False "a" (Mixed [dollars 1]) "" RegularPosting [] Nothing, Posting False "b" missingamt "" RegularPosting [] Nothing ] "") - assertBool "one missing amount should be ok" (isRight e) - assertEqual "balancing amount is added" + assertBool "balanceTransaction allows one missing amount" (isRight e) + assertEqual "balancing amount is inferred" (Mixed [dollars (-1)]) (case e of Right e' -> (pamount $ last $ tpostings e') Left _ -> error' "should not happen") + let e = balanceTransaction Nothing (Transaction (parsedate "2011/01/01") Nothing False "" "" "" [] + [Posting False "a" (Mixed [dollars 1.35]) "" RegularPosting [] Nothing, + Posting False "b" (Mixed [euros (-1)]) "" RegularPosting [] Nothing + ] "") + assertBool "balanceTransaction can infer conversion price" (isRight e) + assertEqual "balancing conversion price is inferred" + (Mixed [Amount{commodity=dollar{precision=2}, + quantity=1.35, + price=(Just $ TotalPrice $ Mixed [Amount{commodity=euro{precision=maxprecision}, + quantity=1, + price=Nothing}])}]) + (case e of + Right e' -> (pamount $ head $ tpostings e') + Left _ -> error' "should not happen") ,"isTransactionBalanced" ~: do let t = Transaction (parsedate "2009/01/01") Nothing False "" "a" "" [] diff --git a/tests/prices.test b/tests/prices.test new file mode 100644 index 000000000..1fe0167be --- /dev/null +++ b/tests/prices.test @@ -0,0 +1,123 @@ +# price-related tests +# 1. print a transaction with an explicit unit price +bin/hledger -f- print +<<< +2011/01/01 + expenses:foreign currency €100 @ $1.35 + assets +>>> +2011/01/01 + expenses:foreign currency €100 @ $1.35 + assets €-100 @ $1.35 + +# 2. convert to cost basis +bin/hledger -f- print -B +<<< +2011/01/01 + expenses:foreign currency €100 @ $1.35 + assets +>>> +2011/01/01 + expenses:foreign currency $135.00 + assets $-135.00 + +# 3. with a historical price directive +bin/hledger -f- print -B +<<< +P 2010/12/31 € $1.34 +P 2011/01/01 € $1.35 +P 2011/01/02 € $1.36 + +2011/01/01 + expenses:foreign currency €100 + assets + +>>> +2011/01/01 + expenses:foreign currency $135.00 + assets $-135.00 + +# 4. with a total price +bin/hledger -f - print +<<< +2011/01/01 + expenses:foreign currency €100 @@ $135 + assets +>>> +2011/01/01 + expenses:foreign currency €100 @@ $135 + assets €-100 @@ $135 + +# 5. when the balance has exactly two commodities, both unpriced, infer an +# implicit conversion price for the first one in terms of the second. +bin/hledger -f - print +<<< +2011/01/01 + expenses:foreign currency €100 + misc $2.1 + assets $-135.00 + misc €1 + misc €-1 + misc $-2.1 +>>> +2011/01/01 + expenses:foreign currency €100 @ $1.35 + misc $2.10 + assets $-135.00 + misc €1 @ $1.35 + misc €-1 @ $1.35 + misc $-2.10 + +# # 6. when the *cost-basis* balance has exactly two commodities, both +# # unpriced, infer an implicit conversion price for the first one in terms +# # of the second. +# bin/hledger -f - print +# <<< +# 2011/01/01 +# expenses:foreign currency €100 +# assets $-135.00 +# misc $3.1 @ 2 bob +# misc $-3.1 @ 2 bob +# misc £1 @@ 2 shekels +# misc £-1 @@ 2 shekels +# >>> +# 2011/01/01 +# expenses:foreign currency €100 @ $1.35 +# assets €-100 @ $1.35 +# misc $3.1 @ 2 bob +# misc $-3.1 @ 2 bob +# misc £1 @@ 2 shekels +# misc £-1 @@ 2 shekels +# +## 7. another, from ledger tests. Just one posting to price so uses @@. +bin/hledger -f - print +<<< +2002/09/30 * 1a1a6305d06ce4b284dba0d267c23f69d70c20be + c56a21d23a6535184e7152ee138c28974f14280c 866.231000 GGGGG + a35e82730cf91569c302b313780e5895f75a62b9 $-17,783.72 +>>> +2002/09/30 * 1a1a6305d06ce4b284dba0d267c23f69d70c20be + c56a21d23a6535184e7152ee138c28974f14280c 866.231000 GGGGG @@ $17,783.72 + a35e82730cf91569c302b313780e5895f75a62b9 $-17,783.72 + +# 8. when the balance has more than two commodities, don't bother +bin/hledger -f - print +<<< +2011/01/01 + expenses:foreign currency €100 + assets $-135 + expenses:other £200 +>>>= !0 +# 9. another +bin/hledger -f - balance -B +<<< +2011/01/01 + expenses:foreign currency €99 + assets $-130 + expenses:foreign currency €1 + assets $-5 +>>> + $-135 assets + $135 expenses:foreign currency +-------------------- + $0 diff --git a/tests/prices.tests b/tests/prices.tests deleted file mode 100644 index b989a5628..000000000 --- a/tests/prices.tests +++ /dev/null @@ -1,66 +0,0 @@ -# price-related tests -# 1. print a transaction with an explicit unit price -bin/hledger -f- print -<<< -2011/01/01 - expenses:foreign currency €100 @ $1.35 - assets ->>> -2011/01/01 - expenses:foreign currency €100 @ $1.35 - assets €-100 @ $1.35 - -# 2. convert to cost basis -bin/hledger -f- print -B -<<< -2011/01/01 - expenses:foreign currency €100 @ $1.35 - assets ->>> -2011/01/01 - expenses:foreign currency $135.00 - assets $-135.00 - -# 2. with a historical price directive -bin/hledger -f- print -B -<<< -P 2010/12/31 € $1.34 -P 2011/01/01 € $1.35 -P 2011/01/02 € $1.36 - -2011/01/01 - expenses:foreign currency €100 - assets - ->>> -2011/01/01 - expenses:foreign currency $135.00 - assets $-135.00 - -# 3. with a total price -bin/hledger -f - print -<<< -2011/01/01 - expenses:foreign currency €100 @@ $135 - assets ->>> -2011/01/01 - expenses:foreign currency €100 @@ $135 - assets €-100 @@ $135 - -# 4. with an implicit price -# bin/hledger -f - print -# <<< -# 2011/01/01 -# expenses:foreign currency €100 @ $1.35 -# assets $-135.00 -# >>> -# 2011/01/01 -# expenses:foreign currency €100 @ $1.35 -# assets -# -# 2009/1/1 opening balance -# Assets:Brokerage 1 AAPL -# Assets:Checking $-20.00 -# -# >>>= 0