From 4049455f26de2974e83f351d84b1f60eba149c34 Mon Sep 17 00:00:00 2001 From: Dmitry Astapov Date: Fri, 24 Nov 2017 21:51:51 +0000 Subject: [PATCH 01/13] lib: Fix splitSpan for nthdayof{week,month} - start of DateSpan was not covered Demonstration: Consider year-test.journal: ``` 2015/02/01 first half expenses $1 assets 2015/07/01 second half expenses $2 assets 2016/02/01 first half expenses $4 assets 2016/07/01 second half expenses $8 assets 2017/02/01 first half expenses $16 assets 2017/07/01 second half expenses $32 assets ``` Year balances are good: ``` $ hledger balance -f year-test.journal -p yearly Balance changes in 2015/01/01-2017/12/31: || 2015 2016 2017 ==========++================== assets || $-3 $-12 $-48 expenses || $3 $12 $48 ----------++------------------ || 0 0 0 ``` Note how first transaction in 2015 is not included. Note that this is old period expression, so this bug exsits in master: ```$ hledger balance -f year-test.journal -p 'every 2nd day of month' Balance changes in 2015/07/02-2017/07/01: || 2015/07/02-2015/08/01 2015/08/02-2015/09/01 2015/09/02-2015/10/01 2015/10/02-2015/11/01 2015/11/02-2015/12/01 2015/12/02-2016/01/01 2016/01/02-2016/02/01 2016/02/02-2016/03/01 2016/03/02-2016/04/01 2016/04/02-2016/05/01 2016/05/02-2016/06/01 2016/06/02-2016/07/01 2016/07/02-2016/08/01 2016/08/02-2016/09/01 2016/09/02-2016/10/01 2016/10/02-2016/11/01 2016/11/02-2016/12/01 2016/12/02-2017/01/01 2017/01/02-2017/02/01 2017/02/02-2017/03/01 2017/03/02-2017/04/01 2017/04/02-2017/05/01 2017/05/02-2017/06/01 2017/06/02-2017/07/01 ==========++======================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================== assets || 0 0 0 0 0 0 $-4 0 0 0 0 $-8 0 0 0 0 0 0 $-16 0 0 0 0 $-32 expenses || 0 0 0 0 0 0 $4 0 0 0 0 $8 0 0 0 0 0 0 $16 0 0 0 0 $32 ----------++------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ || 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ``` Note how 2015 is absent entirely. This is new expression, but i think that general nature of bug is the same... ``` $ hledger balance -f year-test.journal -p 'every 4th Apr' Balance changes in 2016/04/04-2018/04/03: || 2016/04/04-2017/04/03 2017/04/04-2018/04/03 ==========++============================================== assets || $-24 $-32 expenses || $24 $32 ----------++---------------------------------------------- || 0 0 ``` --- hledger-lib/Hledger/Data/Dates.hs | 55 +++++++++++++++++++++++++------ 1 file changed, 45 insertions(+), 10 deletions(-) diff --git a/hledger-lib/Hledger/Data/Dates.hs b/hledger-lib/Hledger/Data/Dates.hs index 6ce584654..ab8590935 100644 --- a/hledger-lib/Hledger/Data/Dates.hs +++ b/hledger-lib/Hledger/Data/Dates.hs @@ -165,9 +165,9 @@ spansSpan spans = DateSpan (maybe Nothing spanStart $ headMay spans) (maybe Noth -- >>> t (Weeks 2) "2008/01/01" "2008/01/15" -- [DateSpan 2007/12/31-2008/01/13,DateSpan 2008/01/14-2008/01/27] -- >>> t (DayOfMonth 2) "2008/01/01" "2008/04/01" --- [DateSpan 2008/01/02-2008/02/01,DateSpan 2008/02/02-2008/03/01,DateSpan 2008/03/02-2008/04/01] +-- [DateSpan 2007/12/02-2008/01/01,DateSpan 2008/01/02-2008/02/01,DateSpan 2008/02/02-2008/03/01,DateSpan 2008/03/02-2008/04/01] -- >>> t (DayOfWeek 2) "2011/01/01" "2011/01/15" --- [DateSpan 2011/01/04-2011/01/10,DateSpan 2011/01/11-2011/01/17] +-- [DateSpan 2010/12/28-2011/01/03,DateSpan 2011/01/04-2011/01/10,DateSpan 2011/01/11-2011/01/17] -- splitSpan :: Interval -> DateSpan -> [DateSpan] splitSpan _ (DateSpan Nothing Nothing) = [DateSpan Nothing Nothing] @@ -461,16 +461,51 @@ prevyear = startofyear . addGregorianYearsClip (-1) nextyear = startofyear . addGregorianYearsClip 1 startofyear day = fromGregorian y 1 1 where (y,_,_) = toGregorian day -nthdayofmonthcontaining n d | d1 >= d = d1 - | otherwise = d2 - where d1 = addDays (fromIntegral n-1) s - d2 = addDays (fromIntegral n-1) $ nextmonth s +-- | For given date d find month-long interval that starts on nth day of month +-- and covers it. +-- +-- Examples: lets take 2017-11-22. Month-long intervals covering it that +-- start on 1st-22nd of month will start in Nov. However +-- intervals that start on 23rd-30th of month should start in Oct: +-- >>> let wed22nd = parsedate "2017-11-22" +-- >>> nthdayofmonthcontaining 1 wed22nd +-- 2017-11-01 +-- >>> nthdayofmonthcontaining 12 wed22nd +-- 2017-11-12 +-- >>> nthdayofmonthcontaining 22 wed22nd +-- 2017-11-22 +-- >>> nthdayofmonthcontaining 23 wed22nd +-- 2017-10-23 +-- >>> nthdayofmonthcontaining 30 wed22nd +-- 2017-10-30 +nthdayofmonthcontaining n d | nthOfSameMonth <= d = nthOfSameMonth + | otherwise = nthOfPrevMonth + where nthOfSameMonth = addDays (fromIntegral n-1) s + nthOfPrevMonth = addDays (fromIntegral n-1) $ prevmonth s s = startofmonth d -nthdayofweekcontaining n d | d1 >= d = d1 - | otherwise = d2 - where d1 = addDays (fromIntegral n-1) s - d2 = addDays (fromIntegral n-1) $ nextweek s + +-- | For given date d find week-long interval that starts on nth day of week +-- and covers it. +-- +-- Examples: 2017-11-22 is Wed. Week-long intervals that cover it and +-- start on Mon, Tue or Wed will start in the same week. However +-- intervals that start on Thu or Fri should start in prev week: +-- >>> let wed22nd = parsedate "2017-11-22" +-- >>> nthdayofweekcontaining 1 wed22nd +-- 2017-11-20 +-- >>> nthdayofweekcontaining 2 wed22nd +-- 2017-11-21 +-- >>> nthdayofweekcontaining 3 wed22nd +-- 2017-11-22 +-- >>> nthdayofweekcontaining 4 wed22nd +-- 2017-11-16 +-- >>> nthdayofweekcontaining 5 wed22nd +-- 2017-11-17 +nthdayofweekcontaining n d | nthOfSameWeek <= d = nthOfSameWeek + | otherwise = nthOfPrevWeek + where nthOfSameWeek = addDays (fromIntegral n-1) s + nthOfPrevWeek = addDays (fromIntegral n-1) $ prevweek s s = startofweek d ---------------------------------------------------------------------- From 7acb5d45aafa6e22cfaea49a0ebce5ab10b21758 Mon Sep 17 00:00:00 2001 From: Dmitry Astapov Date: Sat, 25 Nov 2017 00:42:39 +0000 Subject: [PATCH 02/13] lib: make month names in period expressions case-insensitive Currently only lower-case account names are supported, both on the command line, and in the journal (in periodic transactions): This works: $ hledger balance -p nov This does not: $ hledger balance -p Nov First transaction will parse, second will not: ``` cat every-month.journal ~/devel/haskell/darcs-get/hledger/examples ~ aug to sep assets expenses $1 ~ Aug to Sep assets expenses $2 ``` $../bin/hledger-budget bal -f every-month.journal hledger-budget: Failed to parse "Aug to Sep": date parse error () This commit fixes both cases. --- hledger-lib/Hledger/Data/Dates.hs | 18 ++++++------------ 1 file changed, 6 insertions(+), 12 deletions(-) diff --git a/hledger-lib/Hledger/Data/Dates.hs b/hledger-lib/Hledger/Data/Dates.hs index ab8590935..c454afd28 100644 --- a/hledger-lib/Hledger/Data/Dates.hs +++ b/hledger-lib/Hledger/Data/Dates.hs @@ -257,7 +257,7 @@ earliest (Just d1) (Just d2) = Just $ min d1 d2 -- | Parse a period expression to an Interval and overall DateSpan using -- the provided reference date, or return a parse error. parsePeriodExpr :: Day -> Text -> Either (ParseError Char MPErr) (Interval, DateSpan) -parsePeriodExpr refdate = parsewith (periodexpr refdate <* eof) +parsePeriodExpr refdate s = parsewith (periodexpr refdate <* eof) (T.toLower s) maybePeriod :: Day -> Text -> Maybe (Interval,DateSpan) maybePeriod refdate = either (const Nothing) Just . parsePeriodExpr refdate @@ -671,14 +671,8 @@ monthabbrevs = ["jan","feb","mar","apr","may","jun","jul","aug","sep","oct","n -- weekdays = ["monday","tuesday","wednesday","thursday","friday","saturday","sunday"] -- weekdayabbrevs = ["mon","tue","wed","thu","fri","sat","sun"] -#if MIN_VERSION_megaparsec(6,0,0) -lc = T.toLower -#else -lc = lowercase -#endif - -monthIndex t = maybe 0 (+1) $ lc t `elemIndex` months -monIndex t = maybe 0 (+1) $ lc t `elemIndex` monthabbrevs +monthIndex t = maybe 0 (+1) $ t `elemIndex` months +monIndex t = maybe 0 (+1) $ t `elemIndex` monthabbrevs month :: SimpleTextParser SmartDate month = do @@ -718,12 +712,12 @@ lastthisnextthing = do return ("", T.unpack r, T.unpack p) -- | --- >>> let p = parsewith (periodexpr (parsedate "2008/11/26")) :: T.Text -> Either (ParseError Char MPErr) (Interval, DateSpan) --- >>> p "from aug to oct" +-- >>> let p s = parsewith (periodexpr (parsedate "2008/11/26")) (T.toLower s) :: Either (ParseError Char MPErr) (Interval, DateSpan) +-- >>> p "from Aug to Oct" -- Right (NoInterval,DateSpan 2008/08/01-2008/09/30) -- >>> p "aug to oct" -- Right (NoInterval,DateSpan 2008/08/01-2008/09/30) --- >>> p "every 3 days in aug" +-- >>> p "every 3 days in Aug" -- Right (Days 3,DateSpan 2008/08) -- >>> p "daily from aug" -- Right (Days 1,DateSpan 2008/08/01-) From f1b4618f2da26e8e89db50a769a24123c85bc513 Mon Sep 17 00:00:00 2001 From: Dmitry Astapov Date: Fri, 24 Nov 2017 22:43:53 +0000 Subject: [PATCH 03/13] lib: support "every 11th Nov" in period expressions Useful for periodic transactions Without it, once-per-year periodic transactions always occur on 1st Jan. --- hledger-lib/Hledger/Data/AutoTransaction.hs | 13 ++- hledger-lib/Hledger/Data/Dates.hs | 93 +++++++++++++++++---- hledger-lib/Hledger/Data/Types.hs | 1 + 3 files changed, 88 insertions(+), 19 deletions(-) diff --git a/hledger-lib/Hledger/Data/AutoTransaction.hs b/hledger-lib/Hledger/Data/AutoTransaction.hs index d9ed91280..4cc903005 100644 --- a/hledger-lib/Hledger/Data/AutoTransaction.hs +++ b/hledger-lib/Hledger/Data/AutoTransaction.hs @@ -136,7 +136,8 @@ renderPostingCommentDates p = p { pcomment = comment' } -- -- Note that new transactions require 'txnTieKnot' post-processing. -- --- >>> mapM_ (putStr . show) $ runPeriodicTransaction (PeriodicTransaction "monthly from 2017/1 to 2017/4" ["hi" `post` usd 1]) nulldatespan +-- >>> let gen str = mapM_ (putStr . show) $ runPeriodicTransaction (PeriodicTransaction str ["hi" `post` usd 1]) nulldatespan +-- >>> gen "monthly from 2017/1 to 2017/4" -- 2017/01/01 -- hi $1.00 -- @@ -146,6 +147,16 @@ renderPostingCommentDates p = p { pcomment = comment' } -- 2017/03/01 -- hi $1.00 -- +-- >>> gen "every Nov 29th from 2017 to 2019" +-- 2016/11/29 +-- hi $1.00 +-- +-- 2017/11/29 +-- hi $1.00 +-- +-- 2018/11/29 +-- hi $1.00 +-- runPeriodicTransaction :: PeriodicTransaction -> (DateSpan -> [Transaction]) runPeriodicTransaction pt = generate where base = nulltransaction { tpostings = ptpostings pt } diff --git a/hledger-lib/Hledger/Data/Dates.hs b/hledger-lib/Hledger/Data/Dates.hs index c454afd28..517ed15b8 100644 --- a/hledger-lib/Hledger/Data/Dates.hs +++ b/hledger-lib/Hledger/Data/Dates.hs @@ -88,6 +88,7 @@ import Data.Time.Clock import Data.Time.LocalTime import Safe (headMay, lastMay, readMay) import Text.Megaparsec.Compat +import Text.Megaparsec.Perm import Text.Printf import Hledger.Data.Types @@ -168,6 +169,10 @@ spansSpan spans = DateSpan (maybe Nothing spanStart $ headMay spans) (maybe Noth -- [DateSpan 2007/12/02-2008/01/01,DateSpan 2008/01/02-2008/02/01,DateSpan 2008/02/02-2008/03/01,DateSpan 2008/03/02-2008/04/01] -- >>> t (DayOfWeek 2) "2011/01/01" "2011/01/15" -- [DateSpan 2010/12/28-2011/01/03,DateSpan 2011/01/04-2011/01/10,DateSpan 2011/01/11-2011/01/17] +-- >>> t (DayOfYear 11 29) "2011/10/01" "2011/10/15" +-- [DateSpan 2010/11/29-2011/11/28] +-- >>> t (DayOfYear 11 29) "2011/12/01" "2012/12/15" +-- [DateSpan 2011/11/29-2012/11/28,DateSpan 2012/11/29-2013/11/28] -- splitSpan :: Interval -> DateSpan -> [DateSpan] splitSpan _ (DateSpan Nothing Nothing) = [DateSpan Nothing Nothing] @@ -179,6 +184,7 @@ splitSpan (Quarters n) s = splitspan startofquarter (applyN n nextquarter) s splitSpan (Years n) s = splitspan startofyear (applyN n nextyear) s splitSpan (DayOfMonth n) s = splitspan (nthdayofmonthcontaining n) (applyN (n-1) nextday . nextmonth) s splitSpan (DayOfWeek n) s = splitspan (nthdayofweekcontaining n) (applyN (n-1) nextday . nextweek) s +splitSpan (DayOfYear m n) s= splitspan (nthdayofyearcontaining m n) (applyN (n-1) nextday . applyN (m-1) nextmonth . nextyear) s -- splitSpan (WeekOfYear n) s = splitspan startofweek (applyN n nextweek) s -- splitSpan (MonthOfYear n) s = splitspan startofmonth (applyN n nextmonth) s -- splitSpan (QuarterOfYear n) s = splitspan startofquarter (applyN n nextquarter) s @@ -461,6 +467,31 @@ prevyear = startofyear . addGregorianYearsClip (-1) nextyear = startofyear . addGregorianYearsClip 1 startofyear day = fromGregorian y 1 1 where (y,_,_) = toGregorian day +-- | For given date d find year-long interval that starts on given MM/DD of year +-- and covers it. +-- +-- Examples: lets take 2017-11-22. Year-long intervals covering it that +-- starts before Nov 22 will start in 2017. However +-- intervals that start after Nov 23rd should start in 2016: +-- >>> let wed22nd = parsedate "2017-11-22" +-- >>> nthdayofyearcontaining 11 21 wed22nd +-- 2017-11-21 +-- >>> nthdayofyearcontaining 11 22 wed22nd +-- 2017-11-22 +-- >>> nthdayofyearcontaining 11 23 wed22nd +-- 2016-11-23 +-- >>> nthdayofyearcontaining 12 02 wed22nd +-- 2016-12-02 +-- >>> nthdayofyearcontaining 12 31 wed22nd +-- 2016-12-31 +-- >>> nthdayofyearcontaining 1 1 wed22nd +-- 2017-01-01 +nthdayofyearcontaining m n d | mmddOfSameYear <= d = mmddOfSameYear + | otherwise = mmddOfPrevYear + where mmddOfSameYear = addDays (fromIntegral n-1) $ applyN (m-1) nextmonth s + mmddOfPrevYear = addDays (fromIntegral n-1) $ applyN (m-1) nextmonth $ prevyear s + s = startofyear d + -- | For given date d find month-long interval that starts on nth day of month -- and covers it. -- @@ -484,7 +515,6 @@ nthdayofmonthcontaining n d | nthOfSameMonth <= d = nthOfSameMonth nthOfPrevMonth = addDays (fromIntegral n-1) $ prevmonth s s = startofmonth d - -- | For given date d find week-long interval that starts on nth day of week -- and covers it. -- @@ -712,7 +742,7 @@ lastthisnextthing = do return ("", T.unpack r, T.unpack p) -- | --- >>> let p s = parsewith (periodexpr (parsedate "2008/11/26")) (T.toLower s) :: Either (ParseError Char MPErr) (Interval, DateSpan) +-- >>> let p s = parsewith (periodexpr (parsedate "2008/11/26") <* eof) (T.toLower s) :: Either (ParseError Char MPErr) (Interval, DateSpan) -- >>> p "from Aug to Oct" -- Right (NoInterval,DateSpan 2008/08/01-2008/09/30) -- >>> p "aug to oct" @@ -723,6 +753,22 @@ lastthisnextthing = do -- Right (Days 1,DateSpan 2008/08/01-) -- >>> p "every week to 2009" -- Right (Weeks 1,DateSpan -2008/12/31) +-- >>> p "every 2nd day of month" +-- Right (DayOfMonth 2,DateSpan -) +-- >>> p "every 2nd day" +-- Right (DayOfMonth 2,DateSpan -) +-- >>> p "every 2nd day 2009-" +-- Right (DayOfMonth 2,DateSpan 2009/01/01-) +-- >>> p "every 29th Nov" +-- Right (DayOfYear 11 29,DateSpan -) +-- >>> p "every 29th nov -2009" +-- Right (DayOfYear 11 29,DateSpan -2008/12/31) +-- >>> p "every nov 29th" +-- Right (DayOfYear 11 29,DateSpan -) +-- >>> p "every Nov 29th 2009-" +-- Right (DayOfYear 11 29,DateSpan 2009/01/01-) +-- >>> p "every 11/29 from 2009" +-- Right (DayOfYear 11 29,DateSpan 2009/01/01-) periodexpr :: Day -> SimpleTextParser (Interval, DateSpan) periodexpr rdate = choice $ map try [ intervalanddateperiodexpr rdate, @@ -765,31 +811,42 @@ reportinginterval = choice' [ return $ Months 2, do string "every" many spacenonewline - n <- fmap read $ some digitChar - thsuffix + n <- nth many spacenonewline string "day" - many spacenonewline - string "of" - many spacenonewline - string "week" + of_ "week" return $ DayOfWeek n, do string "every" many spacenonewline - n <- fmap read $ some digitChar - thsuffix + n <- nth many spacenonewline string "day" - optional $ do - many spacenonewline - string "of" - many spacenonewline - string "month" - return $ DayOfMonth n + optOf_ "month" + return $ DayOfMonth n, + do string "every" + many spacenonewline + let mnth = choice' [month, mon] >>= \(_,m,_) -> return (read m) + d_o_y <- makePermParser $ DayOfYear <$$> (mnth <* many spacenonewline) <||> (nth <* many spacenonewline) + optOf_ "year" + return d_o_y, + do string "every" + many spacenonewline + ("",m,d) <- md + optOf_ "year" + return $ DayOfYear (read m) (read d) ] where - - thsuffix = choice' $ map string ["st","nd","rd","th"] + of_ period = do + many spacenonewline + string "of" + many spacenonewline + string period + + optOf_ period = optional $ try $ of_ period + + nth = do n <- some digitChar + choice' $ map string ["st","nd","rd","th"] + return $ read n -- Parse any of several variants of a basic interval, eg "daily", "every day", "every N days". tryinterval :: String -> String -> (Int -> Interval) -> SimpleTextParser Interval diff --git a/hledger-lib/Hledger/Data/Types.hs b/hledger-lib/Hledger/Data/Types.hs index 77d22e2ea..cbf6424f2 100644 --- a/hledger-lib/Hledger/Data/Types.hs +++ b/hledger-lib/Hledger/Data/Types.hs @@ -90,6 +90,7 @@ data Interval = | Years Int | DayOfMonth Int | DayOfWeek Int + | DayOfYear Int Int -- Month, Day -- WeekOfYear Int -- MonthOfYear Int -- QuarterOfYear Int From 993e3f2b676cf9672d96dd5c23e6d8e15539c1a4 Mon Sep 17 00:00:00 2001 From: Dmitry Astapov Date: Fri, 24 Nov 2017 23:52:34 +0000 Subject: [PATCH 04/13] lib: support "every 2nd Thursday of month" in period expressions Useful for periodic transactions. --- hledger-lib/Hledger/Data/Dates.hs | 55 +++++++++++++++++++++++++++++-- hledger-lib/Hledger/Data/Types.hs | 1 + 2 files changed, 53 insertions(+), 3 deletions(-) diff --git a/hledger-lib/Hledger/Data/Dates.hs b/hledger-lib/Hledger/Data/Dates.hs index 517ed15b8..8b1bf5137 100644 --- a/hledger-lib/Hledger/Data/Dates.hs +++ b/hledger-lib/Hledger/Data/Dates.hs @@ -167,6 +167,8 @@ spansSpan spans = DateSpan (maybe Nothing spanStart $ headMay spans) (maybe Noth -- [DateSpan 2007/12/31-2008/01/13,DateSpan 2008/01/14-2008/01/27] -- >>> t (DayOfMonth 2) "2008/01/01" "2008/04/01" -- [DateSpan 2007/12/02-2008/01/01,DateSpan 2008/01/02-2008/02/01,DateSpan 2008/02/02-2008/03/01,DateSpan 2008/03/02-2008/04/01] +-- >>> t (WeekdayOfMonth 2 4) "2011/01/01" "2011/02/15" +-- [DateSpan 2010/12/09-2011/01/12,DateSpan 2011/01/13-2011/02/09,DateSpan 2011/02/10-2011/03/09] -- >>> t (DayOfWeek 2) "2011/01/01" "2011/01/15" -- [DateSpan 2010/12/28-2011/01/03,DateSpan 2011/01/04-2011/01/10,DateSpan 2011/01/11-2011/01/17] -- >>> t (DayOfYear 11 29) "2011/10/01" "2011/10/15" @@ -183,6 +185,7 @@ splitSpan (Months n) s = splitspan startofmonth (applyN n nextmonth) s splitSpan (Quarters n) s = splitspan startofquarter (applyN n nextquarter) s splitSpan (Years n) s = splitspan startofyear (applyN n nextyear) s splitSpan (DayOfMonth n) s = splitspan (nthdayofmonthcontaining n) (applyN (n-1) nextday . nextmonth) s +splitSpan (WeekdayOfMonth n wd) s = splitspan (nthweekdayofmonthcontaining n wd) (advancetonthweekday n wd . nextmonth) s splitSpan (DayOfWeek n) s = splitspan (nthdayofweekcontaining n) (applyN (n-1) nextday . nextweek) s splitSpan (DayOfYear m n) s= splitspan (nthdayofyearcontaining m n) (applyN (n-1) nextday . applyN (m-1) nextmonth . nextyear) s -- splitSpan (WeekOfYear n) s = splitspan startofweek (applyN n nextweek) s @@ -538,6 +541,35 @@ nthdayofweekcontaining n d | nthOfSameWeek <= d = nthOfSameWeek nthOfPrevWeek = addDays (fromIntegral n-1) $ prevweek s s = startofweek d +-- | For given date d find month-long interval that starts on nth weekday of month +-- and covers it. +-- +-- Examples: 2017-11-22 is 3rd Wed of Nov. Month-long intervals that cover it and +-- start on 1st-4th Wed will start in Nov. However +-- intervals that start on 4th Thu or Fri or later should start in Oct: +-- >>> let wed22nd = parsedate "2017-11-22" +-- >>> nthweekdayofmonthcontaining 1 3 wed22nd +-- 2017-11-01 +-- >>> nthweekdayofmonthcontaining 3 2 wed22nd +-- 2017-11-21 +-- >>> nthweekdayofmonthcontaining 4 3 wed22nd +-- 2017-11-22 +-- >>> nthweekdayofmonthcontaining 4 4 wed22nd +-- 2017-10-26 +-- >>> nthweekdayofmonthcontaining 4 5 wed22nd +-- 2017-10-27 +nthweekdayofmonthcontaining n wd d | nthWeekdaySameMonth <= d = nthWeekdaySameMonth + | otherwise = nthWeekdayPrevMonth + where nthWeekdaySameMonth = advancetonthweekday n wd $ startofmonth d + nthWeekdayPrevMonth = advancetonthweekday n wd $ prevmonth d + +-- | Advance to nth weekday wd after given start day s +advancetonthweekday n wd s = addWeeks (n-1) . firstMatch (>=s) . iterate (addWeeks 1) $ firstweekday s + where + addWeeks k = addDays (7 * fromIntegral k) + firstMatch p = head . dropWhile (not . p) + firstweekday = addDays (fromIntegral wd-1) . startofweek + ---------------------------------------------------------------------- -- parsing @@ -698,8 +730,8 @@ md = do months = ["january","february","march","april","may","june", "july","august","september","october","november","december"] monthabbrevs = ["jan","feb","mar","apr","may","jun","jul","aug","sep","oct","nov","dec"] --- weekdays = ["monday","tuesday","wednesday","thursday","friday","saturday","sunday"] --- weekdayabbrevs = ["mon","tue","wed","thu","fri","sat","sun"] +weekdays = ["monday","tuesday","wednesday","thursday","friday","saturday","sunday"] +weekdayabbrevs = ["mon","tue","wed","thu","fri","sat","sun"] monthIndex t = maybe 0 (+1) $ t `elemIndex` months monIndex t = maybe 0 (+1) $ t `elemIndex` monthabbrevs @@ -716,6 +748,12 @@ mon = do let i = monIndex m return ("",show i,"") +weekday :: SimpleTextParser Int +weekday = do + wday <- choice . map string' $ weekdays ++ weekdayabbrevs + let i = head . catMaybes $ [wday `elemIndex` weekdays, wday `elemIndex` weekdayabbrevs] + return (i+1) + today,yesterday,tomorrow :: SimpleTextParser SmartDate today = string "today" >> return ("","","today") yesterday = string "yesterday" >> return ("","","yesterday") @@ -769,6 +807,10 @@ lastthisnextthing = do -- Right (DayOfYear 11 29,DateSpan 2009/01/01-) -- >>> p "every 11/29 from 2009" -- Right (DayOfYear 11 29,DateSpan 2009/01/01-) +-- >>> p "every 2nd Thursday of month to 2009" +-- Right (WeekdayOfMonth 2 4,DateSpan -2008/12/31) +-- >>> p "every 1st monday of month to 2009" +-- Right (WeekdayOfMonth 1 1,DateSpan -2008/12/31) periodexpr :: Day -> SimpleTextParser (Interval, DateSpan) periodexpr rdate = choice $ map try [ intervalanddateperiodexpr rdate, @@ -833,7 +875,14 @@ reportinginterval = choice' [ many spacenonewline ("",m,d) <- md optOf_ "year" - return $ DayOfYear (read m) (read d) + return $ DayOfYear (read m) (read d), + do string "every" + many spacenonewline + n <- nth + many spacenonewline + wd <- weekday + optOf_ "month" + return $ WeekdayOfMonth n wd ] where of_ period = do diff --git a/hledger-lib/Hledger/Data/Types.hs b/hledger-lib/Hledger/Data/Types.hs index cbf6424f2..34e5999a0 100644 --- a/hledger-lib/Hledger/Data/Types.hs +++ b/hledger-lib/Hledger/Data/Types.hs @@ -89,6 +89,7 @@ data Interval = | Quarters Int | Years Int | DayOfMonth Int + | WeekdayOfMonth Int Int | DayOfWeek Int | DayOfYear Int Int -- Month, Day -- WeekOfYear Int From 950891b55bfac9d90ccab761f4ba135b191fec9e Mon Sep 17 00:00:00 2001 From: Dmitry Astapov Date: Fri, 24 Nov 2017 23:02:55 +0000 Subject: [PATCH 05/13] lib: support "every " A shorter spelling for "every th day of week". --- hledger-lib/Hledger/Data/Dates.hs | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/hledger-lib/Hledger/Data/Dates.hs b/hledger-lib/Hledger/Data/Dates.hs index 8b1bf5137..8e6c29d19 100644 --- a/hledger-lib/Hledger/Data/Dates.hs +++ b/hledger-lib/Hledger/Data/Dates.hs @@ -811,6 +811,12 @@ lastthisnextthing = do -- Right (WeekdayOfMonth 2 4,DateSpan -2008/12/31) -- >>> p "every 1st monday of month to 2009" -- Right (WeekdayOfMonth 1 1,DateSpan -2008/12/31) +-- >>> p "every tue" +-- Right (DayOfWeek 2,DateSpan -) +-- >>> p "every 2nd day of week" +-- Right (DayOfWeek 2,DateSpan -) +-- >>> p "every 2nd day 2009-" +-- Right (DayOfMonth 2,DateSpan 2009/01/01-) periodexpr :: Day -> SimpleTextParser (Interval, DateSpan) periodexpr rdate = choice $ map try [ intervalanddateperiodexpr rdate, @@ -858,6 +864,10 @@ reportinginterval = choice' [ string "day" of_ "week" return $ DayOfWeek n, + do string "every" + many spacenonewline + n <- weekday + return $ DayOfWeek n, do string "every" many spacenonewline n <- nth From 0dfffed52c799f2fa68991cb29d59486ed9bcd84 Mon Sep 17 00:00:00 2001 From: Dmitry Astapov Date: Sat, 25 Nov 2017 00:33:09 +0000 Subject: [PATCH 06/13] doc: expand documentation for period expressions Document "first day of period" behavior. Document new period expressions DayOfYear and WeekdayOfMonth. --- hledger/doc/options.m4.md | 50 ++++++++++++++++++++++++++++++++++----- 1 file changed, 44 insertions(+), 6 deletions(-) diff --git a/hledger/doc/options.m4.md b/hledger/doc/options.m4.md index bd7add548..3bd6bc242 100644 --- a/hledger/doc/options.m4.md +++ b/hledger/doc/options.m4.md @@ -239,21 +239,59 @@ Examples: `-p "quarterly"` ------------------------------------------ +Note that `weekly`, `monthly`, `quarterly` and `yearly` intervals will +always start on the first day on week, month, quarter or year +accordingly, and will end on the last day of same period, even if +associated period expression specifies different explicit start and end date. + +For example: +------------------------------------------ +`-p "weekly from 2009/1/1 to 2009/4/1"` -- starts on 2008/12/29, closest preceeding Monday +`-p "monthly in 2008/11/25"` -- starts on 2018/11/01 +`-p "quarterly from 2009-05-05 to 2009-06-01"` - starts on 2009/04/01, ends on 2009/06/30, which are first and last days of Q2 2009 +`-p "yearly from 2009-12-29"` - starts on 2009/01/01, first day of 2009 +------------------------------------------ + The following more complex report intervals are also supported: `biweekly`, `bimonthly`, -`every N days|weeks|months|quarters|years`, -`every Nth day [of month]`, -`every Nth day of week`. +`every day|week|month|quarter|year`, +`every N days|weeks|months|quarters|years`. + + +All of these will start on the first day of the requested period and end on the last one, as described above. Examples: ------------------------------------------ -`-p "bimonthly from 2008"` -`-p "every 2 weeks"` -`-p "every 5 days from 1/3"` +`-p "bimonthly from 2008"` -- periods will have boundaries on 2008/01/01, 2008/03/01, ... +`-p "every 2 weeks"` -- starts on closest preceeding Monday +`-p "every 5 month from 2009/03"` -- periods will have boundaries on 2009/03/01, 2009/08/01, ... ------------------------------------------ +If you want intervals that start on arbitrary day of your choosing and span a week, month or year, you need to use any of the following: + +`every Nth day of week`, +`every `, +`every Nth day [of month]`, +`every Nth weekday [of month]`, +`every MM/DD [of year]`, +`every Nth MMM [of year]`, +`every MMM Nth [of year]`. + +Examples: + +------------------------------------------ +`-p "every 2nd day of week"` -- periods will go from Tue to Tue +`-p "every Tue"` -- same +`-p "every 15th day"` -- period boundaries will be on 15th of each month +`-p "every 2nd Monday"` -- period boundaries will be on second Monday of each month +`-p "every 11/05"` -- yearly periods with boundaries on 5th of Nov +`-p "every 5th Nov"` -- same +`-p "every Nov 5th"` -- same +------------------------------------------ + + Show historical balances at end of 15th each month (N is exclusive end date): `hledger balance -H -p "every 16th day"` From 597e9c47c9a6300cbe9ea73e0a8cdf079aad79d5 Mon Sep 17 00:00:00 2001 From: Dmitry Astapov Date: Fri, 24 Nov 2017 22:44:10 +0000 Subject: [PATCH 07/13] lib: more periodic transaction tests Some of these demonstrate that runPeriodicTransaction could generate transactions ouside of requested DateSpan. This happens because runPeriodicTransaction uses splitSpan internally, and splitSpan always generates dateSpans that fully cover original DateSpan, extending beyound left/right boundary if necessary. This is ok if transactions are generated for budgeting purpose, but during forecasting care should be taken to check that all generated transactions are happening past the end of the real journal. --- hledger-lib/Hledger/Data/AutoTransaction.hs | 64 ++++++++++++++++++++- hledger-lib/Hledger/Data/Dates.hs | 7 ++- 2 files changed, 67 insertions(+), 4 deletions(-) diff --git a/hledger-lib/Hledger/Data/AutoTransaction.hs b/hledger-lib/Hledger/Data/AutoTransaction.hs index 4cc903005..87d5de4e1 100644 --- a/hledger-lib/Hledger/Data/AutoTransaction.hs +++ b/hledger-lib/Hledger/Data/AutoTransaction.hs @@ -147,7 +147,69 @@ renderPostingCommentDates p = p { pcomment = comment' } -- 2017/03/01 -- hi $1.00 -- --- >>> gen "every Nov 29th from 2017 to 2019" +-- >>> gen "monthly from 2017/1 to 2017/5" +-- 2017/01/01 +-- hi $1.00 +-- +-- 2017/02/01 +-- hi $1.00 +-- +-- 2017/03/01 +-- hi $1.00 +-- +-- 2017/04/01 +-- hi $1.00 +-- +-- >>> gen "every 2nd day of month from 2017/02 to 2017/04" +-- 2017/01/02 +-- hi $1.00 +-- +-- 2017/02/02 +-- hi $1.00 +-- +-- 2017/03/02 +-- hi $1.00 +-- +-- >>> gen "monthly from 2017/1 to 2017/4" +-- 2017/01/01 +-- hi $1.00 +-- +-- 2017/02/01 +-- hi $1.00 +-- +-- 2017/03/01 +-- hi $1.00 +-- +-- >>> gen "every 30th day of month from 2017/1 to 2017/5" +-- 2016/12/30 +-- hi $1.00 +-- +-- 2017/01/30 +-- hi $1.00 +-- +-- 2017/02/28 +-- hi $1.00 +-- +-- 2017/03/30 +-- hi $1.00 +-- +-- 2017/04/30 +-- hi $1.00 +-- +-- >>> gen "every 2nd Thursday of month from 2017/1 to 2017/4" +-- 2016/12/08 +-- hi $1.00 +-- +-- 2017/01/12 +-- hi $1.00 +-- +-- 2017/02/09 +-- hi $1.00 +-- +-- 2017/03/09 +-- hi $1.00 +-- +-- >>> gen "every nov 29th from 2017 to 2019" -- 2016/11/29 -- hi $1.00 -- diff --git a/hledger-lib/Hledger/Data/Dates.hs b/hledger-lib/Hledger/Data/Dates.hs index 8e6c29d19..14500963a 100644 --- a/hledger-lib/Hledger/Data/Dates.hs +++ b/hledger-lib/Hledger/Data/Dates.hs @@ -184,7 +184,7 @@ splitSpan (Weeks n) s = splitspan startofweek (applyN n nextweek) s splitSpan (Months n) s = splitspan startofmonth (applyN n nextmonth) s splitSpan (Quarters n) s = splitspan startofquarter (applyN n nextquarter) s splitSpan (Years n) s = splitspan startofyear (applyN n nextyear) s -splitSpan (DayOfMonth n) s = splitspan (nthdayofmonthcontaining n) (applyN (n-1) nextday . nextmonth) s +splitSpan (DayOfMonth n) s = splitspan (nthdayofmonthcontaining n) (nthdayofmonth n . nextmonth) s splitSpan (WeekdayOfMonth n wd) s = splitspan (nthweekdayofmonthcontaining n wd) (advancetonthweekday n wd . nextmonth) s splitSpan (DayOfWeek n) s = splitspan (nthdayofweekcontaining n) (applyN (n-1) nextday . nextweek) s splitSpan (DayOfYear m n) s= splitspan (nthdayofyearcontaining m n) (applyN (n-1) nextday . applyN (m-1) nextmonth . nextyear) s @@ -456,6 +456,7 @@ thismonth = startofmonth prevmonth = startofmonth . addGregorianMonthsClip (-1) nextmonth = startofmonth . addGregorianMonthsClip 1 startofmonth day = fromGregorian y m 1 where (y,m,_) = toGregorian day +nthdayofmonth d day = fromGregorian y m d where (y,m,_) = toGregorian day thisquarter = startofquarter prevquarter = startofquarter . addGregorianMonthsClip (-3) @@ -514,8 +515,8 @@ nthdayofyearcontaining m n d | mmddOfSameYear <= d = mmddOfSameYear -- 2017-10-30 nthdayofmonthcontaining n d | nthOfSameMonth <= d = nthOfSameMonth | otherwise = nthOfPrevMonth - where nthOfSameMonth = addDays (fromIntegral n-1) s - nthOfPrevMonth = addDays (fromIntegral n-1) $ prevmonth s + where nthOfSameMonth = nthdayofmonth n s + nthOfPrevMonth = nthdayofmonth n $ prevmonth s s = startofmonth d -- | For given date d find week-long interval that starts on nth day of week From 50b4d76ce98aa62232d60dcdb86b970368358cd0 Mon Sep 17 00:00:00 2001 From: Dmitry Astapov Date: Wed, 22 Nov 2017 21:00:57 +0000 Subject: [PATCH 08/13] lib: runPeriodicTransaction's start date must line up with interval This is very helpful for periodic transactions, because in budget mode you need to ensure that no periodic transactions extend past the end of the journal, and in forecast mode you need to make sure that all periodic transactions are strictly after the end of the journal. --- hledger-lib/Hledger/Data/AutoTransaction.hs | 23 ++++++++++++++++++++- 1 file changed, 22 insertions(+), 1 deletion(-) diff --git a/hledger-lib/Hledger/Data/AutoTransaction.hs b/hledger-lib/Hledger/Data/AutoTransaction.hs index 87d5de4e1..67c6abb8f 100644 --- a/hledger-lib/Hledger/Data/AutoTransaction.hs +++ b/hledger-lib/Hledger/Data/AutoTransaction.hs @@ -219,6 +219,14 @@ renderPostingCommentDates p = p { pcomment = comment' } -- 2018/11/29 -- hi $1.00 -- +-- >>> gen "weekly from 2017" +-- *** Exception: Unable to generate transactions according to "weekly from 2017" as 2017-01-01 is not a first day of the week +-- >>> gen "monthly from 2017/5/4" +-- *** Exception: Unable to generate transactions according to "monthly from 2017/5/4" as 2017-05-04 is not a first day of the month +-- >>> gen "every quarter from 2017/1/2" +-- *** Exception: Unable to generate transactions according to "every quarter from 2017/1/2" as 2017-01-02 is not a first day of the quarter +-- >>> gen "yearly from 2017/1/14" +-- *** Exception: Unable to generate transactions according to "yearly from 2017/1/14" as 2017-01-14 is not a first day of the year runPeriodicTransaction :: PeriodicTransaction -> (DateSpan -> [Transaction]) runPeriodicTransaction pt = generate where base = nulltransaction { tpostings = ptpostings pt } @@ -227,5 +235,18 @@ runPeriodicTransaction pt = generate where (interval, effectspan) = case parsePeriodExpr errCurrent periodExpr of Left e -> error' $ "Failed to parse " ++ show (T.unpack periodExpr) ++ ": " ++ showDateParseError e - Right x -> x + Right x -> checkProperStartDate x generate jspan = [base {tdate=date} | span <- interval `splitSpan` spanIntersect effectspan jspan, let Just date = spanStart span] + checkProperStartDate (i,s) = + case (i,spanStart s) of + (Weeks _, Just d) -> checkStart d "week" + (Months _, Just d) -> checkStart d "month" + (Quarters _, Just d) -> checkStart d "quarter" + (Years _, Just d) -> checkStart d "year" + _ -> (i,s) + where + checkStart d x = + let firstDate = fixSmartDate d ("","this",x) + in + if d == firstDate then (i,s) + else error' $ "Unable to generate transactions according to "++(show periodExpr)++" as "++(show d)++" is not a first day of the "++x From f101d5b51549c7c194228855fdce2b443fcb780a Mon Sep 17 00:00:00 2001 From: Dmitry Astapov Date: Sat, 18 Nov 2017 00:40:10 +0000 Subject: [PATCH 09/13] cli: --forecast adds periodic transactions to reports Ledger-style periodic transactions, previously supported only by hledger-budget, have landed as a first-class feature. The --forecast flag activates them, so that any transactions they generate are included in reports. --- bin/hledger-budget.hs | 1 - hledger-lib/Hledger/Data.hs | 2 + hledger-lib/Hledger/Reports/ReportOptions.hs | 3 + hledger/Hledger/Cli/CliOptions.hs | 1 + hledger/Hledger/Cli/Utils.hs | 26 ++++- tests/budget/forecast.test | 102 +++++++++++++++++++ 6 files changed, 133 insertions(+), 2 deletions(-) create mode 100644 tests/budget/forecast.test diff --git a/bin/hledger-budget.hs b/bin/hledger-budget.hs index cf599787a..6fe4251db 100755 --- a/bin/hledger-budget.hs +++ b/bin/hledger-budget.hs @@ -13,7 +13,6 @@ import Data.List import Data.String.Here import System.Console.CmdArgs import Hledger.Cli -import Hledger.Data.AutoTransaction -- hledger-budget REPORT-COMMAND [--no-offset] [--no-buckets] [OPTIONS...] diff --git a/hledger-lib/Hledger/Data.hs b/hledger-lib/Hledger/Data.hs index 4f3e83696..cfd4bef73 100644 --- a/hledger-lib/Hledger/Data.hs +++ b/hledger-lib/Hledger/Data.hs @@ -22,6 +22,7 @@ module Hledger.Data ( module Hledger.Data.StringFormat, module Hledger.Data.Timeclock, module Hledger.Data.Transaction, + module Hledger.Data.AutoTransaction, module Hledger.Data.Types, tests_Hledger_Data ) @@ -42,6 +43,7 @@ import Hledger.Data.RawOptions import Hledger.Data.StringFormat import Hledger.Data.Timeclock import Hledger.Data.Transaction +import Hledger.Data.AutoTransaction import Hledger.Data.Types tests_Hledger_Data :: Test diff --git a/hledger-lib/Hledger/Reports/ReportOptions.hs b/hledger-lib/Hledger/Reports/ReportOptions.hs index cd02e35c3..e76c2e497 100644 --- a/hledger-lib/Hledger/Reports/ReportOptions.hs +++ b/hledger-lib/Hledger/Reports/ReportOptions.hs @@ -104,6 +104,7 @@ data ReportOpts = ReportOpts { -- eg in the income section of an income statement, this helps --sort-amount know -- how to sort negative numbers. ,color_ :: Bool + ,forecast_ :: Bool } deriving (Show, Data, Typeable) instance Default ReportOpts where def = defreportopts @@ -134,6 +135,7 @@ defreportopts = ReportOpts def def def + def rawOptsToReportOpts :: RawOpts -> IO ReportOpts rawOptsToReportOpts rawopts = checkReportOpts <$> do @@ -164,6 +166,7 @@ rawOptsToReportOpts rawopts = checkReportOpts <$> do ,sort_amount_ = boolopt "sort-amount" rawopts' ,pretty_tables_ = boolopt "pretty-tables" rawopts' ,color_ = color + ,forecast_ = boolopt "forecast" rawopts' } -- | Do extra validation of raw option values, raising an error if there's a problem. diff --git a/hledger/Hledger/Cli/CliOptions.hs b/hledger/Hledger/Cli/CliOptions.hs index 35c6537c3..11292f55f 100644 --- a/hledger/Hledger/Cli/CliOptions.hs +++ b/hledger/Hledger/Cli/CliOptions.hs @@ -155,6 +155,7 @@ reportflags = [ ,flagNone ["empty","E"] (setboolopt "empty") "show items with zero amount, normally hidden" ,flagNone ["cost","B"] (setboolopt "cost") "convert amounts to their cost at transaction time (using the transaction price, if any)" ,flagNone ["value","V"] (setboolopt "value") "convert amounts to their market value on the report end date (using the most recent applicable market price, if any)" + ,flagNone ["forecast"] (\opts -> setboolopt "forecast" opts) "generate forecast transactions" ] -- | Common output-related flags: --output-file, --output-format... diff --git a/hledger/Hledger/Cli/Utils.hs b/hledger/Hledger/Cli/Utils.hs index 1934b3f71..813a6d38b 100644 --- a/hledger/Hledger/Cli/Utils.hs +++ b/hledger/Hledger/Cli/Utils.hs @@ -31,7 +31,7 @@ import Data.List import Data.Maybe import qualified Data.Text as T import qualified Data.Text.IO as T -import Data.Time (Day) +import Data.Time (Day, addDays) import Data.Word import Numeric import Safe (readMay) @@ -70,6 +70,7 @@ withJournalDo opts cmd = do . anonymiseByOpts opts . journalApplyAliases (aliasesFromOpts opts) <=< journalApplyValue (reportopts_ opts) + <=< journalAddForecast opts either error' f ej -- | Apply the pivot transformation on a journal, if option is present. @@ -117,6 +118,29 @@ journalApplyValue ropts j = do = id return $ convert j +-- | Run PeriodicTransactions from journal from today or journal end to requested end day. +-- Add generated transactions to the journal +journalAddForecast :: CliOpts -> Journal -> IO Journal +journalAddForecast opts j = do + today <- getCurrentDay + -- Create forecast starting from end of journal + 1 day, and until the end of requested reporting period + -- If end is not provided, do 180 days of forecast. + -- Note that jdatespan already returns last day + 1 + let startDate = fromMaybe today $ spanEnd (jdatespan j) + endDate = fromMaybe (addDays 180 today) $ periodEnd (period_ ropts) + dates = DateSpan (Just startDate) (Just endDate) + withForecast = [makeForecast t | pt <- jperiodictxns j, t <- runPeriodicTransaction pt dates, spanContainsDate dates (tdate t) ] ++ (jtxns j) + makeForecast t = txnTieKnot $ t { tdescription = T.pack "Forecast transaction" } + ropts = reportopts_ opts + if forecast_ ropts + then return $ journalBalanceTransactions' opts j { jtxns = withForecast } + else return j + where + journalBalanceTransactions' opts j = + let assrt = not . ignore_assertions_ $ inputopts_ opts + in + either error' id $ journalBalanceTransactions assrt j + -- | Write some output to stdout or to a file selected by --output-file. -- If the file exists it will be overwritten. writeOutput :: CliOpts -> String -> IO () diff --git a/tests/budget/forecast.test b/tests/budget/forecast.test new file mode 100644 index 000000000..0e9a1904f --- /dev/null +++ b/tests/budget/forecast.test @@ -0,0 +1,102 @@ +# Test --forecast switch +hledger bal -M -b 2016-11 -e 2017-02 -f - --forecast +<<< +2016/12/31 + expenses:housing $600 + assets:cash + +~ monthly from 2016/1 + income $-1000 + expenses:food $20 + expenses:leisure $15 + expenses:grocery $30 + assets:cash +>>> +Balance changes in 2016/12/01-2017/01/31: + + || 2016/12 2017/01 +==================++================== + assets:cash || $-600 $935 + expenses:food || 0 $20 + expenses:grocery || 0 $30 + expenses:housing || $600 0 + expenses:leisure || 0 $15 + income || 0 $-1000 +------------------++------------------ + || 0 0 + +>>>2 +>>>=0 + + +hledger print -b 2016-11 -e 2017-02 -f - --forecast +<<< +2016/12/31 + expenses:housing $600 + assets:cash + +~ monthly from 2016/1 + income $-1000 + expenses:food $20 + expenses:leisure $15 + expenses:grocery $30 + assets:cash +>>> +2016/12/31 + expenses:housing $600 + assets:cash + +2017/01/01 Forecast transaction + income $-1000 + expenses:food $20 + expenses:leisure $15 + expenses:grocery $30 + assets:cash + +>>>2 +>>>=0 + + +hledger register -b 2016-11 -e 2017-02 -f - --forecast +<<< +2016/12/31 + expenses:housing $600 + assets:cash + +~ monthly from 2016/1 + income $-1000 + expenses:food $20 + expenses:leisure $15 + expenses:grocery $30 + assets:cash +>>> +2016/12/31 expenses:housing $600 $600 + assets:cash $-600 0 +2017/01/01 Forecast transact.. income $-1000 $-1000 + expenses:food $20 $-980 + expenses:leisure $15 $-965 + expenses:grocery $30 $-935 + assets:cash $935 0 +>>>2 +>>>=0 + +# Check that --forecast generates transactions only after last transaction in journal +hledger register -b 2015-12 -e 2017-02 -f - assets:cash --forecast +<<< +2016/01/01 + expenses:fun $10 ; more fireworks + assets:cash + +2016/12/02 + expenses:housing $600 + assets:cash + +~ yearly from 2016 + income $-10000 ; bonus + assets:cash +>>> +2016/01/01 assets:cash $-10 $-10 +2016/12/02 assets:cash $-600 $-610 +2017/01/01 Forecast transact.. assets:cash $10000 $9390 +>>>2 +>>>=0 From 23f3da4e929a1a0485544d06b6f94493550b2e3e Mon Sep 17 00:00:00 2001 From: Dmitry Astapov Date: Sat, 18 Nov 2017 00:40:10 +0000 Subject: [PATCH 10/13] cli: --auto adds automated postings to reports Ledger-style automated postings, previously supported only by hledger-budget, have landed as a first-class feature. The --auto flag activates them, so that any postings they generate are included in reports. --- hledger-lib/Hledger/Reports/ReportOptions.hs | 3 + hledger/Hledger/Cli/CliOptions.hs | 3 +- hledger/Hledger/Cli/Utils.hs | 11 +++ tests/budget/auto.test | 78 ++++++++++++++++++++ 4 files changed, 94 insertions(+), 1 deletion(-) create mode 100644 tests/budget/auto.test diff --git a/hledger-lib/Hledger/Reports/ReportOptions.hs b/hledger-lib/Hledger/Reports/ReportOptions.hs index e76c2e497..8f18c535d 100644 --- a/hledger-lib/Hledger/Reports/ReportOptions.hs +++ b/hledger-lib/Hledger/Reports/ReportOptions.hs @@ -105,6 +105,7 @@ data ReportOpts = ReportOpts { -- how to sort negative numbers. ,color_ :: Bool ,forecast_ :: Bool + ,auto_ :: Bool } deriving (Show, Data, Typeable) instance Default ReportOpts where def = defreportopts @@ -136,6 +137,7 @@ defreportopts = ReportOpts def def def + def rawOptsToReportOpts :: RawOpts -> IO ReportOpts rawOptsToReportOpts rawopts = checkReportOpts <$> do @@ -167,6 +169,7 @@ rawOptsToReportOpts rawopts = checkReportOpts <$> do ,pretty_tables_ = boolopt "pretty-tables" rawopts' ,color_ = color ,forecast_ = boolopt "forecast" rawopts' + ,auto_ = boolopt "auto" rawopts' } -- | Do extra validation of raw option values, raising an error if there's a problem. diff --git a/hledger/Hledger/Cli/CliOptions.hs b/hledger/Hledger/Cli/CliOptions.hs index 11292f55f..fb05d0687 100644 --- a/hledger/Hledger/Cli/CliOptions.hs +++ b/hledger/Hledger/Cli/CliOptions.hs @@ -155,7 +155,8 @@ reportflags = [ ,flagNone ["empty","E"] (setboolopt "empty") "show items with zero amount, normally hidden" ,flagNone ["cost","B"] (setboolopt "cost") "convert amounts to their cost at transaction time (using the transaction price, if any)" ,flagNone ["value","V"] (setboolopt "value") "convert amounts to their market value on the report end date (using the most recent applicable market price, if any)" - ,flagNone ["forecast"] (\opts -> setboolopt "forecast" opts) "generate forecast transactions" + ,flagNone ["forecast"] (setboolopt "forecast") "generate forecast transactions" + ,flagNone ["auto"] (setboolopt "auto") "generate automated postings" ] -- | Common output-related flags: --output-file, --output-format... diff --git a/hledger/Hledger/Cli/Utils.hs b/hledger/Hledger/Cli/Utils.hs index 813a6d38b..9a272ad04 100644 --- a/hledger/Hledger/Cli/Utils.hs +++ b/hledger/Hledger/Cli/Utils.hs @@ -54,6 +54,7 @@ import Hledger.Data import Hledger.Read import Hledger.Reports import Hledger.Utils +import Hledger.Query (Query(Any)) -- | Parse the user's specified journal file, maybe apply some transformations @@ -71,6 +72,7 @@ withJournalDo opts cmd = do . journalApplyAliases (aliasesFromOpts opts) <=< journalApplyValue (reportopts_ opts) <=< journalAddForecast opts + . generateAutomaticPostings (reportopts_ opts) either error' f ej -- | Apply the pivot transformation on a journal, if option is present. @@ -141,6 +143,15 @@ journalAddForecast opts j = do in either error' id $ journalBalanceTransactions assrt j +-- | Generate Automatic postings and add them to the current journal. +generateAutomaticPostings :: ReportOpts -> Journal -> Journal +generateAutomaticPostings ropts j = + if auto_ ropts then j { jtxns = map modifier $ jtxns j } else j + where + modifier = foldr (flip (.) . runModifierTransaction') id mtxns + runModifierTransaction' = fmap txnTieKnot . runModifierTransaction Any + mtxns = jmodifiertxns j + -- | Write some output to stdout or to a file selected by --output-file. -- If the file exists it will be overwritten. writeOutput :: CliOpts -> String -> IO () diff --git a/tests/budget/auto.test b/tests/budget/auto.test new file mode 100644 index 000000000..daf1c07ca --- /dev/null +++ b/tests/budget/auto.test @@ -0,0 +1,78 @@ +# Add proportional income tax (from documentation) +hledger print -f- --auto +<<< +2016/1/1 paycheck + income:remuneration $-100 + income:donations $-15 + assets:bank + +2016/1/1 withdraw + assets:cash $20 + assets:bank + += ^income + (liabilities:tax) *.33 ; income tax +>>> +2016/01/01 paycheck + income:remuneration $-100 + income:donations $-15 + assets:bank + (liabilities:tax) $-33 ; income tax + (liabilities:tax) $-5 ; income tax + +2016/01/01 withdraw + assets:cash $20 + assets:bank + +>>>2 +>>>=0 + +hledger register -f- --auto +<<< +2016/1/1 paycheck + income:remuneration $-100 + income:donations $-15 + assets:bank + +2016/1/1 withdraw + assets:cash $20 + assets:bank + += ^income + (liabilities:tax) *.33 ; income tax +>>> +2016/01/01 paycheck income:remuneration $-100 $-100 + income:donations $-15 $-115 + assets:bank $115 0 + (liabilities:tax) $-33 $-33 + (liabilities:tax) $-5 $-38 +2016/01/01 withdraw assets:cash $20 $-18 + assets:bank $-20 $-38 +>>>2 +>>>=0 + +hledger balance -f- --auto +<<< +2016/1/1 paycheck + income:remuneration $-100 + income:donations $-15 + assets:bank + +2016/1/1 withdraw + assets:cash $20 + assets:bank + += ^income + (liabilities:tax) *.33 ; income tax +>>> + $115 assets + $95 bank + $20 cash + $-115 income + $-15 donations + $-100 remuneration + $-38 liabilities:tax +-------------------- + $-38 +>>>2 +>>>=0 From 6ea5da2d9d031935c7d60c5c929e5ddbc83cd1c2 Mon Sep 17 00:00:00 2001 From: Dmitry Astapov Date: Sun, 19 Nov 2017 01:07:08 +0000 Subject: [PATCH 11/13] bal: --budget shows budget performance Budget goals specified with periodic transactions (as with hledger-budget) can now be displayed in balance report (but not in bs/is/cf). --budget shows the target amount and percentage alongside the actual amount, per account and period. Unbudgeted accounts will be hidden, unless --show-unbudgeted is used. Budgeted accounts are displayed folded (depth-clipped) at a depth matching the budget specification. Unbudgeted accounts, if shown, are displayed at their usual depth (in full detail, or according to --depth). --- hledger/Hledger/Cli/Commands/Balance.hs | 117 ++++++++++++++++++++++-- tests/budget/budget.test | 92 +++++++++++++++++++ 2 files changed, 199 insertions(+), 10 deletions(-) create mode 100644 tests/budget/budget.test diff --git a/hledger/Hledger/Cli/Commands/Balance.hs b/hledger/Hledger/Cli/Commands/Balance.hs index 896987f1f..83ca1ac58 100644 --- a/hledger/Hledger/Cli/Commands/Balance.hs +++ b/hledger/Hledger/Cli/Commands/Balance.hs @@ -246,8 +246,9 @@ module Hledger.Cli.Commands.Balance ( ,tests_Hledger_Cli_Commands_Balance ) where -import Data.List (intercalate) +import Data.List (intercalate, nub) import Data.Maybe +import qualified Data.Map as Map -- import Data.Monoid import qualified Data.Text as T import System.Console.CmdArgs.Explicit as C @@ -283,6 +284,8 @@ balancemode = (defCommandMode $ ["balance"] ++ aliases) { -- also accept but don ,flagReq ["format"] (\s opts -> Right $ setopt "format" s opts) "FORMATSTR" "use this custom line format (in simple reports)" ,flagNone ["pretty-tables"] (\opts -> setboolopt "pretty-tables" opts) "use unicode when displaying tables" ,flagNone ["sort-amount","S"] (\opts -> setboolopt "sort-amount" opts) "sort by amount instead of account name" + ,flagNone ["budget"] (setboolopt "budget") "compute budget from periodic transactions and compare real balances to it" + ,flagNone ["show-unbudgeted"] (setboolopt "show-unbudgeted") "show full names of accounts not mentioned in budget" ] ++ outputflags ,groupHidden = [] @@ -293,7 +296,7 @@ balancemode = (defCommandMode $ ["balance"] ++ aliases) { -- also accept but don -- | The balance command, prints a balance report. balance :: CliOpts -> Journal -> IO () -balance opts@CliOpts{reportopts_=ropts} j = do +balance opts@CliOpts{rawopts_=rawopts,reportopts_=ropts} j = do d <- getCurrentDay case lineFormatFromOpts ropts of Left err -> error' $ unlines [err] @@ -319,12 +322,58 @@ balance opts@CliOpts{reportopts_=ropts} j = do "csv" -> \ropts r -> (++ "\n") $ printCSV $ balanceReportAsCsv ropts r _ -> balanceReportAsText writeOutput opts $ render ropts report - _ -> do + + _ | boolopt "budget" rawopts -> do + let budget = budgetJournal opts j + j' = budgetRollUp opts budget j + report = multiBalanceReport ropts (queryFromOpts d ropts) j' + budgetReport = multiBalanceReport ropts (queryFromOpts d ropts) budget + render = case format of + -- XXX: implement csv rendering + "csv" -> (++ "\n") . printCSV . multiBalanceReportAsCsv ropts + _ -> multiBalanceReportWithBudgetAsText ropts budgetReport + writeOutput opts $ render report + + | otherwise -> do let report = multiBalanceReport ropts (queryFromOpts d ropts) j render = case format of - "csv" -> \ropts r -> (++ "\n") $ printCSV $ multiBalanceReportAsCsv ropts r - _ -> multiBalanceReportAsText - writeOutput opts $ render ropts report + "csv" -> (++ "\n") . printCSV . multiBalanceReportAsCsv ropts + _ -> multiBalanceReportAsText ropts + writeOutput opts $ render report + +-- | Re-map account names to closet parent with periodic transaction from budget. +-- Accounts that dont have suitable parent are either remapped to ":topAccount" +-- or left as-is if --show-unbudgeted is provided +budgetRollUp :: CliOpts -> Journal -> Journal -> Journal +budgetRollUp CliOpts{rawopts_=rawopts} budget j = j { jtxns = remapTxn <$> jtxns j } + where + budgetAccounts = nub $ concatMap (map paccount . ptpostings) $ jperiodictxns budget + remapAccount origAcctName = remapAccount' origAcctName + where + remapAccount' acctName + | acctName `elem` budgetAccounts = acctName + | otherwise = + case parentAccountName acctName of + "" | boolopt "show-unbudgeted" rawopts -> origAcctName + | otherwise -> T.append (T.pack ":") acctName + parent -> remapAccount' parent + remapPosting p = p { paccount = remapAccount $ paccount p, porigin = Just . fromMaybe p $ porigin p } + remapTxn = mapPostings (map remapPosting) + mapPostings f t = txnTieKnot $ t { tpostings = f $ tpostings t } + +-- | Generate journal of all periodic transactions in the given journal for the +-- entireity of its history or reporting period, whatever is smaller +budgetJournal :: CliOpts -> Journal -> Journal +budgetJournal opts j = journalBalanceTransactions' opts j { jtxns = budget } + where + dates = spanIntersect (jdatespan j) (periodAsDateSpan $ period_ $ reportopts_ opts) + budget = [makeBudget t | pt <- jperiodictxns j, t <- runPeriodicTransaction pt dates] + makeBudget t = txnTieKnot $ t { tdescription = T.pack "Budget transaction" } + journalBalanceTransactions' opts j = + let assrt = not . ignore_assertions_ $ inputopts_ opts + in + either error' id $ journalBalanceTransactions assrt j + -- single-column balance reports @@ -494,16 +543,66 @@ multiBalanceReportAsText opts r = CumulativeChange -> "Ending balances (cumulative)" HistoricalBalance -> "Ending balances (historical)" +-- | Render two multi-column balance reports as plain text suitable for console output. +-- They are assumed to have same number of columns, one of them representing +-- a budget +multiBalanceReportWithBudgetAsText :: ReportOpts -> MultiBalanceReport -> MultiBalanceReport -> String +multiBalanceReportWithBudgetAsText opts budget r = + printf "%s in %s:\n\n" typeStr (showDateSpan $ multiBalanceReportSpan r) + ++ renderBalanceReportTable' opts showcell tabl + where + tabl = combine (balanceReportAsTable opts r) (balanceReportAsTable opts budget) + typeStr :: String + typeStr = case balancetype_ opts of + PeriodChange -> "Balance changes" + CumulativeChange -> "Ending balances (cumulative)" + HistoricalBalance -> "Ending balances (historical)" + showcell (real, Nothing) = showamt real + showcell (real, Just budget) = + printf "%s [%s]" (showamt real) (showamt budget) + showamt | color_ opts = cshowMixedAmountOneLineWithoutPrice + | otherwise = showMixedAmountOneLineWithoutPrice + -- combine reportTable budgetTable will combine them into a single table where cells + -- are tuples of (actual, Maybe budget) numbers. Main assumptions is that + -- row/column titles of budgetTable are subset of row/column titles or reportTable, + -- and there are now row/column titles in budgetTable that are not mentioned in reporTable. + -- Both of these are satisfied by construction of budget report and process of rolling up + -- account names. + combine (Table l t d) (Table l' t' d') = Table l t combinedRows + where + -- For all accounts that are present in the budget, zip real amounts with budget amounts + combinedRows = [ combineRow row budgetRow + | (acct, row) <- zip (headerContents l) d + , let budgetRow = + if acct == "" then [] -- "" is totals row + else fromMaybe [] $ Map.lookup acct budgetAccts + ] + -- Budget could cover smaller interval of time than the whole journal. + -- Headers for budget row will always be a sublist of headers of row + combineRow r br = + let reportRow = zip (headerContents t) r + budgetRow = Map.fromList $ zip (headerContents t') br + findBudgetVal hdr = Map.lookup hdr budgetRow + in map (\(hdr, val) -> (val, findBudgetVal hdr)) reportRow + budgetAccts = Map.fromList $ zip (headerContents l') d' + -- | Given a table representing a multi-column balance report (for example, -- made using 'balanceReportAsTable'), render it in a format suitable for -- console output. renderBalanceReportTable :: ReportOpts -> Table String String MixedAmount -> String -renderBalanceReportTable (ReportOpts { pretty_tables_ = pretty, color_=usecolor }) = +renderBalanceReportTable ropts = + renderBalanceReportTable' ropts showamt + where + showamt | color_ ropts = cshowMixedAmountOneLineWithoutPrice + | otherwise = showMixedAmountOneLineWithoutPrice + +renderBalanceReportTable' :: ReportOpts -> (a -> String) -> Table String String a -> String +renderBalanceReportTable' (ReportOpts { pretty_tables_ = pretty}) showCell = unlines . addtrailingblank . trimborder . lines - . render pretty id id showamt + . render pretty id id showCell . align where addtrailingblank = (++[""]) @@ -512,8 +611,6 @@ renderBalanceReportTable (ReportOpts { pretty_tables_ = pretty, color_=usecolor where acctswidth = maximum' $ map strWidth (headerContents l) l' = padRightWide acctswidth <$> l - showamt | usecolor = cshowMixedAmountOneLineWithoutPrice - | otherwise = showMixedAmountOneLineWithoutPrice -- | Build a 'Table' from a multi-column balance report. balanceReportAsTable :: ReportOpts -> MultiBalanceReport -> Table String String MixedAmount diff --git a/tests/budget/budget.test b/tests/budget/budget.test new file mode 100644 index 000000000..e7c457dc9 --- /dev/null +++ b/tests/budget/budget.test @@ -0,0 +1,92 @@ +# Test --budget switch +hledger bal -D -b 2016-12-01 -e 2016-12-04 -f - --budget +<<< +2016/12/01 + expenses:food $10 + assets:cash + +2016/12/02 + expenses:food $9 + assets:cash + +2016/12/03 + expenses:food $11 + assets:cash + +2016/12/02 + expenses:leisure $5 + assets:cash + +2016/12/03 + expenses:movies $25 + assets:cash + +2016/12/03 + expenses:cab $15 + assets:cash + +~ daily from 2016/1/1 + expenses:food $10 + expenses:leisure $15 + assets:cash +>>> +Balance changes in 2016/12/01-2016/12/03: + + || 2016/12/01 2016/12/02 2016/12/03 +=======================++======================================= + :expenses || 0 0 $40 + assets:cash || $-10 [$-25] $-14 [$-25] $-51 [$-25] + expenses:food || $10 [$10] $9 [$10] $11 [$10] + expenses:leisure || 0 [$15] $5 [$15] 0 [$15] +-----------------------++--------------------------------------- + || 0 0 0 + +>>>2 +>>>=0 + +# --show-unbudgeted +hledger bal -D -b 2016-12-01 -e 2016-12-04 -f - --budget --show-unbudgeted +<<< +2016/12/01 + expenses:food $10 + assets:cash + +2016/12/02 + expenses:food $9 + assets:cash + +2016/12/03 + expenses:food $11 + assets:cash + +2016/12/02 + expenses:leisure $5 + assets:cash + +2016/12/03 + expenses:movies $25 + assets:cash + +2016/12/03 + expenses:cab $15 + assets:cash + +~ daily from 2016/1/1 + expenses:food $10 + expenses:leisure $15 + assets:cash +>>> +Balance changes in 2016/12/01-2016/12/03: + + || 2016/12/01 2016/12/02 2016/12/03 +==================++======================================= + assets:cash || $-10 [$-25] $-14 [$-25] $-51 [$-25] + expenses:cab || 0 0 $15 + expenses:food || $10 [$10] $9 [$10] $11 [$10] + expenses:leisure || 0 [$15] $5 [$15] 0 [$15] + expenses:movies || 0 0 $25 +------------------++--------------------------------------- + || 0 0 0 + +>>>2 +>>>=0 From 8cd58b71ab6a617635c28e3738d03f1527769ae6 Mon Sep 17 00:00:00 2001 From: Dmitry Astapov Date: Sun, 19 Nov 2017 13:07:07 +0000 Subject: [PATCH 12/13] bal: show percentage of budget spent --- hledger/Hledger/Cli/Commands/Balance.hs | 10 +++++++- hledger/hledger.cabal | 3 +++ hledger/package.yaml | 3 +++ tests/budget/budget.test | 34 ++++++++++++------------- 4 files changed, 32 insertions(+), 18 deletions(-) diff --git a/hledger/Hledger/Cli/Commands/Balance.hs b/hledger/Hledger/Cli/Commands/Balance.hs index 83ca1ac58..3e4af3743 100644 --- a/hledger/Hledger/Cli/Commands/Balance.hs +++ b/hledger/Hledger/Cli/Commands/Balance.hs @@ -252,6 +252,7 @@ import qualified Data.Map as Map -- import Data.Monoid import qualified Data.Text as T import System.Console.CmdArgs.Explicit as C +import Data.Decimal (roundTo) import Text.CSV import Test.HUnit import Text.Printf (printf) @@ -559,7 +560,14 @@ multiBalanceReportWithBudgetAsText opts budget r = HistoricalBalance -> "Ending balances (historical)" showcell (real, Nothing) = showamt real showcell (real, Just budget) = - printf "%s [%s]" (showamt real) (showamt budget) + case percentage real budget of + Just pct -> printf "%s [%s%% of %s]" (showamt real) (show $ roundTo 0 pct) (showamt budget) + Nothing -> printf "%s [%s]" (showamt real) (showamt budget) + percentage real budget = + case (real, budget) of + (Mixed [a1], Mixed [a2]) | acommodity a1 == acommodity a2 && aquantity a2 /= 0 -> + Just $ 100 * aquantity a1 / aquantity a2 + _ -> Nothing showamt | color_ opts = cshowMixedAmountOneLineWithoutPrice | otherwise = showMixedAmountOneLineWithoutPrice -- combine reportTable budgetTable will combine them into a single table where cells diff --git a/hledger/hledger.cabal b/hledger/hledger.cabal index 2b6f4a957..eadc13ec6 100644 --- a/hledger/hledger.cabal +++ b/hledger/hledger.cabal @@ -113,6 +113,7 @@ library , text >=0.11 , utf8-string >=0.3.5 && <1.1 , wizards ==1.0.* + , Decimal if (!(os(windows))) && (flag(terminfo)) build-depends: terminfo @@ -192,6 +193,7 @@ executable hledger , text >=0.11 , utf8-string >=0.3.5 && <1.1 , wizards ==1.0.* + , Decimal if (!(os(windows))) && (flag(terminfo)) build-depends: terminfo @@ -241,6 +243,7 @@ test-suite test , text >=0.11 , utf8-string >=0.3.5 && <1.1 , wizards ==1.0.* + , Decimal , test-framework , test-framework-hunit if (!(os(windows))) && (flag(terminfo)) diff --git a/hledger/package.yaml b/hledger/package.yaml index f32e18c1e..9560ab69e 100644 --- a/hledger/package.yaml +++ b/hledger/package.yaml @@ -149,6 +149,7 @@ library: - text >=0.11 - utf8-string >=0.3.5 && <1.1 - wizards ==1.0.* + - Decimal executables: hledger: @@ -178,6 +179,7 @@ executables: - text >=0.11 - utf8-string >=0.3.5 && <1.1 - wizards ==1.0.* + - Decimal tests: test: @@ -204,6 +206,7 @@ tests: - text >=0.11 - utf8-string >=0.3.5 && <1.1 - wizards ==1.0.* + - Decimal - test-framework - test-framework-hunit diff --git a/tests/budget/budget.test b/tests/budget/budget.test index e7c457dc9..7d7e22bc6 100644 --- a/tests/budget/budget.test +++ b/tests/budget/budget.test @@ -32,14 +32,14 @@ hledger bal -D -b 2016-12-01 -e 2016-12-04 -f - --budget >>> Balance changes in 2016/12/01-2016/12/03: - || 2016/12/01 2016/12/02 2016/12/03 -=======================++======================================= - :expenses || 0 0 $40 - assets:cash || $-10 [$-25] $-14 [$-25] $-51 [$-25] - expenses:food || $10 [$10] $9 [$10] $11 [$10] - expenses:leisure || 0 [$15] $5 [$15] 0 [$15] ------------------------++--------------------------------------- - || 0 0 0 + || 2016/12/01 2016/12/02 2016/12/03 +=======================++============================================================= + :expenses || 0 0 $40 + assets:cash || $-10 [40% of $-25] $-14 [56% of $-25] $-51 [204% of $-25] + expenses:food || $10 [100% of $10] $9 [90% of $10] $11 [110% of $10] + expenses:leisure || 0 [$15] $5 [33% of $15] 0 [$15] +-----------------------++------------------------------------------------------------- + || 0 0 0 >>>2 >>>=0 @@ -78,15 +78,15 @@ hledger bal -D -b 2016-12-01 -e 2016-12-04 -f - --budget --show-unbudgeted >>> Balance changes in 2016/12/01-2016/12/03: - || 2016/12/01 2016/12/02 2016/12/03 -==================++======================================= - assets:cash || $-10 [$-25] $-14 [$-25] $-51 [$-25] - expenses:cab || 0 0 $15 - expenses:food || $10 [$10] $9 [$10] $11 [$10] - expenses:leisure || 0 [$15] $5 [$15] 0 [$15] - expenses:movies || 0 0 $25 -------------------++--------------------------------------- - || 0 0 0 + || 2016/12/01 2016/12/02 2016/12/03 +==================++============================================================= + assets:cash || $-10 [40% of $-25] $-14 [56% of $-25] $-51 [204% of $-25] + expenses:cab || 0 0 $15 + expenses:food || $10 [100% of $10] $9 [90% of $10] $11 [110% of $10] + expenses:leisure || 0 [$15] $5 [33% of $15] 0 [$15] + expenses:movies || 0 0 $25 +------------------++------------------------------------------------------------- + || 0 0 0 >>>2 >>>=0 From aca82cf400a2214169f4eb85321fc7aeb58e3da8 Mon Sep 17 00:00:00 2001 From: Dmitry Astapov Date: Fri, 24 Nov 2017 00:40:49 +0000 Subject: [PATCH 13/13] doc: budgeting and forecasting how-to, demonstrating new features --- examples/budget.journal | 16 ++ site/budgeting-and-forecasting.md | 323 ++++++++++++++++++++++++++++++ 2 files changed, 339 insertions(+) create mode 100644 examples/budget.journal create mode 100644 site/budgeting-and-forecasting.md diff --git a/examples/budget.journal b/examples/budget.journal new file mode 100644 index 000000000..f8d33d672 --- /dev/null +++ b/examples/budget.journal @@ -0,0 +1,16 @@ +~ monthly from 2013/01 + Expenses:Food 500 USD + Expenses:Health 200 USD + Expenses:Home 2545 USD + Expenses:Transport 120 USD + Expenses:Taxes 4300 USD ;; Taken from monthly average report + Income:US -10700 USD + Assets:US + +~ every Dec 20th from 2014 + Expenses:Food 500 USD ; Prize turkey, the biggest of the big + Assets:US + +~ 2014/11/17 + Assets:US + Expenses:Food 6000 USD ; Birthday, lots of guests diff --git a/site/budgeting-and-forecasting.md b/site/budgeting-and-forecasting.md new file mode 100644 index 000000000..f1d964abd --- /dev/null +++ b/site/budgeting-and-forecasting.md @@ -0,0 +1,323 @@ +# Budgeting and forecasting + +Budgeting and forecasting allows you to keep better track of your expenses and future financial situation. +If you write down your expectations of what your income/expenses/investment yields/etc should be, you can use them to: +- check how far off are your expectations from reality (budgeting) +- project your future account activity or balances (forecasting) + +(This section uses examples/bcexample.hledger from hledger source repository). + +## Periodic budget +To start budgeting, you need to know what your average yearly or weekly expenditures are. Hledger could help you with that. +Usually the interval for which you compute budget figures will be the same as the interval between +your paychecks -- monthly or weekly. + +Lets create monthly (-M) report for years 2013-2014 (-b 2013) of all +top-level expense categories (--depth 2 Expenses), looking for average +figures (-A) in the cost at the time of transaction (-B), limiting +ourselves to USD transactions only, to save screen space: + +```shell +$ hledger balance -f bcexample.hledger -MBA -b 2013 --depth 2 Expenses cur:USD +Balance changes in 2013/01/01-2014/10/31: + + || 2013/01 2013/02 2013/03 ... 2014/07 2014/08 2014/09 2014/10 Average +====================++========================================...================================================================== + Expenses:Financial || 4.00 USD 12.95 USD 39.80 USD ... 30.85 USD 21.90 USD 12.95 USD 4.00 USD 17.83 USD + Expenses:Food || 396.46 USD 481.48 USD 603.32 USD ... 871.20 USD 768.23 USD 466.72 USD 83.00 USD 562.10 USD + Expenses:Health || 290.70 USD 193.80 USD 193.80 USD ... 290.70 USD 193.80 USD 193.80 USD 96.90 USD 207.01 USD + Expenses:Home || 2544.98 USD 2545.02 USD 2544.97 USD ... 2545.12 USD 2545.01 USD 2545.10 USD 0 2429.33 USD + Expenses:Taxes || 5976.60 USD 3984.40 USD 4901.83 USD ... 5976.60 USD 3984.40 USD 3984.40 USD 1992.20 USD 4322.27 USD + Expenses:Transport || 120.00 USD 120.00 USD 120.00 USD ... 0 120.00 USD 120.00 USD 120.00 USD 109.09 USD +--------------------++----------------------------------------...------------------------------------------------------------------ + || 9332.74 USD 7337.65 USD 8403.72 USD ... 9714.47 USD 7633.34 USD 7322.97 USD 2296.10 USD 7647.64 USD +``` + +This report is rather wide and portion of it had been cut out for +brevity. Most interesting column is the last one, it shows average +monthly expenses for each category. Expenses in Food, Health, Home and +Transport categories seem to roughly similar month to month, so lets +create a budget for them. + +Budgets are described with periodic transactions. Periodic transaction +has `~` instead of date and period expression instead of description. In this case +we want to create a monthly budget that will come into effect starting from January 2013, +which will include income of 10000 USD that is partically spent on Food, Health, Home and Transport +and the rest becomes our Assets: + +```journal +~ monthly from 2013/01 + Expenses:Food 500 USD + Expenses:Health 200 USD + Expenses:Home 2545 USD + Expenses:Transport 120 USD + Income:US -10700 USD ;; Taken as monthy average of Income account group + Assets:US +``` + +This transaction could be put into separate file (budget.journal) or +could be kept in the main journal. Normally hledger will ignore it and +will not include it in any computations or reports. + +To put it into action, you need to add `--budget` switch to your balance invocation. If you do that, +you would be able to see how your past expenses aligned with the budget that you just created. This +time, lets not limit accounts in any way: +```shell +$ hledger balance -f bcexample.hledger -f budget.journal -MB -b 2013 --budget cur:USD +Balance changes in 2013/01/01-2014/10/31: + + || 2013/01 2013/02 2013/03 +==========================++=========================================================================================================== + :Expenses || 5980.60 USD 3997.35 USD 4941.63 USD + :Liabilities || 293.09 USD -147.51 USD -66.01 USD + Assets:US || 1893.32 USD [26% of 7335 USD] 2929.77 USD [40% of 7335 USD] -3898.89 USD [-53% of 7335 USD] + Expenses:Food || 396.46 USD [79% of 500 USD] 481.48 USD [96% of 500 USD] 603.32 USD [121% of 500 USD] + Expenses:Health || 290.70 USD [145% of 200 USD] 193.80 USD [97% of 200 USD] 193.80 USD [97% of 200 USD] + Expenses:Home || 2544.98 USD [100% of 2545 USD] 2545.02 USD [100% of 2545 USD] 2544.97 USD [100% of 2545 USD] + Expenses:Transport || 120.00 USD [100% of 120 USD] 120.00 USD [100% of 120 USD] 120.00 USD [100% of 120 USD] + Income:US || -15119.10 USD [141% of -10700 USD] -10331.21 USD [97% of -10700 USD] -11079.40 USD [104% of -10700 USD] +--------------------------++----------------------------------------------------------------------------------------------------------- + || -3599.95 USD -211.30 USD -6640.58 USD + +``` + +Numbers in square brackets give you your budget estimate and percentage of it used by your real expenses. Numbers below 100% mean +that you have some of your budget left, numbers over 100% mean that you went over your budget. + +You can notice that actual numbers for Assets:US seem to be well below computed budget of 7335 USD. Why? Answer to this is in the first +row of the report: we have quite a lot of unbudgeted Expenses! + +Notice that even though we have not limited accounts in any way, report includes just those mentioned in the budget. This is on purpose, +assumption is that when you are checking your budgets you probably do not want unbudgeted accounts getting in your way. Another thing to +note is that budget numbers have been allocated to top-level expense subcategories (like Expenses:Food). Journal has subaccounts under +Food, but to compute budget report they have all been rolled up into a nearest parent with budget number associated with it. Accounts that +do not have such parent went into `` row. + +Allright, it seems that for Jan 2013 we have ~3000 USD of budgeted expenses and almost twice as much unbudgeted. Lets figure out what they are. +We can see more details if we add `--show-unbudgeted` switch: +```shell +$ hledger balance -f bcexample.hledger -f budget.journal -M -b 2013-01 -e 2013-02 --budget cur:USD --show-unbudgeted +Balance changes in 2013/01: + + || 2013/01 +==================================++==================================== + Assets:US || 1893.32 USD [26% of 7335 USD] + Expenses:Financial:Fees || 4.00 USD + Expenses:Food || 396.46 USD [79% of 500 USD] + Expenses:Health || 290.70 USD [145% of 200 USD] + Expenses:Home || 2544.98 USD [100% of 2545 USD] + Expenses:Taxes:Y2013:US:CityNYC || 524.76 USD + Expenses:Taxes:Y2013:US:Federal || 3188.76 USD + Expenses:Taxes:Y2013:US:Medicare || 319.86 USD + Expenses:Taxes:Y2013:US:SDI || 3.36 USD + Expenses:Taxes:Y2013:US:SocSec || 844.62 USD + Expenses:Taxes:Y2013:US:State || 1095.24 USD + Expenses:Transport || 120.00 USD [100% of 120 USD] + Income:US || -15119.10 USD [141% of -10700 USD] + Liabilities:US:Chase:Slate || 293.09 USD +----------------------------------++------------------------------------ + || -3599.95 USD + +``` + +All the accounts that were rolled up into `` category are now shown with their original name, but budgeted accounts are still rolled up. It +is easy to see now that we forgot taxes. Lets add them to our budget: +```journal +~ monthly from 2013/01 + Expenses:Food 500 USD + Expenses:Health 200 USD + Expenses:Home 2545 USD + Expenses:Transport 120 USD + Expenses:Taxes 4300 USD ;; Taken from monthly average report + Income:US -10700 USD + Assets:US +``` + +Lets try again for a couple of month with this updated budget: +```shell +$ hledger balance -f bcexample.hledger -f budget.journal -M -b 2013-01 -e 2013-04 --budget cur:USD +Balance changes in 2013q1: + + || 2013/01 2013/02 2013/03 +==========================++=========================================================================================================== + :Expenses || 4.00 USD 12.95 USD 39.80 USD + :Liabilities || 293.09 USD -147.51 USD -66.01 USD + Assets:US || 1893.32 USD [62% of 3035 USD] 2929.77 USD [97% of 3035 USD] -3898.89 USD [-128% of 3035 USD] + Expenses:Food || 396.46 USD [79% of 500 USD] 481.48 USD [96% of 500 USD] 603.32 USD [121% of 500 USD] + Expenses:Health || 290.70 USD [145% of 200 USD] 193.80 USD [97% of 200 USD] 193.80 USD [97% of 200 USD] + Expenses:Home || 2544.98 USD [100% of 2545 USD] 2545.02 USD [100% of 2545 USD] 2544.97 USD [100% of 2545 USD] + Expenses:Taxes || 5976.60 USD [139% of 4300 USD] 3984.40 USD [93% of 4300 USD] 4901.83 USD [114% of 4300 USD] + Expenses:Transport || 120.00 USD [100% of 120 USD] 120.00 USD [100% of 120 USD] 120.00 USD [100% of 120 USD] + Income:US || -15119.10 USD [141% of -10700 USD] -10331.21 USD [97% of -10700 USD] -11079.40 USD [104% of -10700 USD] +--------------------------++----------------------------------------------------------------------------------------------------------- + || -3599.95 USD -211.30 USD -6640.58 USD + +``` + +Now unbudgeted amounts are much smaller and some of them could be dismissed as noise, and we can see that budget created is actually +close enough to the real numbers, meaning that they are usually close to average that we put in our budget. + +## Envelope budget + +Budget report that we have used so far assumes that any unused budget amount for a given (monthly) period will not contribute to the +budget of the next period. Alternative popular "envelope budget" strategy assumes that you put a certain amount of money into an envelope +each month, and any unused amount stays there for future expenses. This is easy to simulate by adding --cumulative switch. Lets redo +the last report with it: +```shell +$ hledger balance -f bcexample.hledger -f budget.journal -M -b 2013-01 -e 2013-04 --cumulative --budget cur:USD +Ending balances (cumulative) in 2013q1: + + || 2013/01/31 2013/02/28 2013/03/31 +==========================++============================================================================================================ + :Expenses || 4.00 USD 16.95 USD 56.75 USD + :Liabilities || 293.09 USD 145.58 USD 79.57 USD + Assets:US || 1893.32 USD [62% of 3035 USD] 4823.09 USD [79% of 6070 USD] 924.20 USD [10% of 9105 USD] + Expenses:Food || 396.46 USD [79% of 500 USD] 877.94 USD [88% of 1000 USD] 1481.26 USD [99% of 1500 USD] + Expenses:Health || 290.70 USD [145% of 200 USD] 484.50 USD [121% of 400 USD] 678.30 USD [113% of 600 USD] + Expenses:Home || 2544.98 USD [100% of 2545 USD] 5090.00 USD [100% of 5090 USD] 7634.97 USD [100% of 7635 USD] + Expenses:Taxes || 5976.60 USD [139% of 4300 USD] 9961.00 USD [116% of 8600 USD] 14862.83 USD [115% of 12900 USD] + Expenses:Transport || 120.00 USD [100% of 120 USD] 240.00 USD [100% of 240 USD] 360.00 USD [100% of 360 USD] + Income:US || -15119.10 USD [141% of -10700 USD] -25450.31 USD [119% of -21400 USD] -36529.71 USD [114% of -32100 USD] +--------------------------++------------------------------------------------------------------------------------------------------------ + || -3599.95 USD -3811.25 USD -10451.83 USD + +``` + +If you look at Expenses:Food category, you will see that every month budget is increased by 500 USD, and by March total amount budgeted +is 1500 USD, of which 1481.26 USD is spent. If you look back at the previous non-cumulative monthly budget report, you will see that in March food expenses +were 121% of the budgeted amount, but cumulative report shows that taking into account budget carry-over from Jan and Feb we are well withing planned numbers. + +# Forecasting + +Budget transaction that was created could be used to predict what would be our financial situation in the future. If you add `--forecast` switch, you will +see how budgeted income and expense affects you past the last transaction in the journal. Since journal ends in Oct 2014, lets see next two month: + +```shell +$ hledger balance -f bcexample.hledger -f budget.journal -M -b 2014-10 -e 2015 --forecast cur:USD +Balance changes in 2014q4: + + || 2014/10 2014/11 2014/12 +====================================++====================================== + Assets:US || 0 3035 USD 3035 USD + Assets:US:BofA:Checking || -2453.40 USD 0 0 + Assets:US:ETrade:Cash || 5000.00 USD 0 0 + Expenses:Financial:Fees || 4.00 USD 0 0 + Expenses:Food || 0 500 USD 500 USD + Expenses:Food:Restaurant || 83.00 USD 0 0 + Expenses:Health || 0 200 USD 200 USD + Expenses:Health:Dental:Insurance || 2.90 USD 0 0 + Expenses:Health:Life:GroupTermLife || 24.32 USD 0 0 + Expenses:Health:Medical:Insurance || 27.38 USD 0 0 + Expenses:Health:Vision:Insurance || 42.30 USD 0 0 + Expenses:Home || 0 2545 USD 2545 USD + Expenses:Taxes || 0 4300 USD 4300 USD + Expenses:Taxes:Y2014:US:CityNYC || 174.92 USD 0 0 + Expenses:Taxes:Y2014:US:Federal || 1062.92 USD 0 0 + Expenses:Taxes:Y2014:US:Medicare || 106.62 USD 0 0 + Expenses:Taxes:Y2014:US:SDI || 1.12 USD 0 0 + Expenses:Taxes:Y2014:US:SocSec || 281.54 USD 0 0 + Expenses:Taxes:Y2014:US:State || 365.08 USD 0 0 + Expenses:Transport || 0 120 USD 120 USD + Expenses:Transport:Tram || 120.00 USD 0 0 + Income:US || 0 -10700 USD -10700 USD + Income:US:Hoogle:GroupTermLife || -24.32 USD 0 0 + Income:US:Hoogle:Salary || -4615.38 USD 0 0 + Liabilities:US:Chase:Slate || -203.00 USD 0 0 +------------------------------------++-------------------------------------- + || 0 0 0 +``` + +Note that this time there is no roll-up of accounts. Unlike `--budget`, which could be used with `balance` command only, `--forecast` +could be used with any report. Forecast transactions would be added to your real journal and would appear in the report you requested as +if you have entered them on the scheduled dates. + +Since quite a lot of accounts do not have any budgeted transactions, lets limit the depth of the report to avoid seeing lots of zeroes: +```shell +$ hledger balance -f bcexample.hledger -f budget.journal -M -b 2014-10 -e 2015 --forecast cur:USD --depth 2 +Balance changes in 2014q4: + + || 2014/10 2014/11 2014/12 +====================++====================================== + Assets:US || 2546.60 USD 3035 USD 3035 USD + Expenses:Financial || 4.00 USD 0 0 + Expenses:Food || 83.00 USD 500 USD 500 USD + Expenses:Health || 96.90 USD 200 USD 200 USD + Expenses:Home || 0 2545 USD 2545 USD + Expenses:Taxes || 1992.20 USD 4300 USD 4300 USD + Expenses:Transport || 120.00 USD 120 USD 120 USD + Income:US || -4639.70 USD -10700 USD -10700 USD + Liabilities:US || -203.00 USD 0 0 +--------------------++-------------------------------------- + || 0 0 0 +``` + +As you can see, we should expect 3035 USD to be added into Assets:US each month. It is quite easy to see how overal amount of Assets will change with time if you use +`--cumulative` switch: +```shell +$ hledger balance -f bcexample.hledger -f budget.journal -M -b 2014-10 -e 2015 --forecast cur:USD --depth 2 --cumulative +Ending balances (cumulative) in 2014q4: + + || 2014/10/31 2014/11/30 2014/12/31 +====================++============================================ + Assets:US || 2546.60 USD 5581.60 USD 8616.60 USD + Expenses:Financial || 4.00 USD 4.00 USD 4.00 USD + Expenses:Food || 83.00 USD 583.00 USD 1083.00 USD + Expenses:Health || 96.90 USD 296.90 USD 496.90 USD + Expenses:Home || 0 2545 USD 5090 USD + Expenses:Taxes || 1992.20 USD 6292.20 USD 10592.20 USD + Expenses:Transport || 120.00 USD 240.00 USD 360.00 USD + Income:US || -4639.70 USD -15339.70 USD -26039.70 USD + Liabilities:US || -203.00 USD -203.00 USD -203.00 USD +--------------------++-------------------------------------------- + || 0 0 0 +``` + +According to forecast, assets are expected to grow to 8600+ USD by the end of 2014. However, our forecast does not include a couple +of big one-off year end expenses. First, we plan to buy prize turkey for the Christmas table every year from 2014, spending up to 500 USD on it. +And on 17th Nov 2014 we would celebrate birthday of significant other, spending up to 6000 USD in a fancy restaurant: +```journal +~ every 20th Dec from 2014 + Expenses:Food 500 USD ; Prize turkey, the biggest of the big + Assets:US + +~ 2014/11/17 + Assets:US + Expenses:Food 6000 USD ; Birthday, lots of guests +``` + +Note that turkey transaction is not entered as "yearly from 2014/12/20", since yearly/quarterly/monthy/weekly periodic expressions always generate +entries at the first day of the calendar year/quarter/month/week. Thus "monthly from 2014/12" will occur on 2014/12/01, 2015/01/01, ..., whereas +"every 20th of month from 2014/12" will happen on 2014/12/20, 2015/12/20, etc. + +With latest additions forecast now looks like this: +```shell +hledger balance -f bcexample.hledger -f budget.journal -M -b 2014-10 -e 2015 --forecast cur:USD --depth 2 --cumulative +Ending balances (cumulative) in 2014q4: + + || 2014/10/31 2014/11/30 2014/12/31 +====================++============================================ + Assets:US || 2546.60 USD -418.40 USD 2116.60 USD + Expenses:Financial || 4.00 USD 4.00 USD 4.00 USD + Expenses:Food || 83.00 USD 6583.00 USD 7583.00 USD + Expenses:Health || 96.90 USD 296.90 USD 496.90 USD + Expenses:Home || 0 2545 USD 5090 USD + Expenses:Taxes || 1992.20 USD 6292.20 USD 10592.20 USD + Expenses:Transport || 120.00 USD 240.00 USD 360.00 USD + Income:US || -4639.70 USD -15339.70 USD -26039.70 USD + Liabilities:US || -203.00 USD -203.00 USD -203.00 USD +--------------------++-------------------------------------------- + || 0 0 0 +``` + +It is easy to see that in Nov 2014 we will run out of Assets. Using `register` we can figure out when or why it would happen: +```shell +$ hledger register -f bcexample.hledger -f budget.journal -b 2014-10 -e 2014-12 --forecast cur:USD Assets +2014/10/04 "BANK FEES" | "Monthly bank fee" Assets:US:BofA:Checking -4.00 USD -4.00 USD +2014/10/09 "Hoogle" | "Payroll" Assets:US:BofA:Checking 2550.60 USD 2546.60 USD +2014/10/10 "Transfering accumulated savings to o.. Assets:US:BofA:Checking -5000.00 USD -2453.40 USD + Assets:US:ETrade:Cash 5000.00 USD 2546.60 USD +2014/11/01 Forecast transaction Assets:US 3035 USD 5581.60 USD +2014/11/17 Forecast transaction Assets:US -6000 USD -418.40 USD +``` + +It is 6000 USD planned for birthday! Something will have to be done about the birthday plans.