From a33a9d61c221ed67317cbccfd3b3f9a7d18d3324 Mon Sep 17 00:00:00 2001 From: Simon Michael Date: Tue, 25 Feb 2020 09:26:36 -0800 Subject: [PATCH] json: use a simpler, more consumable number representation (#1195) Amounts in JSON are now rendered as simple Numbers with up to 10 decimal places, instead of Decimal objects which would in some cases have 255 digits, too many for most JSON parsers to handle. A provisional fix, see the comment in Json.hs for more detail. --- hledger-lib/Hledger/Data/Json.hs | 27 +++++++++++++++++++++++++++ hledger/hledger.m4.md | 12 ++++++++++-- 2 files changed, 37 insertions(+), 2 deletions(-) diff --git a/hledger-lib/Hledger/Data/Json.hs b/hledger-lib/Hledger/Data/Json.hs index a67034dff..546b0233b 100644 --- a/hledger-lib/Hledger/Data/Json.hs +++ b/hledger-lib/Hledger/Data/Json.hs @@ -46,7 +46,34 @@ import Hledger.Data.Types instance ToJSON Status instance ToJSON GenericSourcePos + +-- https://github.com/simonmichael/hledger/issues/1195 +-- The default JSON output for Decimal is not very practical for JSON consumers. +-- With repeating decimals, which can occur with implicit transaction prices, +-- decimalMantissa will use Decimal's maximum allowance of 255 digits. +-- (And secondly, it sometimes uses scientific notation, and that sometimes +-- looks wrong, eg e254 instead of e-1 ?) +-- JSON output is intended to be consumed by diverse apps and +-- programming languages, which can't necessarily handle numbers with +-- more than 15 or so significant digits. Eg, from #1195: +-- +-- > - JavaScript uses 64-bit IEEE754 numbers which can only accurately +-- > represent integers up to 9007199254740991 (i.e. a maximum of 15 digits). +-- > - Java’s largest integers are limited to 18 digits. +-- > - Python 3 integers are unbounded. +-- > - Python 2 integers are limited to 18 digits like Java. +-- > - C and C++ number limits depend on platform — most platforms should +-- > be able to represent unsigned integers up to 64 bits, i.e. 19 digits. +-- +-- It's not yet clear what is a good compromise. +-- For now, we make Decimals look like floating point numbers with +-- up to 10 decimal places (and an unbounded number of integer digits). +-- This still allows unparseable numbers to be generated in theory, +-- but hopefully this won't happen in practice. instance ToJSON Decimal + where + toJSON d = Number $ fromRational $ toRational $ roundTo 10 d + instance ToJSON Amount instance ToJSON AmountStyle instance ToJSON Side diff --git a/hledger/hledger.m4.md b/hledger/hledger.m4.md index 64d7ff0e9..ae9b45410 100644 --- a/hledger/hledger.m4.md +++ b/hledger/hledger.m4.md @@ -801,12 +801,20 @@ Some notes about JSON output: - The JSON output from hledger commands is essentially the same as the JSON served by [hledger-web's JSON API](hledger-web.html#json-api), but pretty printed, using line breaks and indentation. - -- Our pretty printer has the ability to elide data in certain cases - + Our pretty printer has the ability to elide data in certain cases - rendering non-strings as if they were strings, or displaying "FOO.." instead of FOO's full details. This should never happen in hledger's JSON output; if you see otherwise, please report as a bug. +- Quantities are represented in hledger as Decimal values storing up + to 255 significant digits, eg for repeating decimals. This is too + many digits and too much hassle for most JSON users, so in JSON we + show simple Numbers with up to 10 decimal places. The number of + significant digits is still unbounded, but that part is under your + control. We hope this approach will not cause problems in practice; + if you find otherwise, please let us know. (Cf + [#1195](https://github.com/simonmichael/hledger/issues/1195)) + ## Regular expressions hledger uses [regular expressions](http://www.regular-expressions.info) in a number of places: