Matching LibreOffice to the fifteenth digit
Cells, the spreadsheet in P33, is held against LibreOffice cell by cell, and a number matches only when all fifteen digits LibreOffice writes are the same. With plain JavaScript arithmetic in node, eight cells of the corpus differ, all of them logarithms or loan and cash flow formulas. Two files, cr.mjs and fp.mjs, compute those values exactly with BigInt and round them once, and every cell matches. Below, the whole corpus is worked out in your browser. Switch a part off to see which cells depend on it.
| Workbook and cell | Formula | LibreOffice | Cells |
|---|
The corpus is the fourteen workbooks P33 is tested on and the eighty workbooks of its fuzzer, each worked out and saved by LibreOffice. Your browser builds every workbook with the same code as P33 and compares each formula cell with LibreOffice’s value. The marked digits in the Cells column are where the two values first differ. With a part off, the page uses your browser’s own Math functions, so the count can differ from the command further down, which ran in node.
How the judge compares two numbers
A spreadsheet number is a double, the same floating point type as a number in JavaScript. Writing a double so that it reads back as the same double takes up to seventeen significant digits. LibreOffice writes fifteen of them into the file it saves. P33’s judge (judge.mjs) writes its own number the same way, rounded half away from zero at the fifteenth digit, and compares the two texts.
Two doubles that sit side by side, one unit in the last place apart (an ulp, the gap from a double to the next one), usually round to the same fifteen digits. They read differently when a rounding boundary of the fifteenth digit falls between them. A function that is one ulp off therefore passes most cells and fails a few, and which ones fail depends on the inputs.
A logarithm rounded once
A function is correctly rounded when it returns the double nearest to the true value. The JavaScript specification asks Math.log only for an approximation. The size of the error is left to the engine. , the engine in Chrome and node, returns the nearest double for most inputs and its neighbor for some. Math.log(3) is one of them. Pick a function and a number below to see where the true value lies.
LibreOffice takes its logarithms from the system library of the Mac it ran on. One cell of the corpus depends on the difference: =LOG10(LN(3)), from the fuzzer. The two values of LN(3) read the same in fifteen digits. LOG10 of a number this close to one magnifies their relative difference, so the fifteenth digit of the result changes. LibreOffice saved the logarithm of the correctly rounded LN(3).
cr.mjs computes each value in fixed point: a BigInt that holds the number times a power of two, with P = 160n bits after the point where a double has fifty-two. The logarithm splits its input into a power of two and a factor between three quarters and one and a half. It then sums the series in ln(f) = 2 * atanh((f - 1) / (f + 1)) until a term is zero. The exponential subtracts a whole number of ln(2) and sums the Taylor series of what is left. The result is then rounded once to the nearest double, with ties to the even one, in nearest().
One rounding instead of two
JavaScript computes a * b + c with two roundings: one after the product and one after the sum. The compiler that built LibreOffice for the Mac turned that expression, in the loan formulas, into a fused multiply-add, an instruction that rounds once. JavaScript has no such operation, so fp.mjs writes one. It splits each double into an integer and a power of two, and multiplies the integers exactly as BigInt. It lines the sum up on the smaller power, adds, and rounds the result once.
RATE shows the effect most. It finds the interest rate by Newton’s method. It stops when a step is smaller than one ten millionth. A different rounding in any step changes the steps after it, and so the rate it returns. In the loan workbook, the two ways of rounding a * b + c give rates that first differ in the fourteenth digit.
Whole powers get the same treatment. fp.mjs raises the significant bits of x, read as one whole number, to the power exactly as a BigInt, and rounds once. A negative power divides a power of two by that BigInt, with sixty more bits than the divisor has. A final bit records whether the division left a remainder. In node, Math.pow(1.05, 127) returns the other double, and a cell would show it. The figure above has it as an example, worked out by your browser.
Held against Python and LibreOffice
The true values come from code that shares nothing with cr.mjs or fp.mjs. oracle.py uses Python’s decimal module, at the precision in the first line below, for the logarithms, exponentials and real powers. It uses exact fractions for the fused multiply-add and the whole powers, and rounds each value once to the nearest double. The first table counts the values each side gets wrong. The second runs plain JavaScript against the two files on random inputs. It counts how often plain JavaScript returns another double and how often the fifteen digits change too, and it times each call.
$ node web/notes/digits/ulp.mjs
against oracle.json (Python's decimal at 120 digits, exact fractions), 4,000 values
function values cr.mjs or fp.mjs wrong plain JavaScript wrong
ln 500 0 5
log10 500 0 26
exp 500 0 55
log1p 500 0 45
expm1 500 0 44
a^b, b not whole 500 0 51
a^n, n whole 500 0 57
a * b + c 500 0 133
cr.mjs and fp.mjs gave the true value for all 4,000
100,000 random inputs per function, plain JavaScript against cr.mjs and fp.mjs, in node 22.22.0
function other double other 15 digits plain, per call exact, per call
ln 608 (0.6%) 57 (0.06%) 109 ns 6.1 µs
log10 5,330 (5.3%) 313 (0.31%) 120 ns 6.3 µs
exp 9,579 (9.6%) 723 (0.72%) 43 ns 6.7 µs
log1p 8,024 (8.0%) 643 (0.64%) 79 ns 7.9 µs
expm1 9,250 (9.3%) 598 (0.60%) 54 ns 7.2 µs
a^b, b not whole 9,776 (9.8%) 660 (0.66%) 154 ns 12.6 µs
a^n, n whole 9,726 (9.7%) 566 (0.57%) 174 ns 19.3 µs
a * b + c 21,211 (21.2%) 2,284 (2.28%) 97 ns 888 nsThen the corpus, worked out by the copy of Cells this page runs, with each part switched off on its own and then all three. The two columns on the right are Cells with the parts off and with them on.
$ node web/notes/digits/held.mjs
LibreOffice 26.8.0.3, corpus made 2026-10-04: 14 workbooks and 80 from the fuzzer, 5,472 cells
every part on (as P33): 5,472 of 5,472 match
cr.mjs off: 1 differ
fp.mjs fma off: 5 differ
fp.mjs pow off: 4 differ
all three off: 8 differ
the cells that differ with all three off (LibreOffice, Cells with them off, Cells with them on):
finance H17 =NPV(0.1,-1000,A5:A7) -19.1243767502221 -19.1243767502222 -19.1243767502221
finance H28 =RATE(A2,-500,A3) 0.00618341316125358 0.00618341316125379 0.00618341316125358
finance H29 =RATE(48,-200,8000) 0.00770147248823326 0.00770147248823306 0.00770147248823326
finance H30 =RATE(12,-100,1000,0,1) 0.0350315303622831 0.035031530362283 0.0350315303622831
finance H32 =RATE(4.5,-100,400) 0.0443334638914012 0.0443334638914013 0.0443334638914012
loan I7 =RATE(B3*B4,-B6,B1)*B4 0.0449999999999946 0.0449999999999901 0.0449999999999946
fuzzer seed 51 H47 =B7-0+FV(0.05,182,25000,0,0) -3592212252.33211 -3592212252.3321 -3592212252.33211
fuzzer seed 68 H35 =LOG10(LN(3-C1)) 0.0408444525689207 0.0408444525689206 0.0408444525689207
in 1.3 s, in node 22.22.0What it does not cover
The corpus was saved by one build of LibreOffice on one Mac, named in the first line of the command above. LibreOffice on Windows or Linux uses other system libraries and other compiler settings, and this note says nothing about the digits it gives there.
cr.mjs works with a fixed number of bits and has no proof that they suffice for every input. A correctly rounded result needs enough bits to decide on which side of the midpoint between two doubles the true value falls. An input whose true value lies closer to a midpoint than those bits can resolve would be rounded the wrong way, and cr.mjs does not detect that case. The oracle above found no such input among its values, which is evidence and not a guarantee.
Exact arithmetic costs time: a call to cr.mjs takes microseconds where Math.log takes nanoseconds, as the second table shows. Cells uses the two files only where the last digit can show: the logarithms, EXP, POWER and the power operator, ROUNDUP and ROUNDDOWN, the variance and standard deviation, and the loan and cash flow functions.
The source of cr.mjs and fp.mjs is P33’s, under prototypes/source/p33/ in the portfolio’s repository. The note’s copy of Cells is in web/notes/digits/cells/, written by copy.mjs, and differs only where six files import the two through switch.mjs. held.mjs and ulp.mjs are the commands, wide.mjs computes the true values in the figure, and digits.test.mjs holds the tests.