How to prevent loss of precision for operations on length values?

Question

Value \MM is equal to 1mm and is internally stored as 2.84526pt. Value \MMQUARTER is equal to 0.25mm and is supposed to be stored as 0.711315pt (2.84526/4 = 0.711315) but is stored as 0.7113pt. I need to find a solution to prevent the truncation of 0.000015.

\documentclass[border=5mm]{standalone}
\usepackage{tikz}

\begin{document}
    \begin{tikzpicture}

        \pgfmathsetlengthmacro{\MM}{1mm}
        \pgfmathsetlengthmacro{\MMQUARTER}{0.25mm}

        \path node[align=left]
            {%
                \pgfmathparse
                    {% 'MM' IS 2.84526
                        \MM
                    }   \pgfmathresult
                    % NEXT LINE CAN BE IGNORED
                    \hspace{0.33cm} = 1mm
                    \\ % NEW LINE

                \pgfmathparse
                    {% 'MMQUARTER' MUST BE 0.711315 BUT IS 0.7113
                        \MMQUARTER
                    }   \pgfmathresult
                    % NEXT 3 LINES CAN BE IGNORED
                    \hspace{0.5cm} $\approx$ 0.25mm;
                    \\ % NEW LINE
                    \hspace{1.5cm} must be 0.7113\textcolor{red}{15} (exactly 0.25mm)
            };

    \end{tikzpicture}
\end{document}

Answer 1

The first sentence is a bit misleading

Value \MM is equal to 1mm and is internally stored as 2.84526pt

TeX (almost) never uses floating point arithmetic, and never uses it at a place where any difference in machine arithmetic would be detectable from within TeX.

Almost all arithemtic is exact integer arithmetic with decimal conversion just for display purposes.

TeX uses the conversion 25.4mm=1in=72.27pt, 65536sp=1pt

so 1mm is 186467sp and it is this integer that is stored for the length

dividing by 4 and rounding down gets 46616sp which is displayed as 0.7113pt.

You can see the values interactively by running tex on the commandline:

$ tex
This is TeX, Version 3.14159265 (TeX Live 2017) (preloaded format=tex)
**\dimen0=1mm

*\count0=\dimen0

*\showthe\count0
> 186467.
<*> \showthe\count0

? 

*\divide\count0 by 4

*\showthe\count0
> 46616.
<*> \showthe\count0

? 

*\dimen0=\count0 sp

*\showthe\dimen0
> 0.7113pt.
<*> \showthe\dimen0

? x
No pages of output.
Transcript written on texput.log.

Answer 2

Let me first comment on the comments in your code

    \pgfmathsetlengthmacro{\MM}{1mm}
    \pgfmathsetlengthmacro{\MMQUARTER}{0.25mm}

                {% 'MM' IS 2.84526
                    \MM

                {% 'MMQUARTER' MUST BE 0.711315 BUT IS 0.7113
                    \MMQUARTER

I assume that \pgfmathsetlengthmacro boils down internally at some point to standard TeX. A dimension specification like 0.25mm is parsed this way. I will describe it in terms not corresponding one-by-one to the TeX processing but producing exactly equivalent result.

1. first 0.25 is rounded to an integral multiple N/65536. Here this is an exact match N=16384.

2. then a conversion factor is applied, which in the case of the mm unit is (according to tex.web §458) 7227/2540. This means that TeX evaluates the truncation to an integer of the exact fraction 16384*7227/2540.

   Now:

   $ rlwrap etex
   This is pdfTeX, Version 3.14159265-2.6-1.40.18 (TeX Live 2017/MacPorts 2017_1) (preloaded format=etex)
    restricted \write18 enabled.
   **xintexpr.sty
   entering extended mode
   (/usr/local/texlive/2017/texmf-dist/tex/generic/xint/xintexpr.sty
   (/usr/local/texlive/2017/texmf-dist/tex/generic/xint/xintfrac.sty
   (/usr/local/texlive/2017/texmf-dist/tex/generic/xint/xint.sty
   (/usr/local/texlive/2017/texmf-dist/tex/generic/xint/xintcore.sty
   (/usr/local/texlive/2017/texmf-dist/tex/generic/xint/xintkernel.sty))))
   (/usr/local/texlive/2017/texmf-dist/tex/generic/xint/xinttools.sty))
   *\message{\xinttheiexpr [40] 16384*7227/2540\relax}
   46616.9952755905511811023622047244094488188976 
   *\bye
   No pages of output.
   Transcript written on xintexpr.log.
   

   So in this case TeX will obtain 46616 and we see that it makes here a relatively big error.

3. The conclusion is that 0.25mm is stored internally as 46616sp where 1sp is the 65536th part of a (TeX) pt.

For more details see Why pdf file cannot be reproduced?

Then, what happens when a dimension internally stored as 46616sp is displayed using \the ? The algorithm is in §103 of tex.web and is a bit subtle. We want to produce a value in pt unit which would re-produce exactly the 46616sp if rounded to an integral multiple of 1/65536. Of course first we handle the integer part, which here vanishes. So let's assume we have some initial n with 0 <= n < 65536. Knuth actually considers (indirectly) the decimal expansion of (n + 0.5)/65536. Let's say we look at the first K digits of this decimal expansion, so, for K = 1, 2, 3, 4 and at most 5, we have

(n+0.5)/65536 = D_K/10^K + N_K/(10^K \times 65536)

where actually N_K for those K considered is an integer, positive, and strictly less than 65536 (power of two considerations justify some of my assertions). The N_K are simply obtained by repeated multiplication by 10 modulo 65536. The equation above gives

65536 D_K/10^K = n + 0.5 - N_K/10^K

and if for any K we observe that 0< N_K <= 10^K (positivity is known), the rounding of 65536 D_K/10^K will give back the n as expected. This might happen for K = 1, ... or 4. Imagine it did not. Then it will necessarily at K = 5 but this case is handled especially. If we reach K = 5 it is because we had N_4 > 10000. Then let's M_5 = 10 N_4 - 50000 + 32768. It is > 65536, so we write M_5 = q 65536 + r where q is a digit. After simplification we obtain

n = 65536 (10 D_4 + q)/100000 + (r - 32768)/100000

and with D_5 = 10 D_4 + q which has a fifth digit, we again get a decimal D_5/100000 whose rounding as integral multiple of 1/65536 produces n/65536. (I will have to read again §103 of tex.web but this is my impression of what it boils down to).

We can follow the steps for n = 46616. First we compute 10(46616.5) = 466165 = 7x65536+7413 which gives use N_1 = 7413. It is not <= 10, so algorithm continues. First digit was 7.

Python 3.5.4 |Continuum Analytics, Inc.| (default, Aug 14 2017, 12:43:10) 
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> divmod(74130,65536)
(1, 8594)
>>> divmod(85940,65536)
(1, 20404)
>>> divmod(204040,65536)
(3, 7432)

thus digits are 7, 1, 1, 3, and here we reach N_4 = 7432 which is less than 10000. Algorithm stops here and TeX thus produces 0.7113pt.

From the description above you see it will never produce 6 digits after decimal mark such as 0.711315. To obtain this you must use a math engine (possibly implemented in macros; since e-TeX of 1999 with \numexpr it is easier to do this expandably, at least theoretically) able to handle decimal numbers like those.

You can use \xinttheiexpr [D] macro for such computations where D is the asked-for number of digits after decimal mark of final result, but computations are done exactly.

---

Going back to the quarter of millimeter (using Plain TeX, works in LaTeX too with \usepackage{xintexpr}):

\input xintexpr.sty

\xintdefvar mm:= 7227/2540;

\edef\MYQUARTERMM{\xinttheexpr trunc(0.25mm, 40)\relax}

\show\MYQUARTERMM

\bye

produces in the log

> \MYQUARTERMM=macro:
->0.7113188976377952755905511811023622047244.
l.7 \show\MYQUARTERMM

This shows by the way that the meaning of 0.711315 as arising in the original question isn't that clear.

From the above, an a priori closer 6 digits approximation is 0.711319pt. After checking one sees though that both 0.711319pt and 0.711315pt get converted by TeX into the internal representation 46617sp (\number\dimexpr 0.711319pt\relax, \number\dimexpr 0.711315pt\relax).

And \the\dimexpr 0.711319pt\relax outputs 0.71132pt. Thus 0.71132pt is the most accurate 5 digits representation (after testing it is the only one giving 46617sp internally) of the actual exact 0.25mm dimension. However as explained above, when TeX encounters 0.25mm it will obtain 46616sp. Using the bigger cm unit as in 0.025cm gives a greater loss of precision (46605sp).