2

With the following example

\documentclass{article}

\begin{document}

\dimen0=1sp \dimen1=0.5\dimen0 \dimen2=0.99999\dimen0
\dimen3=0.999992\dimen0 \dimen4=0.999993\dimen0
\count255=\dimen0 \the\count255 sp ~
\count255=\dimen1 \the\count255 sp ~
\count255=\dimen2 \the\count255 sp ~
\count255=\dimen3 \the\count255 sp ~
\count255=\dimen4 \the\count255 sp ~

\end{document}

I got the following result

1sp 0sp 0sp 0sp 1sp

I cound not understand how TeX does its arithmetic calculation.

Z.H.
  • 3,346
  • 21
  • 39
  • 1sp is the smallest unit of length in TeX. Everything smaller than that is defined by rounding and 0.999993 is apparently rounded to 1. – Henri Menke Jun 30 '18 at 03:34
  • 1
    Take a look at this answer and this answer by @jfbu – Ruixi Zhang Jun 30 '18 at 03:40
  • 1
    I guess the question is about 0.999993 being rounded to 1. The answer to that question by @egreg is even more relevant. (TeX basically uses fixed-point arithmetic with 16 bits after the point, i.e. it multiplies everything by 65536 and works with integers.) Working it out: for converting 0.999993 into binary, start with a = 0, then for each digit 3, 9, ... etc, set a = (a + d * 131072) div 10. So: in case of 0.999993, a takes the values 39321, …, 131071, after which a = (a + 1) div 2 results in a = 65536, i.e. 1. In the case of 0.999992 we end up with a = 131070, so 0. – ShreevatsaR Jun 30 '18 at 04:02
  • 1
    Or to put it more simply: in fixed-point arithmetic with 16 bits of precision, the fractional value just below 1.0 (=65536/65536) is 65535/65536 = 0.9999847412109375. And the number 0.999992 is closer to this than to 1 (and is rounded to this), while the number 0.999993 is closer to 1 than to this (and is rounded to 1). – ShreevatsaR Jun 30 '18 at 04:16

1 Answers1

2

This answer by @jfbu explains how TeX does its algebra.

For 0.999992 or less

Step 1. 0.999992*65536 = 65535.475712, which is then rounded to 65535 or less.

Step 2. Conversion: 65535/65536, which must then be rounded down to zero!

For 0.999993 or slightly more

Step 1. 0.999993*65536 = 65535.541248, which is then rounded to 65536!!!

Step 2. Conversion: 65536/65536, which must then be rounded down to one!

Community
  • 1
Ruixi Zhang
  • 9,553