0

A comparison of float calculation and exact expression for 

  NumberForm[0.3^2 - (3/10)^2, 18]

gives an exact zero for $0.3$. However,

R[n_Integer] = n 1./10.;
Rex[n_Integer] = n/10;
Table[R[n]^2 - Rex[n]^2, {n, 1, 5}]

gives

{1.73472*10^-18, 6.93889*10^-18, 2.77556*10^-17, 2.77556*10^-17, 0.}

Thus, here we get

0.3^2 - (3/10)^2 = 2.77556*10^-17

instead of zero. Why is the ouput different in the Table[] calculation?

anderstood
  • 14,301
  • 2
  • 29
  • 80
pawel_winzig
  • 1,577
  • 1
  • 12
  • 21
  • Is your question: why does 0.3^2 - (3/10)^2 return 0. while R[3]^2 - Rex[3]^2 returns 2.77556*10^-17? – anderstood Jul 20 '17 at 18:06
  • Related, maybe a duplicate: https://mathematica.stackexchange.com/questions/135636/understanding-simple-numerical-calculation – Michael E2 Jul 20 '17 at 18:30
  • This could be simplified considerably. It amounts to having done some arithmetic just to get the ".3" in R[3]: `In[17]:= {.3, (3*1./10)} // InputForm

    Out[17]//InputForm= {0.3, 0.30000000000000004}`

     – Daniel Lichtblau Jul 20 '17 at 18:45

0 Answers0