N[..., 5] does not returns what I expected

Question

I have this

Xt = 
  {{1., 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1}, 
   {1, 1.5, 1.5,1.5, 2, 2, 2.2, 2.4, 2.5, 2.5, 2.8, 2.8, 3, 3, 3.2, 3.3}};
Yt = 
  {101.4, 117.4, 117.1, 106.2, 131.9, 146.9, 146.8, 133.9, 111.3, 123, 125.1, 145.2, 134.3, 144.5, 143.7, 146.9};

N[Inverse[Xt.Transpose[Xt]].Xt.Yt, 5]

and the last evaluation returns the same result as if there was no N[..,..]; i.e.,

{93.3422, 15.6485}

Why and how do I get output to five decimals?

Answer 1

The documentation for N says that N[expr, n] attempts to give a result with n-digit precision. In the Wolfram Language, or any language, precision is not the same as decimal places. While it might seem like N is there to give us results with a certain number of decimal places, it actually has more to do with the amount of precision used in calculations behind the scenes. It can be especially confusing as sometimes N does close to what we expect as in N[Pi, 10] which gives us $\pi$ with 10 digits (9 decimal places).

There is a different function called NumberForm that is designed to let us have more control over the display of numbers.

We can use:

NumberForm[Inverse[Xt.Transpose[Xt].Xt.Yt, {20, 5}]

yields

{93.34215, 15.64854}

The first number in NumberForm (i.e. 20) is the total number of digits. We can set this to Infinity and it won't make a difference, but it shouldn't be less than the total number of digits you want to display. The second number (i.e. 5) is the number of decimal places.

Answer 2

Here is one solution.

Xt = Rationalize@{{1., 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 
     1}, {1, 1.5, 1.5, 1.5, 2, 2, 2.2, 2.4, 2.5, 2.5, 2.8, 2.8, 3, 3, 
     3.2, 3.3}};
Yt = Rationalize@{101.4, 117.4, 117.1, 106.2, 131.9, 146.9, 146.8, 
    133.9, 111.3, 123, 125.1, 145.2, 134.3, 144.5, 143.7, 146.9};
Inverse[Xt.Transpose[Xt]].Xt.Yt

$\left\{\frac{892351}{9560},\frac{3740}{239}\right\}$

N[Inverse[Xt.Transpose[Xt]].Xt.Yt, 5]

{93.342, 15.649}

N[Inverse[Xt.Transpose[Xt]].Xt.Yt, 10]

{93.34215481, 15.64853556}

Answer 3

According to this tutorial:

MachinePrecision is considered less precise than any other precision. This is because, while machine-precision numbers always store slightly under 16 digits, no specific number of those digits are known to be correct.

We're working with MachinePrecision, i.e.

Precision[Inverse[Xt.Transpose[Xt]].Xt.Yt]

MachinePrecision

and so N[Inverse[Xt.Transpose[Xt]].Xt.Yt, 5] needs to somehow upsample the precision of the result and that is an ill-defined operation.

In fact the ref page for N states

Unless numbers in expr are exact, or of sufficiently high precision, N[expr, n] may not be able to give results with n-digit precision.

To see why this is ill-defined, consider the reverse. Let's go from higher precision to lower precision.

x1 = N[Pi, 10]

$3.1415926{\color{red}5}4$

x2 = N[Pi + 10^-8, 10]

$3.1415926{\color{red}6}4$

N[{x1, x2}, 5]

{3.1416, 3.1416}

So we see if we reverse this example, going from precision 5 to precision 10 is ambiguous in the choices of the 6th - 10th digits.

---

If you'd like to upsample in a consistent way, use SetPrecision:

When SetPrecision is used to increase the precision of a number, the number is padded with zeros. The zeros are taken to be in base 2. In base 10, the additional digits are usually not zeros.

SetPrecision[Inverse[Xt.Transpose[Xt]].Xt.Yt, 5]

{93.342, 15.649}

If you only care about the display of the answer, consider using NumberForm:

NumberForm[Inverse[Xt.Transpose[Xt]].Xt.Yt, 5]

{93.342, 15.649}