Correct way to handle mysterious NaN` result from MathLink function

Question

I have a Mathematica expression that is mapped onto an external C function via MathLink. The external function passes a double array (using MLPutReal64List[]), which Mathematica interprets as a list of Real's. Sometimes the external function sends values for which the C math library function isnan(value) returns 1 (say from division by zero, or log(0)). Mathematica reads these values as NaN`. For example,

In[1]:= result = externalFunction[<some input>]
Out[1]= {NaN`, 0.18225}

Evaluating {NaN`, 0.182255} gives a syntax error:

In[2]:= {NaN`, 0.182255}
Syntax::tsntxi: "{NaN`,0.182255}" is incomplete; more input is needed.
Syntax::sntxi: Incomplete expression; more input is needed.

which makes sense, since a variable name is expected to follow the `, giving some variable in the NaN context.

But mathematica accepts result[[1]] as a number:

In[3]:= NumericQ[result[[1]]] 
Out[3]= True
In[4]:= NumberQ[result[[1]]]
Out[4]:= True

!!

Yet I haven't been able to find a pattern that matches NaN` and not all other numbers (for use in Position, Cases, etc.).

So what are ways of a) passing NaN's from c into Mathematica so that Mathematica interprets them as Indeterminate, or b) handling these NaNs inside mathematica (with a pattern match to replace them with Indeterminate, for example).

Is it possible to construct the first element of result using keyboard input? Note that InputForm[result[[1]]] gives NaN, and NumericQ[InputForm[result[[1]]]] gives False.

Answer 1

NaN (or Not-a-Number is used in floating point arithmetic to represent values that are undefined or unrepresentable, such as $0/0,\ \infty/\infty$, etc. Mathematica typically returns Indeterminate for these, but several other languages return NaN.

To work with NaNs, you must load the ComputerArithmetic package as <<ComputerArithmetic` prior to calling your external function. If you don't do so, then Mathematica will treat the NaNs like any other symbol (or perhaps with other, unknown consequences depending on the setup). Loading the package will give you the results as expected, and pattern matching is pretty straightforward too.

Answer 2

This perhaps more of a comment than an answer, but I'm putting it in answer form so it will have more visibility.

The IEEE NaN number is not supposed to existing in the WL. That value is represented by the symbol Indeterminate. No WL computational function should ever produce an IEEE Nan, and I'm not aware of any bugs in which this is the case.

Now, when we allow external processes to directly write into our memory, e.g., MathLink or LibraryLink, there is some chance that an IEEE Nan will end up in the system. When that happens, you get this strange NaN` thing. This is the text formatting of the numeric value followed by the machine precision mark, just like N[6/5] will give you 1.2` . The formatting is an accident of the implementation of something that shouldn't exist leaking into the system.

As noted in several comments, the expression NaN` is not the same thing as the NaN symbol of the old ComputerArithmetic package, which is merely a symbolic representation of the IEEE value for the purposes of the package.

So why don't we prevent these from leaking in, and/or have better tools for dealing with them? Well, there are all sorts of complicated tradeoffs. For example, external packages often write large arrays directly into memory. If we stopped to check every single real number that came in, it would enormously slow down the system, defeating the whole purpose of allowing direct memory writes. OTOH, since the core system was written on the assumption that this value can't be created, it is not easy propagate it now.

With NumericArray making it easier for such things to enter the system, we've done some experiments and how to improve the situation. But we haven't yet reached a point where we have a clearly better story to tell, so in the mean time we're sticking with the status quo rather than randomly change things. If and when we have a better store to tell, I'll update this answer.

Answer 3

Recently, I revisited this thread for dealing with NumericArrays produced via numpy where numpy.nan will be embedded as NAN`, which can be simply generated using the following code:

cNaN = ExternalEvaluate["Python", 
    "import numpy as np; np.array([np.nan])"] // Normal // First

And I found a caveat in Barry's approach that a number close to -9223372036854775808 (on 64 bit system) will be misrecognised as NaN.

Meanwhile, I also noticed Threshold[{cNaN}] gives {0.}; so a more robust approach for testing NAN` could be like this:

Clear[NaNQ]
NaNQ[nan_?NumericQ]:=Clip[Abs[Threshold[{nan}]-Round[{nan}]],{1,1},{0,Indeterminate}]==={Indeterminate}

Based on the above approach, it is also possible to have more convenient functions to deal with arrays/lists including intermittent NaN`s:

(*replace NaN with Indeterminate*)
Clear[replaceNaN]
replaceNaN[x_List/;Length[x]>0]:=Clip[Abs[Threshold[x]-Round[x]],{1,1},{0,Indeterminate}]+Threshold[x]
replaceNaN[{}]:={}

and

(*fully remove NaN*)
Clear[removeNaN]
removeNaN[x_List]:=replaceNaN[x]/.Indeterminate:>Nothing

Answer 4

The best way to handle NaNs coming in from MathLink is to replace all NaNs with a symbol such as Indeterminate. One way this can be achieved is using the following piece of code:

NaNQ[x_] :=
 Round[x] == -2147483648;

ReplaceNaNs[x_] := x /. y_?NaNQ :> Indeterminate;

You can then use the ReplaceNaNs function to replace any NaNs that you don't want:

In[1]:= result = externalFunction[<some input>]
Out[1]= {NaN`, 0.18225}
In[2]:= ReplaceNaNs[result]
Out[2]= {Indeterminate, 0.18225}

Note when using 64 bit values the Round[] limit needs to be -9223372036854775808.