List damaged after update with Nothing

Question

In version 10.4.1 (Win 7 Ent) I am implementing an algorithm that needs to update a list after each pass. I initially coded this with Nothing by removing items in the list that where no longer needed after each processing pass. This did not result in the expected outcome and I eventually sourced the error to the use of Nothing. A very minimal example follows.

SeedRandom[19384];
datN = RandomInteger[{1, 5}, 10]
(* {2, 1, 3, 5, 1, 2, 5, 2, 2, 2} *)

For this minimal example FirstPosition[5]@datN will represent the index of the item that needs to be removed after each processing pass. (I'm not trying to delete all 5's in the real code.) There are two 5's. I would expect the first in position 4 to be removed at the end of the first pass.

datN[[Echo@First@FirstPosition[5]@datN]] = Nothing;
datN
>> 4
(* {2, 1, 3, 1, 2, 5, 2, 2, 2} *)

Success! The first pass finds the first 5, removes it, and updates datN. Or does it? Lets try the second pass which should remove the 5 in position 6.

datN[[Echo@First@FirstPosition[5]@datN]] = Nothing;
datN
>> 6
(* {2, 1, 3, 1, 5, 2, 2, 2} *)

What has happened here? The correct position of 6 was identified but instead of removing the 5 there it removed the 2 in position 5. Let us try one more time to remove the 5 now in position 5.

datN[[Echo@First@FirstPosition[5]@datN]] = Nothing;
datN
>> 5
(* {2, 1, 3, 5, 2, 2, 2} *)

It has done it again. The correct position of 5 has been identified but the 1 in position 4 has been removed instead.

Now it gets interesting. On the fourth try and every subsequent try after the fourth it simply refuses to remove anything.

datN[[Echo@First@FirstPosition[5]@datN]] = Nothing;
datN
>> 4
(* {2, 1, 3, 5, 2, 2, 2} *)

The output above will repeat with the datN never being altered from the forth try onwards. Very odd.

Have I completely missed something or is this a bug? If so, is it still around in version 11?

Update

Clearly I have to live with it as it is. However, at this moment, I don't feel it is intuitively correct. I would understand the hold behavior if I used SetDelayed but having used Set it seems a reasonable expectation that the Nothings be removed. At least I understand its mechanism. I am dating myself a bit here but I was expecting a ReDim Preserve type of behavior. 10 Internet points to anyone that gets that reference without looking it up.

Answer 1

This is not a bug.

You are assuming that

arr = {1,2,3}
arr[[2]] = Nothing

removes the second element. It doesn't! It sets the second element to Nothing, just the same as if you used any other symbol.

What happens when you evaluate arr now is the following evaluation steps:

arr -> {1,Nothing,3}
{1,Nothing,3} -> {1,3}

However, the length of arr is still 3.

You can see this with

Definition[arr]
(* arr = {1, Nothing, 3} *)

If you want to remove that element, then you need the extra step

arr = arr

---

Update: I see the Edmund's comment was upvoted by several people, which shows that there's a widespread misunderstanding about what Nothing does. The documentation states that

Nothing is removed as part of the standard evaluation process.

That is, it is removed during evaluation when it appears explicitly as part of a list. It does not however interact in any way with Set or Part. arr[[2]] = Nothing does not behave in a special way. It just sets the second element of the array to Nothing.

Now when you evaluate arr, it goes through two evaluation steps, as I stated above, first into {1,Nothing,3} which then evaluates further into {1,3}. Thus the result of the evaluation of arr is a length-2 list, and Length[arr] returns 2. This does not mean that arr itself is a length-2 list. It only means that it evaluates to a length-2 list. You can see with OwnValues[arr] that it has 3 elements, the second being Nothing.

Answer 2

Citing the Documentation (emphasis is mine):

Nothing is removed as part of the standard evaluation process. It is not removed in expressions that are held or inactive.

So it should be removed upon evaluation, what actually happens, and there is no statement that it should be removed from the original unevaluated list.

Answer 3

I don't think this is exactly a bug. But it is at minimum a woefully underdocumented feature, and identical to one that shows up when using Sequence[]. If you use Information you will get an idea of what is happening.

datN[[First@FirstPosition[5]@datN]] = Nothing;
datN

(* Out[891]= {2, 1, 3, 1, 2, 5, 2, 2, 2} *)

Information[datN]
(* Global`datN
datN={2,1,3,Nothing,1,2,5,2,2,2} *)

Now repeat:

datN[[First@FirstPosition[5]@datN]] = Nothing;
datN

(* Out[912]= {2, 1, 3, 1, 5, 2, 2, 2} *)
In[913]:= Information[datN]

(* Global`datN
datN={2,1,3,Nothing,1,Nothing,5,2,2,2} *)

The problem is as follows. First@FirstPosition[5]@datN evaluates datN, hence gets it's positioning "correct", that is, for the list with the Nothing removed. Part, when used as a left-hand-side of an expression (what we call "setpart"), is not actually evaluating datN. This is a mismatch.

I am not sure if there is a sound reason for this behavior of setpart. It might be necessitated by the HoldFirst attribute of Set. This behavior goes back to version 1 I believe, so I'd be quite reluctant to mess with it even if Set semantics might allow for that.