Version inconsistency with optional arguments: what if the default value doesn't match the pattern?

Question

I just came across some weird behaviour. Take this function definition:

ClearAll[f]
f[vs_List : All] := "match"

The default value of vs is All. Now think about what f[] should return. Should it evaluate? You might say yes, vs will just take the default value. But you might say, wait, All doesn't even match _List so this is nonsense!

It turns out that different versions behave differently:

Version 9.0.1:

{$VersionNumber, $ReleaseNumber}
f[]
(* {9., 1} *)
(* f[] *)

Version 10.0.2:

{$VersionNumber, $ReleaseNumber}
f[]
(* {10., 2} *)
(* f[] *)

Version 10.2.0:

{$VersionNumber, $ReleaseNumber}
f[]
(* {10.2, 0} *)
(* "match" *)

Version 10.3.1:

{$VersionNumber, $ReleaseNumber}
f[]
(* {10.3, 1} *)
(* "match" *)

I don't have version 10.1.0 to try with.

Question: Is there a bug somewhere? Which should be the correct behaviour?

---

Ultimately this is of course a GIGO situation because the pattern arguably doesn't make sense. The solution is easy: just use f[vs : (_List | All) : All] := "match". But I made a mistake and used _List : All instead. Then much later I discovered that my test suite failed in version 10.0 while it was passing in 10.3.

What would be the most user-friendly behaviour in my opinion is for Mathematica to show a warning when it encounters this situation. Either way, it would be nice to have a mention of this behaviour in the documentation.

Answer 1

Update: Daniel Lichtblau authoritatively comments:

This change was intentional, and per request of the boss.

---

I can find no mention of this in the documentation, though I am still looking. My guess is that the old behavior was a common source of problems and someone's fix was to implement the new behavior in version 10.1.0.

On the surface at least I like the change as it shortens and simplifies code. A notable difference is that with the new short scheme the default value is not seen as a valid explicit argument:

f[All]

   (* Out:   f[All]   -- 10.1.0 under Windows *)

I think this actually may prove useful but I can also imagine it being a new source of confusion.

---

Perhaps this change makes behavior more consistent than it was in the past.
Consider this behavior in older versions (here v7):

ClearAll[f, val]

f[vs_List: val] := vs
val = {1, 2, 3};
f[]

   (* Output:   f[]   -- no match *)

ClearAll[f, val]

f[vs : (_List | HoldPattern[val]) : val] := vs
val = "foo";
f[]

   (* Output:   "foo"   *)

I find both cases unexpected even if explainable. The new more permissive behavior makes these cases more similar as both evaluate using the current value of val.

---

Breaking change

There appears to be a serious caveat to this change that I previously failed to note. In Mathematica 7 pattern Symbols are bound to expressions in the default:

f[args : {x_, y_} : {1, 2}] := {x, y}

f[]

{1, 2}

In version 10.1 this is no longer the case:

f[args : {x_, y_} : {1, 2}] := {x, y}

f[]

 {}

Reference comments by Itai Seggev in

• Setting nested optional argument with a default when unpacking from a given Head

Answer 2

In my opinion, Optional should just consistently not match when the default value does not match the pattern, period.

Now we have nonsense like this:

{}~MatchQ~{Optional[0., 0]}

False

You might think this is because 0~MatchQ~0. === False.

But then why is

{}~MatchQ~{Optional[_Real, 0]}

True

when 0~MatchQ~_Real is False?

What is the rule here?

---

The situation becomes even more confusing when you start playing with HoldPattern (for doing things like this):

G[0] := 0
Gh[0] := h@0
{}~MatchQ~{HoldPattern@Optional[_[_], G[0]]}

{}~MatchQ~{HoldPattern@Optional[_?AtomQ, G[0]]}
{}~MatchQ~{HoldPattern@Optional[0, G[0]]}
{}~MatchQ~{HoldPattern@Optional[_?AtomQ, 0]}

{}~MatchQ~{HoldPattern@Optional[h@_, Gh[0]]}
{}~MatchQ~{HoldPattern@Optional[h@_, h@0]}

True

True

False

True

False

True

I would expect to get True for all but the first of these, which is what you get when dropping HoldPattern.

---

It feels like it is not well specified what Optional should do.