Except @ OptionsPattern[] affects OptionValue

Question

The background

This example works as expected, small function with a special case rule:

foo // ClearAll    
foo // Options = {"bar" -> 1};    

foo[x_, y : Except[_Rule], patt : OptionsPattern[]
] := {x, y, OptionValue["bar"]};

foo[x_, opts : OptionsPattern[]] := foo[x, "def", opts]

foo["xx", "bar" -> 3]

{"xx", "def", 3}

Except[_Rule] is needed because I don't want f["xx", "bar"->1] to go directly to the first definition and treat the option as y.

The problem

However, originally I used:

...
foo[x_, y : Except[OptionsPattern[]], patt : OptionsPattern[] ]:=...
...

but then

foo // ClearAll
foo // Options = {"bar" -> 1};

foo[x_, y : Except[OptionsPattern[]], patt : OptionsPattern[]
] := {x,  y, OptionValue["bar"]};

foo[x_, opts : OptionsPattern[]] := foo[x, "def", opts]

foo["xx", "bar" -> 3]

{"xx", "def", 1}

The question

Why wasn't the "bar" passed down?

I don't have time to investigate and will proceed with the workaround but would like to understand that as nothing obvious comes to my mind now.

Answer 1

Recall that OptionValue is a "magic" symbol with nonstandard behavior. The expression OptionValue["bar"] "by magic" gets its value without it being explicitly passed one, working with its "magic" brother OptionsPattern in a not-entirely-transparent way. I think it attaches to the first OptionsPattern[] object which is not used in the final matching with "bar" -> 3.

I don't yet have a clear idea of what is happening but I can give other examples of weird behavior.

exhibit A

ClearAll[foo]
Options[foo] = {"bar" -> 1};

foo[
  first : Except[OptionsPattern[]],
  second : OptionsPattern[]
] := {{first}, {second}}

foo["arg", "bar" -> 3]

{{"bar" -> 3}, {}}

• "arg" disappears entirely. Theoretically for the rule on foo to apply "arg" should be attached to either first or second patterns, but instead it vanishes as though attached to a third pattern expression

• "bar" -> 3 is attached to first, implying that it matches Except[OptionsPattern[]], despite the fact that MatchQ["bar" -> 3, Except[OptionsPattern[foo]]] returns False

exhibit B

Let's see what happens if we name the first appearance of OptionsPattern[] itself.

ClearAll[foo]
Options[foo] = {"bar" -> 1};

foo[
  first : Except[op1 : OptionsPattern[]], 
  second : OptionsPattern[]
] := {{first}, {second}, {op1}}

foo["arg", "bar" -> 3]

{{}, {}, {"bar" -> 3}}

• Once again "arg" goes missing, but this time "bar" -> 3 is not attached to first as it was before, leaving both first and second empty. Instead it is attached (only) to op1.

a note about Except

Except at times causes strange behavior. For example, I believe from an earlier question I cannot at the moment find:

Cases[{1, 0, 2, 0, 3}, Except[x : 0] :> x]

{Removed["$Variable"][1], Removed["$Variable"][1], Removed["$Variable"][1]}

Combining this somewhat peculiar pattern construct with the "magic" of OptionsPattern may simply be a bad idea at this time, unless and until this edge case is specifically addressed by the developers.

---

version differences

In version 10.1 from your second block of code I get:

In[6]:= foo["xx", "bar" -> 3]

During evaluation of In[6]:= OptionValue::rep: def is not a valid replacement rule. >>

Out[6]= {"xx", "bar" -> 3, 1}

Did I enter it wrong or has behavior changed?