Pattern does not match with Orderless head

Question

Why does the following pattern matching not succeed?

ClearAll[f];
SetAttributes[f,Orderless];
HoldForm[f[a,d[],b,c]]/.HoldPattern[f[a,d[],b,z_]]:>{z}
(* f[a,d[],b,c] *)

I expected that because f is Orderless, the pattern matcher should try all possible orders of arguments and find a match.

If the expression is not held, I find:

ClearAll[f];
SetAttributes[f,Orderless];
f[a,d[],b,c]/.HoldPattern[f[a,d[],b,z_]]:>{z}
(* {c} *)

I.e. the desired result. I know that because f is Orderless and the expression is evaluated, the arguments are sorted into canonical order a, b, c, d[]. It would appear in this case that the pattern matcher really does try all possible orders of arguments to find a match.

Why doesn't it work in the first example?

EDIT:

Here are some further examples:

ClearAll[f];
SetAttributes[f, Orderless];

Hold[f[a, b, c, d[]]] /. HoldPattern[f[a, b, c, z_]] :> {"Matched", {z}}
(* Hold[{"Matched", {d[]}}] *)

Hold[f[a, b, c[], d]] /. HoldPattern[f[a, b, c[], z_]] :> {"Matched", {z}}
(* Hold[{"Matched", {d}}] *)

Hold[f[a, b[], c, d]] /. HoldPattern[f[a, b[], c, z_]] :> {"Matched", {z}}
(* Hold[f[a, b[], c, d]] *)

Hold[f[a[], b, c, d]] /. HoldPattern[f[a[], b, c, z_]] :> {"Matched", {z}}
(* Hold[f[a[], b, c, d]] *)

This seems very inconsistent to me; how can I understand what is happening here?

EDIT:

An illuminating example:

Hold[f[a, c, b[], d]] /. HoldPattern[f[a, b[], c, z_]] :> {"Matched", {z}}
(* Hold[{"Matched", {d}}] *)

The Trace shows no argument reordering, but it appears that the pattern matcher must actually be reordering the pattern to (the canonical order) f[a, c, b[], z_] because f is Orderless, even though HoldPattern should prevent "regular" evaluation.

What remains confusing is what exactly is meant by "In matching patterns with Orderless functions, all possible orders of arguments are tried." in the documentation. On face value, the four examples shown above contradict this.

Answer 1

I am betting that this is almost certainly an optimization short-cut. If Orderless had to try every ordering it would be extremely slow when there are a moderate number of arguments, but it is not.

Consider for example:

f @@@ Hold @@ {RandomSample[Range@12]}
Hold @@ {f @@ Range@12 /. {7 -> _}}
MatchQ[%%, %]

Hold[f[2, 6, 11, 7, 12, 10, 4, 1, 3, 5, 8, 9]]
Hold[f[1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 12, _]]
False

This test is surely not performing 479,001,600 comparisons, one for each permutation. Instead I suppose that pattern matching assumes that arguments of Orderless heads will first be placed in order and applies logic accordingly.

Since the first two of your "further examples" match I also suppose that this is because the sorted order of arguments on the RHS matches the verbatim order on the left; perhaps this case is tried first.

Answer 2

The following is an expansion of the explanation given by Mr.Wizard.

---

The pattern-matcher works on the base of the assumption that Orderless attribute is already applied and the arguments of the Orderless function are already sorted in the canonical order:

ClearAll[o]
SetAttributes[o, Orderless]
MatchQ[Hold[o[y, x, a]], Hold[o[_, x, a]]] (* unsorted arguments: {x, a} is NOT ordered *)
MatchQ[Hold[Evaluate@o[y, x, a]], Hold[o[_, x, a]]] (* Orderless attribute is applied *)

False
True

Instead of Hold we can use Unevaluated with the same effect:

MatchQ[Unevaluated[o[y, x, a]], o[_, x, a]] (* {x, a} is NOT ordered *)
MatchQ[Unevaluated[o[y, a, x]], o[_, x, a]] (* {a, x} IS ordered *)

False
True

Hold and Unevaluated prevent applying the Orderless attribute to the arguments of the Orderless function. They only affect the evaluation of the expression, and do not affect the pattern-matcher.

The pattern-matcher does not reorder the arguments of the Orderless function but takes the whole expression verbatim. It assumes that the arguments are already sorted as it happens by default for evaluated function with Orderless attribute. At the same time the pattern-matcher does sort the arguments of the pattern (with the exception of Blanks) regardless of whether the pattern is evaluated or not. Then it compares the sorted pattern with the expression putting Blank(s) in each position possible in the pattern.

In the light of this the Documentation statement

In matching patterns with Orderless functions, all possible orders of arguments are tried.

is misleading: actually only all possible positions of the Blanks in the pattern are tried but other arguments in the pattern are preliminarily sorted in the fixed canonical order. This is evident from the following:

MatchQ[Hold[o[y, a, x]], Hold[o[_, x, a]]] (* {a, x} IS ordered *)
MatchQ[Hold[o[y, x, a]], Hold[o[_, x, a]]] (* {x, a} is NOT ordered  *)
MatchQ[Hold[o[y, a, x]], Hold[o[x, _, a]]] (* {a, x} IS ordered *)
MatchQ[Hold[o[y, a, x]], Hold[o[x, a, _]]] (* {a, x} IS ordered *)

True
False
True
True

In short, Hold prevents applying Orderless attribute in the sense that it prevents sorting of the arguments in the canonical order, but it does not prevent action of this attribute in the pattern (i.e. all possible positions of the Blanks are indeed tried).