Can a pattern examine all partitions of a sequence?

Question

Say I have a list: {a, b, c, d}. I'd like to check that the list has some partition, say {{a, b, c}, {d}} such that all the elements of the partition satisfy a predicate (or match a pattern). So in this case, both {a, b, c} and {d} would match. Can this be done purely with patterns? I notice that all sequence patterns with predicates apply only to individual elements of the sequence.

More concrete example:

I have --

formQ[x___] := (listQ[x] == True) || (atomQ[x] == True)
listQ[token["("], middle, token[")"]] := (formQ[middle] == True)
atomQ[x_token] := x =!= token["("]  && x =!= token[")"]

(of course instead of middle I'd have this pattern I'm looking for)

So for example,

listQ[token["("], token["3"], token[")"]] == True
listQ[token["("], token["3"], token["("], token[")"], token[")"]] == True
formQ[token["3"]] == True
formQ[token["("], token[")"]] == True
formQ[token["("], token[")"], token["3"]] == False

Basically I'm messing around with a scheme-like parser... I already have the tokenizer working, and I'm seeing if I can recognize elements like "forms" and "lists" and "atoms".

A form is something like "(3 4 5)" or "( ( ) ( 5 ) )" but not "( ) )" or "3 4" or "3 ( ) 4". The reason I need to be able to match multiple forms is that a list can contain multiple atoms or sublists, all of which are forms.

EDIT: ReplaceRepeated is helpful, working from the inside out, but doesn't answer the question.

readTokens[x_String] := Map[token, StringCases[x, 
    RegularExpression[" *(\\(|\\)|[^ \\(\\)]+) *"] -> "$1"]]
atomQ[x_token] := x =!= token["("]  && x =!= token[")"]
listQ[x___] := ReplaceRepeated[{x}, {a___, token["("], q___?atomQ, token[")"], 
    b___} :> {a, b}]  == {}

So we have

listQ @@ readTokens["((a b c)))"] == False
listQ @@ readTokens["((a b c))"] == True

score 3 · Answer 1 · edited Apr 13 '17 at 12:55

I have trouble following your example (in the comment) so I shall attempt to answer this question generically. If you can give a few examples of actual input and output that you expect from listQ I shall attempt to implement it.

Individual elements of of pattern sequence are tested when you use PatternTest, but the entire sequence can be easily tested if you use Condition. See:

Using a PatternTest versus a Condition for pattern matching

Considering your target partition {{a, b, c}, {d}} observe:

Replace[
 {7, 2, 4, 3},
 {x__, y__?OddQ} /; Plus[x] == 13 :> {{x}, {y}}
]

{{7, 2, 4}, {3}}

This checks to see that the sum of the sequence of elements bound to x is 13, and that any remaining elements individually pass OddQ. Additional examples of match and non-match of this pattern:

Replace[
 {5, 3, 5, 7, 1},
 {x__, y__?OddQ} /; Plus[x] == 13 :> {{x}, {y}}
]

{{5, 3, 5}, {7, 1}}  (* match and replace *)

Replace[
 {5, 3, 5, 2, 2},
 {x__, y__?OddQ} /; Plus[x] == 13 :> {{x}, {y}}
]

{5, 3, 5, 2, 2}  (* no match, no replace *)

You provided an additional example:

As a simplified example let's say I want a function to return True every time it sees a list of the form {a, b, c, ...} where it can break the list into groups that add up to 13. So it would work with {13, 13} or {10, 3, 13} or even {10, 3, 10, 3}, but not {13, 5, 13}.

This case is tricky because naming a pattern forces any appearances of that pattern to match each other. Therefore if you use a Repeated pattern like {(x__ /; Plus[x] == 13) ..} it will match {10, 3, 10, 3} since the partitions are identical, but it will not match {10, 3, 13} because the partitions would be different from each other. A solution is recursion, which can even be done within patterns, though it is arguable that this is not a self-contained pattern:

p1 = {x__ /; +x == 13, y___} /; MatchQ[{y}, p1 | {}];

MatchQ[
 {10, 3, 10, 3, 13, 2, 4, 7},
 p1
]

True

(+x is simply shorthand for Plus[x].)

@epylar Please used the edit link under your question to add detail. Thank you. — Mr.Wizard, Mar 09 '15 at 00:33
As a simplified example let's say I want a function to return True every time it sees a list of the form {a, b, c, ...} where it can break the list into groups that add up to 13. So it would work with {13, 13} or {10, 3, 13} or even {10, 3, 10, 3}, but not {13, 5, 13}. — epylar, Mar 09 '15 at 00:35
@epylar The problem in your first comment is actually rather tricky. I am trying to remember the right approach to that. — Mr.Wizard, Mar 09 '15 at 00:56
Obviously it can be done without patterns, but if there is a way to do it with patterns, it seems worth knowing. Also if you are allowed to be "greedy" -- that if you find a 10 and a 3 you're allowed to use that as a 13, and then only concern yourself with the rest of the list, you could use repeated replace. — epylar, Mar 09 '15 at 00:58
@epylar Yes, recursion "eating" from left to right seems like the natural approach but it is not purely pattern-based. I think I vaguely remember someone posting a solution but I cannot recall it now. — Mr.Wizard, Mar 09 '15 at 00:59
@epylar I made an update to include this. I do not know if you will find it satisfying. I am very tired right now but I shall return to your updated parser example at a later time, if someone else has not already satisfactorily addressed it by then. — Mr.Wizard, Mar 09 '15 at 01:04

WReach · Accepted Answer · 2015-03-09T04:53:50.203

If I understand the intent correctly, then the following definitions should recognize valid forms:

ClearAll[formQ, listQ, atomQ, sequenceQ]

formQ[x___] := listQ[x] || atomQ[x]

atomQ[token[Except["("|")"]]] := True
atomQ[___] := False

listQ[token["("], x___, token[")"]] := sequenceQ[x]
listQ[___] := False

sequenceQ[] := True
sequenceQ[x__ /; formQ[x], y___ /; sequenceQ[y]] := True
sequenceQ[__] := False

The definitions of formQ and atomQ are essentially unchanged, with some minor tweaks for failure cases.

The main change is to the definition of listQ. A new helper function, sequenceQ has been introduced. Its purpose is to recognize a (possibly empty) sequence of valid forms. sequenceQ performs the "partitioning" requested in the question. It will extract a leading valid non-empty form, and then verify that what remains is a valid sequence. Condition (/;) is used in place of PatternTest (?) since, as noted in the question, the latter operates upon individual matched sequence elements instead of the whole matched sequence.

Be Careful Not To Defeat Back-Tracking

There is a subtlety in this definition. It is crucially important that the formQ and sequenceQ checks appear within the pattern itself and not within the body of the definition -- the following will not work:

(* do not do this! it disables back-tracking *)
sequenceQ[x__, y___] := formQ[x] && sequenceQ[y]

When the conditions appear in the pattern, the pattern matcher will back-track if the tests fail. In the absence of those tests, the pattern matcher will think that there is no need to back-track since it has found a successful match and gone ahead to execute the body of the definition.

It would be acceptable to move the condition onto the right-hand side like this:

(* this will work *)
sequenceQ[x__, y___] := True /; formQ[x] && sequenceQ[y]

On the face of it, the construction True /; ... might seem to be an awkward and unnecessary rephrasing of the previous expression. But do not be fooled: the meaning is quite different. Conditions (/;) are integrated into the pattern matching process, but tests in the main definition body are not.

Conditions of this form are often expressed thus:

(* this will work too *)
sequenceQ[x__, y___] /; formQ[x] && sequenceQ[y] := True

(* and so will this *)
sequenceQ[x__ /; formQ[x], y___ /; sequenceQ[y]] := True

Choose the form that suits your taste.

What About Back-Tracking In The Other Definitions?

formQ and listQ perform checks in their definition bodies in opposition to the advice just given. I left them that way for continuity from the original question. It so happens that those definitions do not require internal back-tracking to function correctly. In recognizers, I recommend adopting a policy of putting recognition checks in pattern conditions as a matter of course. Under such a policy, tests appear in definition bodies only when explicit back-tracking "cuts" are desired.

The definitions can be made to conform to this suggestion like this:

ClearAll[formQ, listQ, atomQ, sequenceQ]

formQ[x___] /; listQ[x] || atomQ[x] := True
formQ[___] := False

listQ[token["("], x___, token[")"]] /; sequenceQ[x] := True
listQ[___] := False

sequenceQ[] := True
sequenceQ[x__, y___] /; formQ[x] && sequenceQ[y] := True
sequenceQ[__] := False

atomQ[token[Except["("|")"]]] := True
atomQ[___] := False

Test Cases

All of the following test cases evaluate to True:

True === atomQ[token["3"]]
True === listQ[token["("], token["3"], token[")"]]
True === listQ[token["("], token["3"], token["("], token[")"], token[")"]]
True === formQ[token["3"]]
True === formQ[token["("], token[")"]]
True === formQ[token["("], token["("], token["("], token[")"], token[")"], token[")"]]
True === formQ[token["("], token["("], token[")"], token["("], token[")"], token[")"]]

False === formQ[token["("]]
False === formQ[token["("], token[")"], token["3"]]
False === formQ[token["("], token["1"], token["2"]]
False === formQ[token["1"], token["2"]]
False === atomQ[token["1"], token["2"]]
False === atomQ[]
False === listQ[]
False === formQ[]

Wow, very interesting. I realize now that most of the intent behind my question was to figure out how to use patterns to do backtracking. Thanks. — epylar, Mar 09 '15 at 14:17
I can confirm this actually does backtrack -- I defined a simpler version of this where a valid form is either ("blue") or ("green") or ("blue", "green") or ("green", "red") and asked whether ("blue", "green", "red") was a valid sequence. If it had insisted on matching "blue" (and possibly "green") and not backtracking, it would have seen "red" and returned False. But it returned True. — epylar, Mar 09 '15 at 14:34

Can a pattern examine all partitions of a sequence?

2 Answers2