Split a Unicode string maintaining uppercase characters

Question

I want to split a string according to a predefined set of substrings (lowercase), though the actual text can contain uppercase characters anyplace. The task is to find the matches, longer preferred over shorter (like "tt" over "t" hence the order of alt) and to maintain upper/lowercase status. Since StringCases has a quirk not being able to correctly recognize some upper/lowercase characters (cf. here), the solution is not trivial.

The following code matches every element in alt correctly, though it also converts everything to lowercase:

alt = "tt" | "t" | "a" | "á";
word = "TtattÁatT";
StringCases[ToLowerCase@word, alt]

{"tt", "a", "tt", "á", "a", "tt"}

The next one, while correctly maintains capital letters, fails to recognize "Á" as the uppercase version of "á" (or of a, see later).

StringCases[word, alt, IgnoreCase -> True]

{"Tt", "a", "tt", "a", "tT"}

The problem here is that StringMatchQ returns False:

StringMatchQ["A", "a", IgnoreCase -> True]       (* ==> True *)
StringMatchQ["Á", "á", IgnoreCase -> True]       (* ==> False *)
StringMatchQ["Á", "a", IgnoreCase -> True]       (* ==> False *)
StringMatchQ["Á", "á" | "a", IgnoreCase -> True] (* ==> True *)
(* for the record *)
ToLowerCase@"Á" === "á"       (* ==> True *)
ToUpperCase@"á" === "Á"       (* ==> True *)

I think it actually is a bug, as this behaviour is rather inconsistent. Nevertheless, the next code I came up with to match "Á" fails:

StringCases[word, __?(StringMatchQ[#, alt, IgnoreCase -> True] &), IgnoreCase -> True]

{"Ttatt", "atT"}

It cannot recognize "Á", and it splits by the longest match instead of exact matches.

Is it possible to overcome somehow this issue without explicitly listing all combinations of upper/lowercase letters (like "a"|"A"|"á"|"Á"|"tt"|"tT"|"Tt"|...) or saving and restoring character states (upper/lowercase) manually? There are nice examples how the string patternmatcher can be used effectively, I hope this can be solved as well.

Question

Given a possibly large list of short elementary strings like "tt" | "t" | "a" | "á", how can a larger string be split into elementary substrings with keeping all uppercase letters correctly. For example:

"TtattÁatT" == > {"Tt", "a", "tt", "Á", "a", "tT"}

Can anyone comfirm/explain the bug-like behaviour reported above? — István Zachar, Apr 02 '12 at 09:53
This may be related to problems reported in this question, and duscussed also in comments below it. — Leonid Shifrin, Apr 02 '12 at 13:19

score 15 · Accepted Answer · answered Apr 01 '12 at 21:41

15

You can do the lower case conversion as a condition on the pattern, and thereby retain the original:

StringCases["TtattÁatT", c__ /; MemberQ[List@@alt, ToLowerCase[c]]]

{"Tt", "a", "tt", "Á", "a", "tT"}

answered Apr 01 '12 at 21:41

wxffles

14,246
1
43
75

Ingenious! Though I wonder, why these two give different results: StringCases["TtattÁatT", c__ /; MemberQ[List @@ alt, ToLowerCase[c]]] and StringCases["TtattÁatT", __?(MemberQ[List @@ alt, ToLowerCase[#]] &)] – István Zachar Apr 01 '12 at 22:00
4

For a StringExpression the ? test works character by character rather than on the string as a whole. – wxffles Apr 01 '12 at 22:14
1

@Istvan Probably because PatternTest tests each entry in the sequence individually, while Condition applies to the entire sequence / substring (I did not test, but this is what I'd think). – Leonid Shifrin Apr 01 '12 at 22:14

score 9 · Answer 2 · answered Apr 01 '12 at 22:40

9

Here is an alternative which might be somewhat faster on large strings:

StringTake[word, StringPosition[ToLowerCase@word, alt, Overlaps -> None]]

{"Tt", "a", "tt", "Á", "a", "tT"}

answered Apr 01 '12 at 22:40

Leonid Shifrin

114,335
15
329
420

Split a Unicode string maintaining uppercase characters

2 Answers2

Linked