A simple StringCases problem

Question

Given this text:

text1 = "  A  Vitamin D Deficiency (ICD-9-CM 268.9) (ICD-9-CM 268.9) 09/11/2015  01 ";

Yes, the ICD code values are duplicates- possibly but not certainly in all the records.

What string expression will match only the code values - twice if duplicated? I've tried:

StringCases[text1, "(ICD-9-CM " ~~ code__ ~~ ")" .. :> code, 
 Overlaps -> False]

(* {"268.9) (ICD-9-CM 268.9"} *)

StringCases[text1, "(ICD-9-CM " ~~ code__ ~~ ")" .. :> code,   Overlaps -> True]

(* {"268.9) (ICD-9-CM 268.9", "268.9"} *)

StringCases[text1, "(ICD-9-CM " ~~ code__ ~~ ")" .. :> code, 
 Overlaps -> All]

(* {"268.9) (ICD-9-CM 268.9", "268.9", "268.9"} *)

Removing the .. repeat pattern doesn't alter the outputs. What's the explanation for this behavior?

Answer 1

A couple of details:

• Restricting code__ to NumberString will prevent it from being greedy (else it might stop only at the second ))
• You need to wrap the entire pattern (which is what we want to repeat) in parentheses to respect the precedence of the .. operator.

---

The following pattern works:

StringCases[text1, ("ICD-9-CM " ~~ code : NumberString) .. :> code]
(* {"268.9", "268.9"} *)

Use ToExpression@code if you want them as numbers instead of a string.

Answer 2

You could simply find the shortest match:

StringCases[text1, "(ICD-9-CM " ~~ Shortest[code__] ~~ ")" :> code]

{"268.9", "268.9"}

If it is possible that there is additional space or other characters a combination may be more robust:

text2 = "  A  Vitamin D Deficiency (ICD-9-CM   268.9) (ICD-9-CM: 268.9) 09/11/2015  01 ";

StringCases[text2, Shortest["(ICD-9-CM" ~~ __ ~~ code : NumberString ~~ ")"] :> code]

{"268.9", "268.9"}

---

Albert Retey suggested a method scanning for Except[")"] to restrict the pattern to a single field. I believe this is equivalent to my use of Shortest, but I was curious to find out how the two compared. We can view the Regular Expression that Mathematica internally converts each StringExpression to with StringPattern`PatternConvert. I will include two placements of Shortest.

StringPattern`PatternConvert /@ {
   "(ICD-9-CM " ~~ Shortest[code__] ~~ ")",
   Shortest["(ICD-9-CM " ~~ code__ ~~ ")"],
   "(ICD-9-CM " ~~ code : (Except[")"] ..) ~~ ")"
 } // Column

{"(?ms)\\(ICD-9-CM (.+?)\\)", {{Hold[code], 1}}, {}, Hold[None]}
{"(?ms)\\(ICD-9-CM (.+?)\\)", {{Hold[code], 1}}, {}, Hold[None]}
{"(?ms)\\(ICD-9-CM ((?:[^)])+)\\)", {{Hold[code], 1}}, {}, Hold[None]}

We see that both Shortest placements produce the same RE. We also see that the method using Except is more complex, and would appear to be performing additional testing that is ultimately unnecessary. I would expect it to be slower, and it is, but only slightly:

textBig = StringJoin @ ConstantArray[text1, 1*^6];

StringCases[textBig, Shortest["(ICD-9-CM " ~~ code__ ~~ ")"] :> code] // Timing // First

StringCases[textBig, "(ICD-9-CM " ~~ code : (Except[")"] ..) ~~ ")" :> code] // 
  Timing // First

0.718

0.812

Interestingly the hand-written RE is quite a bit slower:

StringCases[textBig, RegularExpression["\\(ICD-9-CM\\s*([^\\)]*)\\s*\\)"] :> "$1"] // 
  Timing // First

1.872

I started a new question about this here: Why is StringExpression faster than RegularExpression?

Answer 3

With RegularExpression you could use:

text1 = "  A  Vitamin D Deficiency (ICD-9-CM 268.9) (ICD-9-CM 268.9) 09/11/2015  01 ";
StringCases[text1,RegularExpression@"\\(ICD-9-CM (\\d+(\\.)?(\\d+)?)\\)"-> "$1"]

Answer 4

Regular Expressions

This is typically solved with pattern matching. You specify a pattern by creating a regular expression (or short regex) using RegularExpression.

text1="  A  Vitamin D Deficiency (ICD-9-CM 268.9) (ICD-9-CM 268.9) 09/11/2015  01 ";

StringCases[text1,RegularExpression["\\(ICD-9-CM\\s*([0-9\\.]+)\\)"]:>"$1"]

yields: {268.9,268.9}

Explanation:

The regular expression language does have so called metacharacters:

\ | ( ) [ { ^ $ * + ? .

If you want to match these metacharacters you have to escape (prefix with \\) them:

(hello world)  <=> \\(hello world\\)

There are several character classes pre-defined in Mma's regex engine:

• \\d = digit 0-9
• \\w = word characters
• \\s = space, new-line, tab, or other whitespace character

(you can find more of them in the Details section in the Documentation of RegularExpression.

Imagine you'd like to recognize ICD-9 codes with a regular expression.

An ICD-9 code looks like this:

• two/three digit numeric code (may have leading zeroes to pad to three digits)
• an optional dot
• if that dot is present, there will be one or two following digits, depending on the preceding three digits. Which digits are allowed specifically is very variable.
• Some codes are prefixed by an E or V.

Your example does not include the E or V prefix, so you're lucky with the solutions proposed so far. The best solution, imho, is to specify a grammatic of ICD-9 code using a regex:

Solution:

StringCases[text1,
    RegularExpression["ICD-9-CM\\s([EV]?\\d{2,3}(?:\\.\\d{1,2})?)"]:>"$1"]

This will match your ICD-9-CM codes and will verify on the fly if they are ill-formed (so to say, no match => someone produced a typo).

This regular expression is production safe and according the specification. So you're safe :)

Answer 5

If you know that the code you are searching for is a number, then I think the suggestions to use patterns which recognize such number strings are your best bet. For completeness I wanted to show another way which I think is a useful standard approach for such tasks and will work even when you can't make such a premise. The idea is to simply only let code match everything except the next closing parenthesis:

StringCases[text1,"(ICD-9-CM " ~~ code : (Except[")"] ..) ~~ ")" :> code]

or, if you prefer with regular expressions (somewhat complicated with all those backslashes needed as escape character):

StringCases[text1, RegularExpression["\\(ICD-9-CM\\s*([^\\)]*)\\s*\\)"] :> "$1"]

EDIT: As Mr. Wizard has pointed out, this is somewhat slower and more complicated than the string expression shown above. That mainly is due to the additional complexity of allowing one or more whitespace characters (\\s*) before and after the matches. A slightly less complicated version of the regular expression which also is a more direct translation of the string expression is this:

StringCases[text1, RegularExpression["\\(ICD-9-CM ([^)]+)\\)"] :> "$1"]

which I would also expect to be slightly faster.