Searching for literal strings with TextSearch

Question

The documentation indicates that TextSearch can be used to look for a "literal string" inside documents, but it appears Mathematica may be applying a regex-style word boundary rule under the hood. Is it possible to get around this?

TextSearch["path/to/logs/", "driver_1_2015-08-03.csv"]

TextSearch::invqs: Query string "driver_1_2015-08-03.csv" is not a word or sequence of words. >>

note this is new in 10.2 and marked "Experimental" – george2079 Aug 31 '15 at 20:41 — george2079, Aug 31 '15 at 20:41

score 8 · Accepted Answer · edited Apr 13 '17 at 12:55

I'd say that the acceptable forms of the query string are much more limited than the documentation lets on. In fairness though, TextSearch is marked [[Experimental]] in the docs, so it may very well be in a state of flux and you would probably not want to use it for anything critical.

Having said that, some spelunking of the definition of TextSearch reveals that the query string is first passed to an implementation iTextSearch function, which passes it to a CompileQuery function, which in turn passes it to a comp intermediate function with multiple definitions.

In the case of a single string query like yours, the matching definition of comp is:

comp[s_String] := compStr[StringTrim[ToLowerCase[s]]]

StringTrim and ToLowerCase are perfectly happy with your string and leave it unmodified.

compStr is where the problem lies, in my view. This is its definition:

compStr[s_] := Which[

  StringMatchQ[s, WordCharacter ..], 
        QString[s],

  StringMatchQ[s, (WordCharacter .. ~~ Whitespace) .. ~~ WordCharacter ..], 
        QIntersection[QString /@ StringSplit[s]],

  True, 
        Message[TextSearch::invqs, s]; Throw[$Failed];
]

Strings like yours containing characters beyond word characters and whitespaces (i.e. spaces, tabs, and newlines) won't match any of those patterns, so the Which will select the default True branch, throwing the TextSearch::invqs message you see and failing out.

A rather brutal workaround is to redefine the compStr function to get around the check:

Clear[TextSearch`IndexSearch`PackagePrivate`compStr]

TextSearch`IndexSearch`PackagePrivate`compStr[s_] :=
  Which[
   StringMatchQ[s, (WordCharacter .. ~~ Whitespace) .. ~~ WordCharacter ..],
     TextSearch`IndexSearch`PackagePrivate`QIntersection[TextSearch`IndexSearch`PackagePrivate`QString /@ StringSplit[s]],

   StringQ[s],
     TextSearch`IndexSearch`PackagePrivate`QString[s],

   True,
     Message[TextSearch::invqs, s]; Throw[$Failed]
]

After the redefinition, the following call to TextSearch seemed to work fine:

TextSearch["sandbox", "ala_a"]

Output of textsearch

The result is correct as well: the file returned is indeed the only one in the folder to contain the "ala_a" string.

Searching for literal strings with TextSearch

1 Answers1