33

I have hand-written 60 pixel times 60 pixel squares. I need to detect whether they are empty, x or circle. TextRecognize function fails. Is there some other function to process this kind of raster images with text?

enter image description here

Harder example

Circles: (0,0..9), (0..5,0), (0..5,9), (5,0..9)

Crosses: (2,3..6), (4,4..5)

Empty: (1,1..8), (2,1..2), (3,1..8), (4,1..3), (4,6..8)

I try to summarize and help people to solve the harder puzzle. Work in progress. Have fun!

I. Preprocessing (example)

1.1. thread about getting grid from raster image 

<p>1.2. <a href="https://stackoverflow.com/questions/10196198/how-to-remove-convexity-defects-in-sudoku-square">convexity
fix</a></p>

<p>1.3. <a href="https://i.stack.imgur.com/kaWDq.png" rel="nofollow noreferrer">rotation</a></p>

II. Testing

2.1. Further info about mathematical morphology and Mathematica's intro.

Community
  • 1
hhh
  • 2,603
  • 2
  • 19
  • 30
  • 2
    Can you add a few sample images, so potential answerers can test their answers? – Niki Estner Apr 18 '13 at 19:08
  • 1
    In general, ImageCorrelate and ComponentMeasurements might be worth a try. – Niki Estner Apr 18 '13 at 19:10
  • As to the failure of TextRecognize: I seem to remember having read somewhere it uses a dictionary approach and it is therefore better in recognizing words than separate characters. – Sjoerd C. de Vries Apr 18 '13 at 20:47
  • @SjoerdC.deVries here – rm -rf Apr 19 '13 at 00:10
  • @rm-rf Good find. What is strange: I have been using TextRecognize successfully to get (Dutch) names from scanned pages in a yearbook. So, I don't quite understand how that chimes with the use of a dictionary. – Sjoerd C. de Vries Apr 19 '13 at 05:22
  • @Sjo Most dictionaries (and I mean ones used in, say, Word or browsers and not OED) also have a list of common names from top X countries. I'm sure Dutch names (at least, the more common ones) are in that list. For instance, Chrome does not highlight any word in "Sjoerd de Vries" as a typo when I type it in a box, nor does it for "Heike", "Jeroen", "Johann", etc., but it does highlight "Arnoud" (less common?). Arnoud also mentioned that it uses other heuristics, so perhaps there's more to it such as using a database of common letter combinations for local match and a dictionary for global... – rm -rf Apr 19 '13 at 13:35
  • @rm-rf OK, I see. Thanks – Sjoerd C. de Vries Apr 19 '13 at 14:20
  • 1
    many of the approaches here will fail if the input is even a little sloppy, not closing the "O". – george2079 May 15 '13 at 18:14
  • @Mr.Wizard there is one thing I cannot understand: when I run your code, it outputs tiny-tiny O, X and ERRs. Is it possible to get the output as ASCII so easier to read or is there some setup in Mathematica to handle the output format? – hhh May 15 '13 at 20:29
  • @george2079 that is a totally new question: perhaps some machine-learning algorithms for this kind of things where the user needs to specify first the ambiguities and then the program would learn with it? The sloppy O is easily 6 and 9: any machine learning specialist to ask a question about this? :) – hhh May 15 '13 at 21:29

3 Answers3

33

For robustness I think it would be best to apply multiple tests and check the results for agreement. Here is one simple test you could include in that suite:

f = MorphologicalEulerNumber[Blur @ #, 0.8] &;

The function should return 0 for X's and 1 for O's. Quoting the documentation:

MorphologicalEulerNumber[image] by default gives the total number of connected white regions in image, minus the number of black holes that occur inside those regions.

You can tune the parameter 0.8 according to the density of your images and the noise level.
You may also try other pre-filters besides Blur, such as MedianFilter.

Example:

imgs = Import /@
  {"https://i.stack.imgur.com/gDzKy.png", 
   "https://i.stack.imgur.com/LMPQq.png"}

f /@ imgs

enter image description here

{0, 1}

The whole enchilada

f2 =
 Switch[
    {
      MorphologicalEulerNumber @ #,
      500 < Total[1 - ImageData[#], 2]
    } & @ Binarize[# ~Blur~ 4, 0.8],
    {1, True}, "O",
    {0, True}, "X",
    {_, False}, "",
    _, "Err"
 ] &;

img = Import["https://i.stack.imgur.com/NbTGY.jpg"];
grid = ImagePartition[
   ImageCrop[ImageRotate[img, 0.7 \[Degree]], {1180, 720}, {-0.15, 0.2}], {118, 119}];

Map[
 ImageCompose[#, Rasterize[Style[f2@#, Red], Background -> None]] &,
 grid, {2}
] // GraphicsGrid

enter image description here

Mr.Wizard
  • 271,378
  • 34
  • 587
  • 1,371
  • 4
    coming up with MorphologicalEulerNumber to solve this particular problem is a stroke of genius... – Stefan Apr 18 '13 at 20:59
  • Circles (0,2) and (5,5) returns me errors, requiring apparently better preprocessing: argument change in Binarize from 0.8 to 1 gives totally different results. I cannot understand why it worked you like that, I just copied your code -- why the difference? – hhh Apr 18 '13 at 21:25
  • You could also use Euler's characteristic to differentiate different shapes: X has 1 and circle has 0. I am trying to think how it would work with harder shapes -- no Euler characteristic command in Mathematica, odd. – hhh Apr 18 '13 at 21:36
  • @hhh Since I'm not getting the errors ("Err") from my code it is hard for me to debug. I'm surprised as I wouldn't expect any of these functions to change between versions. There are three parameters to the second method you might experiment with: the prefilter type and magnitude (here Blur and 4), the Binarize threshold, and the number of black pixels required to consider the square non-empty (here 500). – Mr.Wizard Apr 19 '13 at 02:09
  • 2
    @Stefan Thanks! I don't get many of those. :^) – Mr.Wizard Apr 19 '13 at 02:09
  • @hhh Version 7 on Windows 7. I too would like to understand the apparent discrepancy. When you have more time please leave a comment here, and perhaps we can Chat about it. – Mr.Wizard Apr 20 '13 at 02:02
  • @Mr.Wizard Testing on Mathematica 9 Ubuntu 12.04 LTS - HP BL460c 2 x X5650 @ 2.67GHz 96 GB RAM server running Mathematica over X, the same errors seem to appear as with my own computer OS X running Mathematica 8 -- yet it is very hard to see the crosses and circles because the font is so small :( I wish I could see it in ASCII so easier to debug...or any easy way to specify the size of the image. Over the SSH, it is pretty slow to adjust things by mouse... – hhh May 15 '13 at 20:38
  • Moved the new question about output-formatting here. – hhh May 15 '13 at 21:25
19

As I said in the comment, ComponentMeasurements is a easy and robust way to differentiate simple shapes.

Using your image and binarizing it:

img = ColorConvert[Import["https://i.stack.imgur.com/NbTGY.jpg"], 
  "Grayscale"];
bin = ColorNegate@Binarize[ImageAdjust[GaussianFilter[img, 5]]]

ComponentMeasurements calculates a list of measurements for each connected component.

components = 
  ComponentMeasurements[
   bin, {"Centroid", "Area", "FilledCircularity", 
    "EquivalentDiskRadius"}, #2 > 100 &];

In this case, it calculates the centroid, area and the filled area/perimeter ratio compared to a circle. That's a very good measure to recognize circles. The EquivalentDiskRadius is just there for the display:

Show[img,
 Graphics[
  {
   Thick,
   {If[#[[3]] > 0.5, Red, Blue], Circle[#[[1]], #[[4]]]} & /@ 
    components[[All, 2]]
   }]]

Objects with FilledCircularity < 0.5 are displayed as blue circles, FilledCircularity > 0.5 red:

result

Niki Estner
  • 36,101
  • 3
  • 92
  • 152
  • Too bad ComponentMeasurements isn't in version 7. It looks spiffy. :-) – Mr.Wizard Apr 18 '13 at 20:28
  • Suppose differentiation btw o and #. The morphological euler number would fail. Define perimeter for # as total length of lines in it. Now the area/perimeter_# < area/perimeter_o so this measure probably better for differentation. Differentiating chars used in ASCII games such as @,#,u,&,x,o -- can be pretty tricky. Have to consider new methods for differentiation or more distinct chars, someone must have considered this earlier... – hhh Apr 18 '13 at 21:55
10

This trick sometimes works:

i1 = Import["https://i.stack.imgur.com/gDzKy.png"];
i2 = Import["https://i.stack.imgur.com/LMPQq.png"];
TextRecognize@ImageAssemble[{i1, i2, i1, i2, i1, i2}]
(*
--> "XOXOXO"
*)

Use it by appending the unknown square to a bunch of known ones, and detecting the last character.

Dr. belisarius
  • 115,881
  • 13
  • 203
  • 453
  • Ok, Ok ... and the string length in case the new char is blank. – Dr. belisarius Apr 18 '13 at 19:12
  • I cannot understand why it does not work with one image like "TextRecognize@i1" or "TextRecognize@{i1}" but "TextRecognize@ImageAssemble[{i1,i2}]" works. – hhh Apr 18 '13 at 19:31
  • 1
    @hhh TextRecognize is designed to work on words, not individual characters. See this, also answered by belisarius. – Mr.Wizard Apr 18 '13 at 19:39
  • This method results into non-robust implementation at least in my Mathematica (one lower than newest), http://mathematica.stackexchange.com/questions/25250/textrecognize-with-x-and-o-results-into-oddities-why?lq=1. Why? – hhh May 15 '13 at 18:00
  • @hhh probably you're trying to match blank spaces. It is robust as long as you manage blank spaces by "hand" – Dr. belisarius May 15 '13 at 18:02
  • @belisarius yes but I have different amount of spaces in the grid. My goal is to print readable ASCII output about the image, I cannot neglect the spaces -- possible to make Mathematica read them as some special character or something like that? – hhh May 15 '13 at 18:03