How can I detect a rectangular region of interest in a picture?

Question

Here is a downsampled input image I converted to greyscale.

enter image description here

To select the rectangular region of interest we first preform preprocessing:

binarizedImage = 
 Dilation[Erosion[
   Dilation[Binarize[FillingTransform@ColorNegate@resizedGreyImage], 
    3], 4], 3]

And then find the relevant components and corners

regions = 
 SelectComponents[
  DeleteSmallComponents[binarizedImage, 
   Method -> "Mean"], {"Rectangularity"}, 1]

corners = 
 ImageCorners[regions, 4, 0, 10, MaxFeatures -> 4, 
  "MaxRefinement" -> 0]

Here is the result:

Show[input, 
 Graphics[{Opacity[0.5], PointSize[.05], Yellow, 
   Polygon[corners[[ConvexHull[corners]]]], Thickness[0.4], Blue, 
   Point /@ corners, White, Text[ToString@#, #] & /@ corners, Green, 
   Point /@ centers, EdgeForm[{Red, Thick}], FaceForm[Opacity[0]], 
   Rectangle @@ # & /@ boundingBoxes}], ImageSize -> 600]

enter image description here

I'd like this to be more robust. I'm not sure that "Rectangularity" is the best measure, for instance, in this picture, selecting the component with highest "Rectangularity" doesn't always work:

![enter code here][3]

DeleteSmallComponents[
     ImageForestingComponents[ImageResize[input, 300]]]

% // Colorize
regions = SelectComponents[%%, {"Rectangularity"}, 1] // Colorize

enter image description here

Also, I'm coding this in opencv, I'm only using mathematica to prototype so if you know the underlying methods used that would be very helpful.

Here are some test images:

enter image description here

I wonder how do you distinguish your "rectangular" ROI from other "rectangular" items in your picture — Dr. belisarius, Mar 11 '13 at 16:04
What's making it hard to answer this question is that we don't have a good idea of the scope of the kinds of rectangles that you really want to identify. If it really is just this one, then you can do so by clicking on four points! Do you have a collection of images where we can see the target rectangle(s). — bill s, Mar 20 '13 at 17:39
Also, I don't think downsampling and color conversion as a first step is a good idea. You may be losing useful info. — Dr. belisarius, Mar 20 '13 at 17:54
Maybe not, speed is important and we can save a copy of the original to compare against. — M.R., Mar 20 '13 at 17:56
You should look at nikie's answers here and here for some ideas... — rm -rf, Mar 20 '13 at 18:07
I've seen that, but her method fails in the second step on all on my images, the heuristic of taking the largest filled binary region is waaaaay too simplistic to work somewhat generally. — M.R., Mar 20 '13 at 18:18
(a) you're not planning to code this in Mathematica eventually, and (b) "Rectangularity" seems poorly defined. Kind of hard to find a good starting point. — Jens, Mar 20 '13 at 20:08
Finding four straight (-ish) lines with angles that add to approximately 360 degrees might be fruitful.... — geordie, Mar 20 '13 at 23:14
@geordie But how to tell apart a rectangle with parallax from a planar parallelogram? Or how to distinguish a rectangle in a 3D scene with perspective vanishing points from a trapezoid in 2D? And how rounded are the corners allowed to be? I think this problem isn't a good fit for this forum. — Jens, Mar 21 '13 at 03:39
@Jens Fair point. I was assuming that the goal was to find 'potential' rectangles not just actual rectangles. I realise this is a somewhat idiosyncratic distinction... "Ceci n'est pas une pipe" — geordie, Mar 21 '13 at 11:04
@M.R. Do you know what SelectComponents' equivalent function is in opencv? Do you have a reference suggestion? — rm -rf, Jan 17 '14 at 03:04
I wonder if SelectComponents convex hull properties, Opening with a BoxMatrix or DiamondMatrix, and perhaps the number of convex hull points could be of use — Sterling, Feb 24 '21 at 21:38

Dr. belisarius · Answer 1 · 2013-03-24T22:20:51.067

13

It's more a region-of-interest detection than a rectangle identification algorithm and still half-baked, but anyway: (using Mma 8.0 on WinXP)

l = {"https://i.stack.imgur.com/t95sV.jpg", "https://i.stack.imgur.com/MVP47.jpg", 
     "https://i.stack.imgur.com/QpiLb.jpg", "https://i.stack.imgur.com/pHHZ4.jpg", 
     "https://i.stack.imgur.com/wlFpH.jpg", "https://i.stack.imgur.com/x7YO9.jpg", 
     "https://i.stack.imgur.com/zCHSm.jpg"};

i = ImageResize[#, 500] & /@ (Import /@ l);
i1 = ColorSeparate[#, "HSB"][[2]] & /@ i;
ie1 = EdgeDetect /@ ImageAdjust /@ (LaplacianGaussianFilter[#, 2] & /@ i1);
dc1 = DeleteSmallComponents[#, 100] & /@ ie1;
lines1 = ImageLines[#, .15, .611] & /@ dc1;
slope[{p1_, p2_}] := (p2[[2]] - p1[[2]])/(p2[[1]] - p1[[1]])
Show[#[[1]], Graphics[{Thick, Red, Line /@ #[[2]]}], 
      ImageSize -> 100] & /@ Transpose[{i, lines1}]

Mathematica graphics

The next step is to discard those lines whose slopes are not paired/grouped within a tolerance. I need only some spare time ...

edited Mar 24 '13 at 22:20

answered Mar 24 '13 at 21:51

Dr. belisarius

115,881
13
203
453

I just tried your code (V9, Win7) and I only get 1 red line per image... – s0rce Mar 24 '13 at 22:08
Same here with v8 on Win7. – Simon Woods Mar 24 '13 at 22:14
@SimonWoods I'm using Mma 8.0 on WinXP – Dr. belisarius Mar 24 '13 at 22:21
@s0rce Probably the Imagelines[] function isn't consistent across platforms. – Dr. belisarius Mar 24 '13 at 22:23
@belisarius You're still using 8.0.0, right? Probably the same issue as in your answer on DSP. – rm -rf Mar 24 '13 at 23:39
@rm-rf Well, I'm unable to test and fix it on another platform, so ... – Dr. belisarius Mar 24 '13 at 23:44
@rm-rf A couple of bug fixes in version 9 must explain the difference in performance. Altering a bit the parameters set by belisarius should produce proper results. – Matthias Odisio Mar 25 '13 at 14:53
@MatthiasOdisio Do you care to edit the answer with the modified parms for v9? – Dr. belisarius Mar 25 '13 at 15:27
@belisarius Setting LaplacianGaussianFilter to 6 and parameters for ImageLines to .111 and .207 yield roughly comparable results on my 9.0.1 on OSX. Don't ask me what has changed, maybe some parameters inside Hough transform ImageLines uses? – kirma Mar 25 '13 at 19:16

Matthias Odisio · Answer 2 · 2013-03-27T14:23:05.300

At his request, this is how I would edit @belisarius's answers (using 9.0.1):

l = {"https://i.stack.imgur.com/t95sV.jpg", 
     "https://i.stack.imgur.com/MVP47.jpg",
     "https://i.stack.imgur.com/QpiLb.jpg",
     "https://i.stack.imgur.com/pHHZ4.jpg",
     "https://i.stack.imgur.com/wlFpH.jpg",
     "https://i.stack.imgur.com/x7YO9.jpg",
     "https://i.stack.imgur.com/zCHSm.jpg"};

i = ImageResize[Import@#, 500] & /@ l;
i1 = Flatten[ColorSeparate[#, "Saturation"] & /@ i];
ie1 = EdgeDetect[#, 6, .04] & /@ i1;
dc1 = DeleteSmallComponents[#, 100] & /@ ie1;
lines1 = ImageLines[#, 0, .2, MaxFeatures -> 4] & /@ dc1;

Show[#[[1]], Graphics[{Thick, Red, Line /@ #[[2]]}], ImageSize -> 100] & /@ Transpose[{i, lines1}]

enter image description here

It still looks a bit fragile though. With grayscale images, the saturation channel is black. In these cases (ImageMeasurements[saturation, "Max"] == 0.), one just goes ahead using the actual image, not the saturation map.

Yeah, it's very fragile. I get very different results using 9.0.0 on OS X. It works on only two of the images :/ — C. E., Mar 26 '13 at 20:14
I meant fragile with respect to the input images, not to software bugs (Upgrade to 9.0.1 if possible). I have not tried this method on other images, and I'm pretty sure it's not the silver bullet, but it provides a good result under some circumstances. — Matthias Odisio, Mar 26 '13 at 22:59

How can I detect a rectangular region of interest in a picture?

2 Answers2

Linked