13

I am trying to use a stereo camera for scene reconstruction, but I can usually only obtain sparse point clouds (i.e. over half the image does not have any proper depth information).

I realize that stereo processing algorithms rely on the presence of texture in the images and have a few parameters that can be tweaked to obtain better results, such as the disparity range or correlation window size. As much as I tune these parameters, though, I am never able to get results that are even remotely close to what can be obtained using an active sensor such as the Kinect.

The reason why I want that is because very often point clouds corresponding to adjacent regions don't have enough overlap for me to obtain a match, so reconstruction is severely impaired.

My question to the Computer Vision experts out there is the following: what can I do to obtain denser point clouds in general (without arbitrarily modifying my office environment)?

georgebrindeiro
  • 1,461
  • 9
  • 16
  • This is a good question, but I think more details are necessary. What algorithms have you tried to compute matchings? Could you clarify what you mean by "adjacent regions" and "overlap?". – Josh Vander Hook Nov 17 '12 at 14:50
  • When I say "adjacent regions", I mean portions of physical space that are not equivalent but have a non-empty intersection, which I called "overlap". That is, the kind of regions that would generate point clouds that could be matched and stitched in an ICP algorithm. – georgebrindeiro Nov 18 '12 at 15:36
  • About the algorithm, I am still using the simplest possible solution I could find: the stereo_image_proc ROS node, which applies the global block matching algorithm available in OpenCV.

    I would be very interested in hearing about parameter settings that might not be directly accessible to me through the ROS node or other algorithms that are known to provide better results.

     – georgebrindeiro Nov 18 '12 at 15:43

4 Answers4

4

You can try to skip the salient point detection, and just densely sample over the image (as grid or so) and compute a feature descriptor at every sample point. You can probably even go as far as computing a descriptor for every pixel.

You might lose scale-invariance, but I think this won't hurt too much for stereo vision as objects will be at approximately the same scale in both images.

Another approach might be to combine multiple salient point detection algorithms: corners, edges, blobs and so on. Then you have to use the same feature descriptor algorithm for all detected points, however this latter part could be somewhat tricky to implement.

muksie
  • 141
  • 1
3

So the stereo image processing algorithms I have used in the past were implemented pixel by pixel. We just used the pinhole camera model and did some old fashioned measurements with measuring tape until our depth estimations matched the real thing.

The equations for a pair of parallel cameras are:

  • $d = $half the distance between the cameras
  • $f = $the focal length of the cameras (assumed to be the same)
  • Coordinate Frames:
    • $x, y, z = $ coordinate frame between the cameras (i.e. the camera base frame)
    • $u_R, v_R$ camera coordinates in the right camera from the perspective of the robot (u is horizontal, v is vertical)
    • $u_L, v_L$ camera coordinates in the left camera
    • Note: the camera coordinates have their origins at the coordinate frame between the cameras (i.e. the u axes face opposite directions)

$u_L = \frac{f(x-d)}{z}$, $u_R = \frac{f(x+d)}{z}$

$zu_R = f(x+d)$, $zu_L = f(x-d)$

$z(u_R - u_L) = 2df$

$z = \frac{2df}{u_R - u_L}$

$y = \frac{v_L*z + df}{f}$

$x = \frac{u_L*z + df}{f}$

Using these equations you can compute a dense stereo cloud. One for each pixel on your cameras.

thealmightygrant
  • 151
  • 2
  • Thanks for the effort, but I am familiar with stereo modeling. The problem is usually exactly that of correspondence between the (u,v) coordinates in the left and right cameras. muksie gave some ideas on how to deal with that and sylvain.joyeux pointed out a great stereo library that leads to improved results, though... – georgebrindeiro Nov 24 '12 at 04:16
1

When you say, "over half the image does not have any proper depth information", which half ?

One issue we ran into is that if distance-to-object is of the same order of magnitude than your baseline (usually associated with very wide angle cameras), then the "standard" dense stereo algorithms don't work so well. We've been using the libelas library, and its developers told us that this is called "large baseline stereo" and is yet another problem.

sylvain.joyeux
  • 403
  • 3
  • 7
  • When I said half the image did not have any proper depth information, I meant that out of all pixels, only about half had computed disparities (not necessarily forming one contiguous region). I am not having the same issue you mentioned with the distance-to-object/baseline ratio, but it was very interesting to hear about that library and that this can be a problem. Thanks! – georgebrindeiro Nov 18 '12 at 16:01
  • 1
    Either way, I will try libelas out since I found a ROS wrapper for it! – georgebrindeiro Nov 18 '12 at 16:04
0

Have a look at the KITTI Stereo Benchmark leaders. This benchmark penalizes algorithms for not producing a disparity at any given point, so the top performers produce dense outputs (even if they don't have much grounds for their estimate at many points). Some methods have publicly available code that is linked, which you can try out.

Note that they are not penalized for taking a long time, so many methods will take on the order of minutes per image to run and may not be suitable for your application. There have been many CNN-based methods introduced though that do very well, while still running in less than a second (on a GPU). At least one of these (CRL) has public code.

If you still don't get decent depth estimates with these on your images, then you may just be seeing the limits of what's feasible from a visible approach. There fundamentally needs to be distinctive texture in the area to be able to match points well, or you need a basis for making some strong assumptions (e.g., smoothness) in textureless regions.

surtur
  • 324
  • 1
  • 5