How to project back rendered depth maps to recover the ground-truth 3D shape using Blender?

Question

You can discard the images I have uploaded and can simply use some camera angles of your choice to render depth maps of a 3D shape and then fuse the rendered depth maps to recover the underlying 3D shape

We recently published a paper on 3D shape generation in a computer vision conference (CVPR). My co-author wrote the code (in C++ and using OpenCV) for fusing the depth maps and getting the final 3D shapes from the produced multi-view outputs. The inputs to the code he wrote is 20 depth maps, ground-truth camera angles (posted below) and the distance from the centroid of the shapes to the camera (distance to the shape centroid=1.5 on a sphere). The centroid of shapes are calculated as follow:

First, the centers of faces (triangle) of a mesh centroid of a mesh is calculated. Then the faces areas are computed. The new centroid is the average of the mesh faces' centers, weighted by their area.

Here some people have written algorithms on how to compute the centroids.

Unfortunately my friend is not available to help me on this and I cannot use OpenCV and C++ for this new project that I'm beginning to work on. So any help would be appreciated. My goal is to write some code using Blender's Python API instead of using my co-author's C++ code to do the same thing. But before I move on, I wonder if Blender has some built-in functions that can generate the final 3D shape given rendered depth maps of that shape, camera angles and the distance to the camera? If not, can anyone give me some ideas on how I should do that and give me a code sample for it?

Here I have uploaded a set of rendered depth maps a headphone's 3D shape that you can use for backward projection (reconstructing the 3D shape). And if you prefer to start with a 3D shape directly, here you can download a 3D shape of a different headphone we used in our work before. You can render depth maps the 3D shape using the camera angles posted below.

FYI, here is my co-author's high-level description on how his approach works:

In the final step, all depth maps are projected back to the 3D space to create the final rendering. We reconstruct 3D shapes from multi-view silhouettes and depth maps by first generating a 3D point cloud from each depth image with its corresponding camera setting (x, y, z coordinates). The union of these point clouds from all views can be seen as an initial estimation of the shape.

And here are the camera angles we used for doing the rendering in the first place:

-0.57735  -0.57735  0.57735
0.934172  0.356822  0
0.934172  -0.356822  0
-0.934172  0.356822  0
-0.934172  -0.356822  0
0  0.934172  0.356822
0  0.934172  -0.356822
0.356822  0  -0.934172
-0.356822  0  -0.934172
0  -0.934172  -0.356822
0  -0.934172  0.356822
0.356822  0  0.934172
-0.356822  0  0.934172
0.57735  0.57735  -0.57735
0.57735  0.57735  0.57735
-0.57735  0.57735  -0.57735
-0.57735  0.57735  0.57735
0.57735  -0.57735  -0.57735
0.57735  -0.57735  0.57735
-0.57735  -0.57735  -0.57735

Answer 1

After looking at the following image from your paper, it seems to me as if your image is basically a sprite map, so I propose the following workflow to get you there.

Proposed Workflow:

I'm certain that you could mimic the polyhedron shape in blender to set active planes (viewports) on programatically, and use an UV coordinate mapping per sprite location to get a depth map per the corresponding camera location/angle (position).

Set up a highly subdivided plane for each view, then attach the Displacement Modifier to it, along with a custom UV layout to the sprite sheet that corresponds with the appropriate viewpoint.

I would then add a global driver to attempt to control the displacement modifier strength per projection, and when they get close to each other...

Apply the modifier, Join the objects together, and remove duplicate verts.

---

Uploaded a Video