First off: A viewport render of a overlayed, 400x400, colored point cloud and a accuracy comparison:
This is a portion of the blender 2.83 example scene
I think the distortion stems from the wrong projection and not the file format, I made a little illustration, that could explain your phenomenon, although I didn't try to investigate that issue too much:
"If I have seen further it is by standing on the shoulders of Giants”
This answer is patchwork quilt of answers and questions from stackexchange and stackoverflow, special credit goes to the user "lemon" and his answer of
this question
The script works in 4 parts:
- Render the depth and image pass
- Calculate the reverse projection of the depth pass
- Save the depth and image pass to a colored pointcloud (.ply) with open3d
- Create an empty at (0,0,0) and use the "Point Cloud Visualizer" to project the .ply
More detailed instructions
Switch to Compositing layout, activate Use Nodes, connect the nodes, change File Output nodes Name and their inputs in the following fashion:
File Output node - color
File Output node - depth
Go to Properties Panel -> Output Properties and change the Post Processing settings as displayed:
Go to "C:\Program Files\Blender Foundation\Blender 2.83\2.83\Python\bin" and install the following modules (in the same fashion, if you are missing something else):
- cv2: "python.exe -m pip install opencv-python"
- open3d: "python.exe -m pip install open3d"
There was some kind of error, which I can't remember, but I had to install anaconda and use 2 commands (sry)
Module installation process for Mac users.
Copy this script:
import bpy
import cv2
import numpy as np
from math import tan
from mathutils import Vector
import open3d as o3d
import os
def point_cloud(depth,cam):
# Distance factor from the camera focal angle
factor = 2.0 * tan(cam.data.angle_x/2.0)
rows, cols = depth.shape
c, r = np.meshgrid(np.arange(cols), np.arange(rows), sparse=True)
# Valid depths are defined by the camera clipping planes
valid = (depth > cam.data.clip_start) & (depth < cam.data.clip_end)
# Negate Z (the camera Z is at the opposite)
z = -np.where(valid, depth, np.nan)
# Mirror X
# Center c and r relatively to the image size cols and rows
ratio = max(rows,cols)
x = -np.where(valid, factor * z * (c - (cols / 2)) / ratio, 0)
y = np.where(valid, factor * z * (r - (rows / 2)) / ratio, 0)
return np.dstack((x, y, z))
start = 1
end = 10
step = 1
bpy.data.scenes[0].frame_start = start
bpy.data.scenes[0].frame_end = end
for i in range(start, end+1, step):
bpy.data.scenes['Scene'].frame_current = i
framenumber = str(10000+i)[1:]
# Render Image so depth and image can be output
base_path = '/tmp/'
color_name = 'color{}.jpg'.format(framenumber)
depth_name = 'depth{}.exr'.format(framenumber)
color_path = base_path + color_name
depth_path = base_path + depth_name
print(color_name)
bpy.data.scenes['Scene'].node_tree.nodes['color_output'].base_path = base_path
bpy.data.scenes['Scene'].node_tree.nodes['depth_output'].base_path = base_path
bpy.ops.render.render()
# Read depth
print(depth_path)
depth = cv2.imread(depth_path, cv2.IMREAD_ANYCOLOR | cv2.IMREAD_ANYDEPTH)
depth = depth[:,:,1]
# Read color
color = cv2.imread(color_path)
# Get the camera
cam = bpy.data.objects['Camera']
# Calculate the points
points = point_cloud(depth, cam)
# Get the camera matrix for location conversion
cam_mat = cam.matrix_world
# Translate the points
verts = [cam_mat @ Vector(p) for r in points for p in r]
# Convert color from 0-255 to 0-1, with datatype float64
#bpy.data.objects[empty_name].point_cloud_visualizer.filepath
color = np.divide(color.astype(np.float64), 255)
# Reshape from img shape to shape (width*height, 3), (like 1080, 1920, 3) -> 1080*1920,3
color = np.reshape(color, (len(verts), 3))
# Set Pointcloud outputpath, create a pointcloud with depth and color information and save it
'''saving and loading is totally unecessary (but was easier to programm) if you want to save some time work on this'''
ply_file_path = base_path + '/{}_data.ply'.format(str(i))
pcd = o3d.geometry.PointCloud()
print(color.shape)
print(type(color))
pcd.points = o3d.utility.Vector3dVector(verts)
pcd.colors = o3d.utility.Vector3dVector(color)
o3d.io.write_point_cloud(ply_file_path, pcd)
Now we have to open blender and go to the scripting tab, paste the script and run it.
"Scripting -> + New -> Copy pasterino" -> Little triangle
The script now places the .ply file in '/tmp/', should be 'C:/tmp/' for most users (sry linux/mac users). This will freeze blender and take more time, the larger the render resolution is. The script is
runs at 1fps for me at HD (3900x, 1080ti) with the above scene (EEVEE) without any optimizations (400x400 res). I guess it could run at 30+ if we didn't save and load the .ply
Just as a sidenote: If you need the highest accuracy, change the datatypes and increase the resolution
Because the sequence loading isn't automated yet, you have to load the .ply's by hand. Use the pointcloud visualizer "draw" button (after adding an empty and making it the active object!):
Anyway..
I didn't have the time to make the sequence display automated, but I'm sure you can take it from here.
Example file (windows):
