How to get the 3D coordinates of the visible vertices in a rendered image in Blender

Question

I'm new to Blender and python and here is what I need:

I have a 3D model in Blender and I've set camera parameters to view a specified region of the model. The following is the rendered image:

I need to know the 3D coordinates of the vertices that are visible in the rendered image. Some of the points whose 3D coordinates I need are circled in RED in the figure.

I've some sources that mention using "bpy_extras.view3d_utils" function to extract the 3D coordinates from a rendered image, but I'm not able to code it in python.

Even if I manage to get the 3D coordinates of all the pixels in the rendered image, it should be a great help.

Answer 1

A partial solution, due to some approximation in ray cast (or some bug in my script?).

This solution uses 'world_to_camera_view' from bpy_extras.object_utils module. 'world_to_camera_view' returns the projection of a vertex in camera's coordinates, which means that the vertex is inside the camera view if the projected coordinates are between 0 and 1.

From that, the script is testing if the corresponding vertex is visible from the camera (and not hidden by another part of the mesh). To do that, it's using a ray cast from the camera location to the vertex.

Unfortunately, some of these ray casts fail... but this script is close to your need and that's why I provide it as an answer.

Hope that can help though.

import bpy
from mathutils import Vector
from mathutils.bvhtree import BVHTree
from bpy_extras.object_utils import world_to_camera_view

# Create a BVH tree and return bvh and vertices in world coordinates 
def BVHTreeAndVerticesInWorldFromObj( obj ):
    mWorld = obj.matrix_world
    vertsInWorld = [mWorld * v.co for v in obj.data.vertices]

    bvh = BVHTree.FromPolygons( vertsInWorld, [p.vertices for p in obj.data.polygons] )

    return bvh, vertsInWorld

# Deselect mesh polygons and vertices
def DeselectEdgesAndPolygons( obj ):
    for p in obj.data.polygons:
        p.select = False
    for e in obj.data.edges:
        e.select = False

# Get context elements: scene, camera and mesh
scene = bpy.context.scene
cam = bpy.data.objects['Camera']
obj = bpy.data.objects['Cube']

# Threshold to test if ray cast corresponds to the original vertex
limit = 0.0001

# Deselect mesh elements
DeselectEdgesAndPolygons( obj )

# In world coordinates, get a bvh tree and vertices
bvh, vertices = BVHTreeAndVerticesInWorldFromObj( obj )

print( '-------------------' )

for i, v in enumerate( vertices ):
    # Get the 2D projection of the vertex
    co2D = world_to_camera_view( scene, cam, v )

    # By default, deselect it
    obj.data.vertices[i].select = False

    # If inside the camera view
    if 0.0 <= co2D.x <= 1.0 and 0.0 <= co2D.y <= 1.0: 
        # Try a ray cast, in order to test the vertex visibility from the camera
        location, normal, index, distance = bvh.ray_cast( cam.location, (v - cam.location).normalized() )
        # If the ray hits something and if this hit is close to the vertex, we assume this is the vertex
        if location and (v - location).length < limit:
            obj.data.vertices[i].select = True

del bvh

Answer 2

@lemon - it turns out a few more users are suffering from the error in the ray_cast near the contours and edges. This is resolved by adding cubes to each vertex to find a proper hit for the casted ray and increasing the threshold distance that was previously used. I've used scene.ray_cast function.

Below is a screenshot from the detection. The related post link guided me I thank IPv6 for giving me a proper suggestion. Thanks again and Cheers!

import bpy
from mathutils import Vector
from bpy_extras.object_utils import world_to_camera_view
Deselect mesh polygons and vertices
def DeselectEdgesAndPolygons( obj ):
    for p in obj.data.polygons:
        p.select = False
    for e in obj.data.edges:
        e.select = False
Get context elements: scene, camera and mesh
scene = bpy.context.scene
cam = bpy.data.objects['Camera']
obj = bpy.data.objects['Cube']
Threshold to test if ray cast corresponds to the original vertex
limit = 0.1
Deselect mesh elements
DeselectEdgesAndPolygons( obj )
In world coordinates, get a bvh tree and vertices
mWorld = obj.matrix_world
vertices = [mWorld * v.co for v in obj.data.vertices]
print( '-------------------' )
for i, v in enumerate( vertices ):
    # Get the 2D projection of the vertex
    co2D = world_to_camera_view( scene, cam, v )
bpy.ops.mesh.primitive_cube_add(location=(v))
bpy.ops.transform.resize(value=(0.01, 0.01, 0.01))

# By default, deselect it
obj.data.vertices[i].select = False

# If inside the camera view
if 0.0 <= co2D.x <= 1.0 and 0.0 <= co2D.y <= 1.0 and co2D > 0: 
    # Try a ray cast, in order to test the vertex visibility from the camera
    location= scene.ray_cast( cam.location, (v - cam.location).normalized() )
    # If the ray hits something and if this hit is close to the vertex, we assume this is the vertex
    if location[0] and (v - location[1]).length < limit:
        obj.data.vertices[i].select = True

UPDATE: The script modified for the new version of Blender is here: https://blender.stackexchange.com/a/87774/113612

Answer 3

Select the mesh and activate edit mode with tab. Then make sure you are in vertex editing mode (instead of edge or face mode) and select one of the vertices.

If you do not have the 3D view's right panel active press the n key to make it visible. The coordinates of the selected vertex (or median of multiple) should be visible at the top of the n-panel. There is even a radio button beneath it to toggle between the global coordinate system and the object's local coordinate system (which can vary if there is any translation, rotation, or scaling on the object).