2

I got my object and I need to know all pixels that are displaying this object in my rendered picture.

I read about a smiliar question here.

update I tried this code from here:

def tocam(scene, ob):
    cam_vec = cam.matrix_world.to_3x3() * Vector((0, 0, -1))
    R = cam_vec.to_track_quat('-Z', 'Y').to_matrix().to_4x4()

    # scale based on resolution
    S = Matrix() # resX=resY
    # translate such that origin is middle point of image (and hence cam)
    T = Matrix.Translation((-0.5, -0.5, 0))

    ob.data.transform(ob.matrix_world)
    ob.matrix_world = Matrix()
    for v in ob.data.vertices:
        vec = w2cv(scene, cam, v.co)
        v.co = vec.x, vec.y, 0

    ob.data.transform(S * T)

    ob.matrix_world = R
    angle_x = cam.data.angle_x
    x = (0.5 /  tan(angle_x / 2)) * cam_vec.normalized()
    ob.matrix_world.translation = cam.matrix_world.translation + x
    if cam.data.type == 'ORTHO':
        ob.scale *= cam.data.ortho_scale

    res_x = 640
    res_y = 640

    # 2d data printout:
    rnd = lambda i: round(i)

    for v in ob.data.vertices:
        print("{},{}".format(rnd(res_x*v.co.x), rnd(res_y*v.co.y)))

when I plot the points I get this:

enter image description here

but my rendered image is this:

enter image description here

So you can see, that it is not on the correct position and also the shape is not really correct.

Can anyone help me on this? Thanks a lot!

blenderNewbie
  • 45
  • 12
  • Related https://blender.stackexchange.com/questions/159914/perspective-flatten-geometry-by-shrinkwrap/160388?r=SearchResults&s=1|27.8688#160388 – batFINGER Feb 03 '20 at 15:21
  • Can you tell me more about what your use case is? Would it be viable to you, if the script was run from the cmd and you'd then get a numpy array that you can work with or does it have to be run from inside blenders scripting tab? – WhatAMesh Feb 07 '20 at 08:54
  • @WhatAMesh it would be good if it's inside the same script. I am rendering multiple images in my script. The goal is to use them as trainings data for a neural network to do segmentation. So to use the renered images as trainingsdata I need to know which pixels in the rendered Image belong to my object I rendered. At the moment I am starting blender with my script like this: ./blender first.blend --background --python render.py – blenderNewbie Feb 07 '20 at 14:26

1 Answers1

5

Basically one can just render the masks of the objects as images, import them and generate the arrays. This process is the similar for materials (a mask for every material) and depth. enter image description here enter image description here

The script can be (at least) run in 2.8x and 2.79b, just adjust the code with the commented code. If you use 2.79b, put blender in the same folder as the your python.exe.

Master script:

import matplotlib.pyplot as plt
import numpy as np
import glob, os
import PIL.Image as pil
import subprocess

pathToSlave = r'C:\Users\YourName\PycharmProjects\bse\slave.py'


pathToFile = r'C:\Users\YourName\Desktop\segmentation.blend'
fileName = 'segmentation.blend'


pathToRenderOutput = r'C:\Users\YourName\Desktop\trash\'
pathToBlender = r'C:\Users\YourName\AppData\Local\Programs\Python\Python36\blender-2.79b-windows64\'


subprocess.call('{}blender {} -b  --python {}'.format(pathToBlender, pathToFile, pathToSlave))


images = glob.glob(os.path.join(pathToRenderOutput, '*.png'))
for img in images:
    arr = np.asarray(pil.open(img))
    arr = arr.sum(axis=-1)
    arr = arr.squeeze()


plt.imshow(arr)
plt.show()

Blender part: A mask is generated for every mesh-object

import bpy

bpy.context.scene.render.engine = 'CYCLES'
bpy.context.scene.use_nodes = True
bpy.context.scene.cycles.samples = 1


For 2.8x


bpy.context.view_layer.use_pass_object_index = True


bpy.context.scene.view_layers['ViewLayer'].use_pass_object_index = True


For 2.79b


#bpy.context.scene.render.layers["RenderLayer"].use_pass_object_index = True


tree = bpy.context.scene.node_tree
#tree = bpy.data.scenes['Scene'].node_tree


nodes = tree.nodes
links = tree.links
indexOBOutput = tree.get('IndexOB Output')


idMaskList = []


outputPath = r'C:\Users\YourName\Desktop\trash\'


fileOutput = nodes.new(type="CompositorNodeOutputFile")
fileOutput.base_path = outputPath
fileOutput.file_slots.remove(fileOutput.inputs[0])


for index, obj in enumerate(bpy.context.scene.objects, start=1):
    if obj.type == 'MESH':
        obj.pass_index = index


    idNode = nodes.new(type='CompositorNodeIDMask')
    idNode.index = index

    links.new(nodes.get('Render Layers').outputs.get('IndexOB'), idNode.inputs[0])
    fileOutput.file_slots.new('Object_{}'.format(index))

    links.new(idNode.outputs[0], fileOutput.inputs[index - 1])




bpy.ops.render.render(use_viewport=False)
pyCod3R
  • 1,926
  • 4
  • 15
WhatAMesh
  • 1,169
  • 1
  • 10
  • 25
  • Thank you so much for your work. I will try this tomorrow evening when I am back at my pc. – blenderNewbie Feb 08 '20 at 00:02
  • I get AttributeError: 'Scene' object has no attribute 'view_layers' could this be because I am using blender 2.79? Maybe I can do this easier anyway as I just want a mask from one obj? – blenderNewbie Feb 10 '20 at 09:26
  • @blenderNewbie Adjusted the answer for 2.79b. Only you know how far it can be simplified, if you have a specific object you can change the line to "for obj in bpy.data.objects['your_object_name']", I just wrote for the more general case in 'segmentation': one image containing multiple classes (objects) – WhatAMesh Feb 10 '20 at 10:43
  • Thank you! You are awesome. That helped me big time. One more question: Do you know how I can manipulate the output name of the rendered mask? now it is something like object_11000.png – blenderNewbie Feb 10 '20 at 11:01
  • At the moment the fileoutputname is generated with fileOutput.file_slots.new('Object_{}'.format(indexPass)), you can change that however you like. The issue seems to be, that the frame number e.g. 000, 001 will always be added to the filename, which seems to be a bug: https://blender.stackexchange.com/questions/46062/how-to-keep-blender-from-adding-frame-numbers-to-output-filename. In the masterscript you can rename the files with https://stackoverflow.com/questions/2491222/how-to-rename-a-file-using-python – WhatAMesh Feb 10 '20 at 11:41
  • Thank you. Can I remove a specific link? Because of my structure I render the image and after that increase my indexPass and render a new image. But when doing this the mask from the first one is cleared and I get an empty mask. – blenderNewbie Feb 10 '20 at 15:10
  • Okay, nvm. I fixed it myself. – blenderNewbie Feb 10 '20 at 18:43
  • How could you adapt it for instance segmentation? i.e. creating one mask image with all objects differentiated. – Lluis C Jun 09 '22 at 19:15
  • @LluisC Lets say we would want to generate an image, where a human can differentiate the n objects/instances. Then we generate n different colors (far enough apart in the spectrum, so that we can recognize the difference). We then generate a new array with the same size as one of the mask images (with 3 channels for rgb). Then for every mask, we take one generated color and fill the new array with that color at the places, where the masks is 1 (1 = object is there for example) – WhatAMesh Jun 09 '22 at 23:02