0

First of all, I hope this is the right place to post the question I'm about to ask.

A little background about my problem : I am a data scientist / machine learning engineer / you name it (phd in applied neural network). One big problem about machine learning is to get a dataset large enough to correctly train your model to do the task you want it to do. This problem is especially frequent in image detection / classification. I recently found a paper about some people training neural networks with 3D rendered images so I wanted to try it out.

I'm totally new to bender (5 days of tutorial following at most) but I am kind of getting the scenes I want with eevee. The reason I'm posting here is I didn't find a plug-in/add-on able to automatically detour specific visible rendered items (otherwise I'll have to do it myself on several hundred images).

boat detoured like I need

I tested the picture on a panoptic neural network (detectron 2) and it seems to work. Generating whole datasets of images train-ready would be really huge.

Is there a way to automatically do it (or to get a tiny button somewhere dumping coordinates of the detouring polygon in a file ?) ? Is it possible to create an add-on doing it (I code ML models in python so I should be ok) ?

EDIT : About detouring - With 3D rendered scene I can have enough data to train a model. However, I still have to manually create the polygon containing the object I want to detect (so the model knows what I'm looking for during training) and doing it on hundreds or thousands of images could take ages. I can use coordinates of the angle points of the corresponding polygon, or I can manage to extract it if the corresponding mask is also generated (like of the following picture mask example).

The whole point of this is to :

  1. Create a scene and place my camera
  2. Take a screenshot / get a render picture
  3. Get the mask / coordinates /... of the item I'm interested in the screenshot / render
  4. Move the camera / tweak the scene, and go back to step 2

Note : The steps 2 and 3 should be as fast as possible, given that I'll have to do it hundreds of time (or maybe it is possible to automate that too ? this is beyond my current comprehension of blender)

Nosk
  • 101
  • 1
  • 4
    What does "detouring" mean in this context? – Alexandre Marcati Jun 10 '21 at 18:32
  • "detouring" in this context is about getting the 2d coordinates of the angle points of the polygon "containing" the visible part of specific items (like when I select the boat in object mode but only the visible part, not the submerged part of the boat, for example) – Nosk Jun 10 '21 at 21:34
  • I don't know if I understood correctly (this seems a bit beyond my current level of knowledge), but would a "object color" pass help? Like, a render in which the objects are rendered as solid color silhouettes, and each object has a different color. – Alexandre Marcati Jun 11 '21 at 02:48
  • I realize that I'm quite obscure about the subject, I edited the original question to add clarifications – Nosk Jun 11 '21 at 13:59
  • You'd like to get just an additional image containing a black and white mask for each object right? – brockmann Jun 11 '21 at 14:18
  • @brockmann either that for specific object (I could work with a manual selection) or the ideal would be the coordinates of each points in a file – Nosk Jun 11 '21 at 15:49
  • What do you mean by points? Can be 2d or 3d... The location per pixel? How about a depth map: https://en.wikipedia.org/wiki/Depth_map ...? Related: https://blender.stackexchange.com/questions/215385/rendering-multi-layer-exr-using-file-output-node-in-python/215487#215487 – brockmann Jun 11 '21 at 16:11
  • The points I need would correspond to their position on the 2D rendered image. Ideal would be the number of pixels separating the point from the low border and from the left border for x and y. – Nosk Jun 14 '21 at 16:02
  • They do, read the wikipedia link I provided. Render to exr, enable the mist or depth pass and you'll get the position in space per pixel (the xy coordinates you'll get for free since this is how pixels are strored in any image). – brockmann Jun 16 '21 at 20:59

0 Answers0