2

LOOK FOR BOX TYPE IMAGES

I need to annotate each SINGLE plant separately from others, for accurate annotation, right now I've it scripted for ID MASK. It gives each object from a specifically named collection iterated ID pass, renders original scene in .png, and then renders and converts pixels (every pixel, which is not black) from every ID MASK image to .json. But as you can see it doesn't works as expected. Is there something similar I could do with cryptomatte? Again, For annotation I need to know in advance which objects which color will use or something like that to know exactly or define an object. maybe I can adjust emission shader or make my own so it can handle transparent textures correctly as well? or something along those lines enter image description here UPD cryptomatte works only when I render plants separately from scene Look here cryptomatte whole scene vs cryptomatte plants only enter image description here enter image description here

Blender doesn't render all pixels on distant objects, it gets better when I make resolution higher, like 4k, but it doesnt fixes my problem, but makes image pixels better. Also raising sample doesnt help. I tried to render with denoising and AA in compositor, that didnt help. How can I fix this, so it will render all object pixels on ID MASK using box pixel type? I need an alpha of an object as an ID MASK, for accurate object annotation.

Render Settings

Engine: Cycles

Render Resolution: 1280x720

Samples: 128

Pixel type: Blackman-Harris

Pixel width: 0.01

Full Image: enter image description here Problem:enter image description here enter image description here

Additional info: ID_MASK of the object in compositor enter image description here

Box type pixels looks so much better on the Image, but ID Mask dont recognize them, now how to fix this one?

Box type render: enter image description here enter image description here Box Type ID Mask: enter image description here Box type Render 4k: enter image description here enter image description here Box type 4k ID MASK: enter image description here

Annotations

Here's an addon, which do almost what I needed, but it uses emission shader, which doesnt work with custom textures.

By annotation I mean assigning each pixel of the original (rendered) image a corresponding value of instanceId

Where instanceId maps a specific blender object.

enter image description here

Here's more examples: Original Image without annotations: enter image description here Annotated Image: enter image description here enter image description here

For now I just need objectId and categories.

Right now I'm using ID masks which is the only reliable solution I've come up with.

As you can see I've tried a lot of other options such as: Cryptomatte Emission shader Denoising Playing with AlphaTreshold

Since ID masks don't work as reliable as I thought they would. I was thinking about writing my own shader that will use emission (to get the perfect color) and some magic to fix the transparent textures.

Here's why emission shader won't work in any case: enter image description here enter image description here enter image description here

I know I should start a new question, but... how to achive a solid color with nodes? If there's a way to get a fully opaque color w/o aa or any kind of transparency? That would solve my problem perfectly. Here's my node attempt: enter image description here enter image description here

Cryptomatte did the job, but it needs a manual nodes pre-setup enter image description here enter image description here

cxnt
  • 397
  • 1
  • 5
  • 27
  • Increase the output resolution and you are done – lost protonbender Mar 25 '20 at 14:50
  • Is this the only solution? – cxnt Mar 25 '20 at 22:10
  • You might need to provide your scene setting about those plants. – HikariTW Mar 26 '20 at 06:29
  • I don't think there is any way to make ID mask to identify a pixel which is not the main sampled pass. Id mask only provide yes, no, transparent information, what you want is the alpha value of the solid color from that plant right? Did you try Cryptomatte? – HikariTW Mar 26 '20 at 07:29
  • I need to accurately annotate each plant on the scene. Yeh I tried cypromatte, but not with Box type, if that matters?@HikariTW updated answer with 4k box type – cxnt Mar 26 '20 at 07:32
  • I will probably just render the plants without background, and use the image (alpha channel) as input to do what you want. Yeah, another scene required perhaps – HikariTW Mar 28 '20 at 14:41

2 Answers2

5

Don't use ID Masks

Just don't. They have little use, this is a really poorly designed feature. You simply cannot make a good mask with a 2bit color depth image.

Cryptomatte is the solution

Look into using Cryptomatte. You need to enable it's layers in the View Layer properties tab:

enter image description here

You can then get any masks easily using Cryptomatte node in the compositor and they will be a lot higher quality:

enter image description here

Community
  • 1
Martynas Žiemys
  • 24,274
  • 2
  • 34
  • 77
  • I have updated my question, please take a look – cxnt Apr 02 '20 at 15:58
  • I am not sure I understand the issue. Cryptomatte should provide masks as good as they get as far as I know. You can output them to files with File Output node and run any scripts you need for further image processing outside Blender. They can be saved in Multilayer EXR format and you can then open them in other software like in Photoshop with this free plugin: https://www.exr-io.com/. I have not tried it, but I think OpenEXR should also be quite a programming-friendly format if you need to process it programmatically. I might be mistaken, but it might be worth looking into it. – Martynas Žiemys Apr 03 '20 at 07:26
  • If you look in to the very first image of the question, you can see, that I'm already rendering masks in .exr. Please take a look at the Problem images, and at the Box type render. You can see that ID mask node doesnt cover those pixels rendered with box filter type, which I need to be also covered for accurate annotation – cxnt Apr 03 '20 at 10:51
  • Also which way in your opinion would be best for annotation of 3D objects? – cxnt Apr 03 '20 at 10:58
  • Just figured it out cryptomatte works fine, when I render only plants, but not when I render the whole scene – cxnt Apr 03 '20 at 11:49
  • What do you mean by annotation? Could you give some example of how it needs to work in practice? – Martynas Žiemys Apr 03 '20 at 12:36
  • Sure, 1min, I'll let u know when Ill upload – cxnt Apr 03 '20 at 12:39
  • Look at the bottom of the question, I've updated it – cxnt Apr 03 '20 at 12:52
  • Cryptomatte otputs names of objects or materials with EXR format. I am not sure what software you use or how to make it work with it, but it works with Photoshop so it should be possible to get them from EXR programmatically. As for the mask picking up some background around anti-aliased(or should I say sampled) object, I think it is impossible to avoid unless the objects are rendered on transparent background and then the pixels contain only colors from the objects and nothing else since pixels usually contain color from multiple places. – Martynas Žiemys Apr 03 '20 at 13:08
  • Is there a way to isolate single object (single plant in my case) in compositor with cryptomatte ? – cxnt Apr 03 '20 at 13:16
  • Sure, just set up the Cryptomatte node to output it's mask and then use File Output node to save the file during rendering. It's just a bit tedious because you then have to do everything manually - set everything up, name the file... – Martynas Žiemys Apr 03 '20 at 13:21
  • I'm getting a black image, could you show nodes setup pls? Using Photoshop with latest exr-io – cxnt Apr 03 '20 at 13:40
  • 1
    Sure – Martynas Žiemys Apr 03 '20 at 13:56
  • Or better share a .blend file – cxnt Apr 03 '20 at 14:26
  • Is there any way to automatically add object to Matte objects? Right now I have to use Pick output of Cryptomatte and then use eyedropper to add all the objects – cxnt Apr 03 '20 at 14:40
  • Only Python scripting as far as I know. You can ask another question about that to get more help with that. – Martynas Žiemys Apr 03 '20 at 14:43
  • i've updated the question. could you please take a look? – cxnt Apr 04 '20 at 12:34
  • 1
    Why didn't Cryptomatte work? I cannot imagine a situation where this complete solution for masking would not be better than ID masks in all ways possible. You are now asking for something that is impossible, because every pixel gets sampled many times and takes it's value from multiple locations, so an image cannot be masked with only black and white mask. Pixels around the edges of objects and materials just have values from a few different things and it is impossible to separate them once they are mixed, this is not the same as anti-aliasing in raster renders, you can't remove it. – Martynas Žiemys Apr 04 '20 at 13:41
  • I will test cryptomatte today a lot, and will tell you if it work – cxnt Apr 04 '20 at 14:45
2

Is there any reason that you need to use width : 0.01px Blackman-Harris Filter? You are not sampling all area from that pixel but rather choose a very tiny dot to represent that pixel. See this post for further description.

If you want all the pixel being equally sampled, use box or set the width to about 0.67 px width enter image description here

Or just render the plants itself and use a math node to retrieve all pixel that have been rendered with color: enter image description here

HikariTW
  • 7,821
  • 2
  • 18
  • 37
  • So box should fix the problem? I've already tried box, that didnt make any difference tho. Should I try box + higher res? – cxnt Mar 26 '20 at 06:21
  • I'm not limited in using render settings, so any solution will be fine – cxnt Mar 26 '20 at 06:22
  • @cxnt1 Can you provide the box rendered image? ID-Mask do a threshold control which does not support transparent(make sense) – HikariTW Mar 26 '20 at 06:31
  • WHat is treshold control? do you mean alpha treshold? – cxnt Mar 26 '20 at 06:49
  • Question updated – cxnt Mar 26 '20 at 07:04
  • You will have to increase the resolution of whole image. There are not pixels to show branches of your plant/grass. It happens when size of the object is smaller than available pixel to draw that object – lost protonbender Mar 28 '20 at 10:47
  • So I should scale all objects to 1 or bigger? – cxnt Mar 30 '20 at 10:58
  • @cxnt1 well, lost protonbender are not correctly talking the thing. If any ray cast on to the plant, that pixel will contain the plant no matter how big is that pixel. But when Blender need to choose a main object Blender will probably choose the Majority of the ray cast sample. And in your case, the dirt might be chosen. So you might need to separated them or maybe there is a better solution – HikariTW Mar 30 '20 at 12:31