10

I am trying to write the depth map of a scene in a 16-bit PNG, without any sort of normalization. I would like to pixels of such depth image to be set to the distance with respect to the camera, and of course the far clip plane value if no object present.

I tried with the following node setup, but I get values up in the PNG pixels which do not make any sense, in the range of [3100,65000] while my object is located rougly at 1 meter from the camera. Near and far clipping plans are 1cm and 10m respectively. I guess they are suffering some sort of normalization, but I cannot figure it out.

enter image description here

Thanks

Dan
  • 245
  • 1
  • 3
  • 10
  • 1
    The z depth map contain values higher than 1,to store it in an image you will have to normalize it,Can you clarify the result yo want? – Omar Emara May 10 '16 at 16:42
  • How do these numbers relate to the coordinates of your objects? Remember that 'meters' is an abstraction. Nothing in Blender is actually measured in 'meters.' It just says that some number of Blender units is called a "meter." Take a look at the "real" numbers of those distances, and I'll be they make more sense. – Matt May 10 '16 at 16:43
  • @Matt I set my scene to use Units in Metric, and I can see the location of each object in meters when I look at the properties. For example, my camera is located at (0,0,0) and a plane object at (0,0,10m) (this are meters, explicitly displayed in the Transform box). I would like then the Z values of my depth maps to 10, if the object is in the camera view. – Dan May 10 '16 at 16:54
  • @OmarAhmad I want the z depth map values to be the distance with respect the camera, in Metric units (as my scene is set) – Dan May 10 '16 at 16:55
  • @Dan that might be much more difficult than you think it is. There's conversion going on behind the scenes, and you're going to have to confirm what numbers Blender actually uses, rather than what it shows the user. – Matt May 10 '16 at 16:58
  • @Matt I implemented something similar in OpenGL a while ago... if you have access to the depth buffer it is actually trivial. I thought Blender would provide such functionality some how? – Dan May 10 '16 at 17:00
  • @Dan When you get data you don't expect, there are two possibilities. The one every jumps to is "the data is wrong." But the other possibility is "your expectations are wrong." Maybe the data is right, but it's not what you expected to see. My next step would be to convert that PNG to a normalized PNG, and see if that produces an image. If it produces garbage, then you know that the data you have is not the data you want. If it DOES make an image, then you know you've got the right data. You just have to figure out how Blender is transforming 65000 to 10.000, or whatever. – Matt May 10 '16 at 17:31

4 Answers4

18

Short Answer

If you are looking for accuracy, Do not save the Z pass as PNG or any other display referred format. OpenEXR is the only format that will allow you to use the linear information from the Z pass.

The only way to get a Z depth pass without distortion is by avoiding any kind of processing except [proper] inversion or normalization. Just plug the Z information to a file output node, and save as OpenEXR.

Long Answer

First let's separate the apples from the oranges:

The Z pass should not be thought of as an image, since its values do not fall into the parameters set for most image formats.

  • What is the Z pass then?

    The Z pass is the absolute distance from the camera to objects in the scene for every given pixel on the image. The camera is the 0 on the scale and the units used for its values are the working units set for the scene (blender units, meters, cm, inches, etc). The minimum and maximum value are clipped according to the camera's start and end clip distances.

    The Z depth values work in a linear scale, meaning that 6 will be twice as far than 3 and one half of 3 will always be 1.5

  • The Z pass is not meant to be viewed.

    The main issue is that Z depth information, will generate values larger than 1 and image formats cannot deal with values larger than 1 (where 0 is black, and 1 is white)

  • What happens when you normalize the Z pass?

    The minimum and maximum values are set as 0 and 1 respectively and all other values are spread in a linear scale in between.

    The real effect of doing this is that the scale is both compressed and offset from the real values. The values for absolute distance are converted in relative values and from then on you have to do conversions to find what the real values are.

  • What happens when you invert the Z pass?

    The maximum value is set to 0 and the minimum value is set to the maximum value. All other numbers in between are also inverted. Inversion is a reversible (invertible) operation (source)

  • Why do people normalize the Z pass then?

    Because the information contained can then be viewed as an image, since it has values from 0 to 1 that can then be expressed as an image. Normalize only when you want to make sure that the Z pass is working, or to use it on programs that cannot deal with the Z channel properly and need the information as an image in grayscale.

  • So why not PNGs or other formats?

    Normalizing the image to make it fit on a 0 to 1 scale is really not the problem (because even though normalizing has moved the values for the scale, it has kept the values linear). But when you save as an image format you are re-mapping those values yet again to a very limited number of values. Saving to an 8bit format will result in 255 possible values, greatly reducing the precision of the information. Even on 16 bit where you can have a much larger different gradations from black to white (4096) you might find that values that are close together are no longer represented.

    But the main reason not to use display referred formats like PNG is that those formats presume to be encoding values in sRGB, that is not a linear format anymore, but has a transfortmation curve in which 3+3 might not equal 6 anymore... The values will have been re-mapped yet again to a point that will not reflect the proper scale of the distances represented on the Z depth information.

  • How is OpenEXR different?

    OpenEXR is a format that has been designed to store linear data and has no issues with values larger than 1. The information suffers no distortion and is as precise as when you rendered your scene in the first place.

Amir
  • 3,074
  • 2
  • 29
  • 55
  • there is an option in blender to store png images as BW. This should preserve the linearity? Can you confirm this? – Armen Avetisyan Jan 06 '17 at 01:27
  • @ArmenAvetisyan Z depth shouldn't be thought of as images but as data. If you use PNG and save as data (with no display transform applied to it by disabling save as render) you will still only be able to store only the values between 0 and 1, or a compressed scale if you normalize. Display referred image formats like PNG are probably useful for very small scale scenes... .. –  Jan 06 '17 at 03:34
  • I understand that 8-bits do not resolve 32-bits which is your point in essence. Apart from that, the normalization in [0...255] or [0.0 ... 1.0] is a linear transform where magnitude/extent information vanish. However, my question is more if saving in BW png file format you are able to preserve linearity which may be distorted by sRGB file format. Because BW is not sRGB. – Armen Avetisyan Jan 06 '17 at 13:42
  • 1
    @ArmenAvetisyan you are confusing things: black and white (and color) can be encoded as non linear or not, the display transform set in the color management section determines that. The default options on blender will encode images through a sRGB transform, which has a "gamma" curve, It is a misconception that black and white images are exempt from such a transform. You can indeed save an image without the sRGB transform as raw or "data", color or not has nothing to with it. –  Jan 06 '17 at 15:12
  • This answer is helpful - especially because exactly what I needed (and failed to comprehend) was that I needed to normalize the Z layer before rendering. I'm using the Z layer as a depth image in Substance Designer. – Jesse Williams Aug 10 '17 at 00:30
  • @cegaton Would first inverting the values and then normalizing be a valid, reversible operation as well? – Amir Mar 10 '18 at 06:40
  • @cegaton "The only way to get a Z depth pass without distortion is by avoiding any kind of processing. Just plug the Z information to a file output node, and save as OpenEXR." Does this mean that using View node will distort the data?Based on the answer given here it seems that obtaining the raw data from bpy.data.images['Viewer Node'].pixels should still give you the raw, 32-bit values of the rendering. Am I correct? – Amir Mar 10 '18 at 06:43
  • 1
    @Amir I was referring to saving the image as EXR instead of display referred formats. If you use the viewer you can sample the depth pass correctly as well. –  Mar 10 '18 at 07:29
5

Use the z output and divide it by the value you set as the far clipping plane of the camera. Your output image will be black for a pixel located in the camera and completely white for positions at the far clipping plane. I'd switch to Blender units for the values and than back to metrics, just to be sure.

Node setup

In this example I have the "end" clipping value to 100.

You shouldn't use the normalize node because you won't know the farthest distance from the camera. Let's say you have a plane in the background at 80 from the camera, with normalizing it will output a value of 1 (since it is the biggest distance in the scene) instead of 0.8 the divide node will output.

piegames
  • 1,969
  • 1
  • 11
  • 26
  • Sounds good! Any idea if this can be saved as PNG-16 bit, or must use EXR? – Dan May 10 '16 at 18:36
  • The image will be outputted where and how you set it in the

    Properties panel -> Render tab -> Output

     – piegames May 10 '16 at 19:18
  • 2
    Do not use PNG! Export the Z pass with no processing or any nodes in between. PNG will destroy the linear scale of the Z inormation. Use OpenEXR, is the only format that will keep your information undistorted. –  May 10 '16 at 19:49
  • How doeas PNG destroy/distort the data? The z pass output returns the distance of the pixel to the camera in Blender units. The maximum distance it will return is the distance of the far clipping plane of the camera. If I divide the z value by this, I'll get a factor from 0 to one that will be mapped to 0-255 in PNG without any errors. I use this quite often and never had any problems. – piegames May 11 '16 at 14:28
  • 2
    Besides the clipping issues of data larger than 1, and the loss of precision (by mapping the full range to 255 values in 8bit or 4096 in 16 bit) PNG being a display refered format will use a sRGB transform curve. So the data will not be linear any more. –  May 11 '16 at 15:12
2

Please refer to piegames answer. PNG is perfectly fine for a "transport" data format. You can even use PNG and push a full 32bit raw into each pixel, then in your shader or depth system unpack the RGB/A into the original depth value.

Im not sure why @user1853 is pushing EXR, this is not used very heavily for depth maps in my experience. EXR is not a great transport format for games, simulations and many other systems. You often need to "touch up" depth maps and having an easily editable format is often very handy. EXR is often _over_kill_ for scenes. I have seen EXR used for textures with depth maps incorporated into the Alpha component, but you can do this in PNG, and other formats.

Additionally, depth maps when used in render systems are rarely linear. Nearly all depth map rendering software uses some form of logarithmic format - high detail near camera, low detail far away. This is very common. So consider a depth map specific format.

The OP solution is also fine. What you are missing is that you have packed your data into a 16bit (65535 max) data range, so it is quite correct ;)

Divide by your data size to get the correct depth 0..1 value. For example: [3100,65000] == [0.0473, 0.9918] Also remember that will be scaled by far-near: 10m - 1cm = 99cm To be clear this means [0.0473, 0.9918] represents 99cm.

Hope this helps.

-- Some notes to above. Clarity here is important, and I think some information is being very muddled.

  1. There is no "perfect" graphics data transport system It depends heavily on your use case, and even EXR cannot guarantee replication (has a number of different representation systems within the format) https://en.wikipedia.org/wiki/OpenEXR

Dont be fooled into the concept of a "file format" that will rid you of precision errors - that is just not true. Source data may be in 32bit or float 32bit.. which is very innaccruate if you are doing 200Km train sim scenes :)

  1. PNGs are fine for Z depth. Even JPEGS can be used (have done so on many web projects to minimise size constraints). It all depends on your use case. Generally these days, depth buffers are not even put into files (its kind of a waste of time). Render the object in the target render system and capture the buffer and use it there.

Formats Ive used in production of games, sims and other: PNG, TGA, PPM, RAW, JPEG, even BMP :)

  1. Depth Maps are subjective. They depend on the following important factors:
    • Target use case - is it a single object, or a whole scene, or a combination? Consider in engine rendering if you need high accuracy.
    • Data complexity. If you have only a simple plane or cube, then a high quality depth map is obviously not needed. High frequency surfaces are very hard to capture in a depth map, and I wouldn't recommend it.
    • Combining sources. Often you want to use a Depth Map, but then layer in a Normal, Tangent and Bitangent setup so you can have properly detailed surfaces. Depth Maps alone are usually not very useful.
    • Alternatives - As mentioned, in-render depth maps, a voxel format (these are much more compact and just as accurate), or a simple raw data format (floating point or 32bit/64bit etc data stream).
David Lannan
  • 21
  • 1
-2

My question might be a bit irrelevant to this.

I have created a simulated knee environment in Blender and for every frame, I have the depth, camera pose, and the rendered image. Using these three ingredients, I should be able to reconstruct the 3 point cloud and mesh of the environment using other approaches such as TSDF (https://github.com/andyzeng), but that is not happening. Even though I put the correct camera matrix intrinsic (I think).

To cross check the Blender camera pose info, I used only the depth images and Kinect Fusion to estimate the camera pose and used the estimated camera pose info for depth image on TSDF and still it did not work. I tried the same approach (estimating the camera pose using Kinect Fusion) on the demo data of the TSDF and it worked (Kinect Fuseion-estimate pose + groundtruth depth). That means the Kinect Fusion could produce the correct camera pose info as good as groundtruth pose, to be used with TSDF. This narrows down the issue I have to the groundtruth depth that Blender produces.

My question is that, do I need to do any transformation on the depth images that Blender produces?

metaphor_set
  • 6,625
  • 2
  • 22
  • 35
Yaqub Jonmohamadi
  • 1
  • This is a question, not an answer. New questions should be posted as their own distinct question and not as an answer to existing questions. – Gary Carlyle Cook Jun 22 '22 at 12:25