As a Corollary to this question (Using Blender with wider gamut than 709 primaries (like ACES)). How can I use blender to integrate CG, and footage from cinema cameras using ACEScg?
My goal would be to convert the footage to ACEScg in Resolve, generate and composite in blender and return to resolve without having to re-grade the images. Is that workflow doable?
As a loose outline, it’s much better to skip the ACES complexity and focus on what is required. ACES comes with its own entire set of issues, not the least of which is obfuscating the goals and the technical needs.
Integration requires using a specific encoding. That is, the key here is that we seek the radiometric ratios as they existed in the scene. This is perhaps the number one aspect of integration that is, and has been, overlooked for a long, long time amongst many folks.
The key here is to understand that cameras capture scene data. However, despite all cameras doing so, they all do not encode the data in a manner that is useful. Typically, camera encodings can be described in one of two manners:
You’ll find countless examples online where people take random footage and attempt to integrate it. Some even can look pretty remarkable, such as some of Ian Hubert’s work.
The sad part is that this isn’t the appropriate means to make life easy for themselves. If the footage “looks ok” when played back, it likely is the absolutely wrong encoding to work with, and will result in no end of massaging, sucking up countless hours of time and energy, only to never-quite-look terrific.
This should be common knowledge now for anyone who has tried to take a generic JPEG photo found online and attempted to integrate it into a Filmic based scene; it just can’t be done well. Why? Because the display-ready image has become an impossible to decode combination lock on its way to becoming pretty to look at.
It might shock a majority of readers to learn that you don’t need an Arri Alexa LF to get these sorts of magical encodings. Sony, Nikon, Panasonic, Fuji, Canon, and other vendors now commonly include a log-like encoding scheme in their affordable consumer grade cameras. A good number of them offer the ever-more-useful-and-important on-board 10 bit or greater bit depth recording.
While countless YouTube videos wax lyrical with misinformation about “flat” picture styles and what they are good for, the value of the documented log encodings is entirely and completely missed; log-like encodings capture and encode scene reflectance!
That is, for every log-like encoding scheme such as Sony’s SLog1-2-3, Fuji’s F-Log, Panasonic’s V-Log, Nikon’s N-Log or Canon’s C-Log1-2-3, there is a documented function available in a PDF that describes how the values are encoded with a unique transfer function.
Unlike a display’s transfer function that encodes display linear light however, the gold is in the little detail that those normalized code values can be unlocked back to scene reflectivity!
Where sRGB’s transfer function can be undone to reveal the display linear light output from 0-100%, a log-like encoding can be taken back to the scene’s ratios from some low value to some high value.
To really hammer this point home, a simple and revealing test can be done:
Once saved. Open up another instance of Blender.
Note the values? Despite the fact that the sixteen bit integer TIFF was encoded using a device referred 0-100% range in both cases, when properly transformed via an appropriate transfer function such as Filmic Log, the values magically are transformed right back into the scene referred domain. The aesthetic “looks good” Filmic with Base Contrast? Hrm... only 0.0 to 1.0 representing the display linear aesthetically warped light, not the scene.
And guess what? Those logs for all of those awesome cameras listed above? They’ll do exactly the same thing!
This allows one to take the log-like encoded footage and seamlessly integrate it into any CGI with nearly no effort whatsoever. Use them as reflections, as emissions, and have interactive lighting on all your work.
There is one little gotcha with the log-like encodings; they are almost always coupled with a unique set of primaries that are different from the BT.709 assumed lights in your sRGB display, and the default set of rendering lights in Blender for Cycles and Eevee. The details of the particular encoding are unfortunately often referenced as some sort of a “gamut”. That is:
One must take care to, after linearization with the proper transfer function, to also take the primary lights to the same primaries in the rendering / working space. This is a slightly nuanced subject, and likely requires another answer unto itself once the camera specifics are known.
Beyond all doubt, this is a mind blowing experiment for the intrepid 3D peep to muck with, and hugely encouraged for anyone who has a log-capable camera.
But what about all of us schmucks who don’t have the releativey more recent cameras with log encoding options? There is indeed hope with a little bit of effort.
It’s entirely possible to take generic camera footage and document the transfer function manually. It’s a little elbow work, but the results will be similar to a native log, and the output will be significantly better than ignoring the core issues. Thankfully, there is an absolutely amazing set of tutorials to achieve just this via Paul Chambers’ website. After having read that amazing series, there is another little hidden gem on his site to help you use that radiometric scene reflectance too!
The key point in all of this is never let some overhyped bit of a plug-in or system prevent one from learning the core principles. No amount of any canned system can help you here; it takes a bit of elbow grease and learning to unleash a heck of a lot of amazing work. You never know... there might even be a super top secret and visually stunning project being developed in dark crustacean labs. I wonder what would happen if someone asked them...
