Raw histograms and raw converters

[ErikKaffehr]

In a way, I would suggest that photographers interested in image quality would be interested in a tool for viewing the raw data. A good such tool is RawDigger.



So, this shows a very small overexposed area. Seems to be ideal...

Now, looking at the same image in Capture One at defaults the clouds seem to be clipped...

Changing tone curve to 'Linear Scientific' we get:

Opening the file in Lightroom at default setting, we got no overexposure at all.


What I think the images above illustrate is that the histograms in the raw converter are no true representations of the raw data.

The image above is my raw conversions in Capture One


While this one is my conversion in Lightroom.

What are the benefits of 'Exposing to the right'?

Noise in the image is directly coupled to exposure. Increasing exposure by one EV yields a 41% advantage in noise. So, to get cleanest tonal response we would try to keep exposure at maximum, still avoiding clipping significant highlights in the raw data.

Best regards
Erik

 

[georgecleansman]

This is a very important aspect to get best image quality.
The right exposure has to be done on location and I guess that the histogram of many cameras is based on a jpg conversion image and don't show us no RAW data.

 

[pegelli]

Thanks Erik, I recently bought a license to RawDigger exactly for this reason and fully concur with your observations.
The simplest version of Rawdigger only costed me 20,99 €, which I think is well worth the ability to be 100% sure of the true histogram of all 4 RGGB channels.

The closest approximation of "real" clipped highlights in Lightroom is (for my cameras) to go to the default conversion in Process 2010 (mostly zeroed/blacks 5/brightness 50/contrast 25/linear curve) and set EV to -0,5.
Still not 100% perfect, but probably good enough for most cases. In process 2012 and later Lightroom does some highlight (and shadow) magic below the hood that can hide the clipping by making an educated guess about the colour of the clipped highlights based surrounding areas that are just shy of being clipped.

 

[OleBe]

There is a way for Lightroom to export one of your cameras DNG (just make sure the file is Adobe Standard processed) and load it into the Adobe DNG editor. You can change the curve to linear and save it as a new camera profile. Restart Lightroom and you can select it. This eliminates nearly all magic under the hood of Lightroom.

This is how I do it.

 

[Paul2660]

Use of the Linear Curve is very important, glad Eric mentioned it. It can and will save many images in C1, that appear to have blown highlights, and IMO it also helps prevent an issue I have noticed in C1 for years. This is when you select a sky especially blue, with few clouds to help break it up, if you work within the sky selection you often see a lighter band of sky towards where the sky hits the foreground subject matter (for example a mountain). The highlight tool also seems to accentuate this effect also. C1 loads by default a strong curve, in the Auto selection. There are some images where I prefer it, but it can and will get possibly overpowering.

Also, Eric, thanks for the post. Note that even though C1 shows the clouds to be blown even with the linear curve selected, the actual image produced shows none of this. Overall I tend to prefer the rendering of the C1 file especially the blue hue in the sky.

C1 with the use of layers, and the selection tools within the layers and selection tools to create various layers, and the ability to handle opacity within a layer, just offers to much more than LR and ACR. You still can't fade the opacity of layer in LR (it's not really a layer anyway, just an adjustment via brush or ND filter. And I still prefer the use of sessions in C1.

Paul C

 

[Godfrey]

Does this imply that everyone carries a laptop or tablet with them as they make their exposures and downloads each exposure to run software and analyze its channel signature? Whew. That adds a lot of time and equipment to a 'go make some photos' adventure.

Why not just bracket a few shots where the dynamic range is large and do all that later when selecting what to render?

G

 

[pegelli]

Why not just bracket a few shots where the dynamic range is large and do all that later when selecting what to render?

Nobody suggested this is happening in the field, so indeed it's happening (at least in my case) at home and then use one or another tool to determine which is the best one to pick.

Shall I get you a bucket?

 

[SrMphoto]

Thank you for the post, Erik, and +1 for RawDigger.

One good use of RawDigger is to evaluate bracketed shots for the best exposure (most light, no blown relevant highlights). As Erik shows, raw converters are not accurate representations of the raw data.

With ETTR, there is always the danger of blowing the relevant highlights. That is is worse than having more noise in the image. Our experience with the camera can help us avoid that and bracket when we are in doubt.

P.S.: I have found ETTR to be even more critical with smaller sensors because of the typically lower SNR.

 

[docholliday]

Also remember that "Linear Response" in C1 is a profile with a very minimal curve attached. In reality, it's "no" curve with just some high points tweaked to prevent clipping. Once you get to "Linear Scientific", it's a true flat profile with absolutely no curve attached - and no clipping protection at all.

LR doesn't have a linear response in that their "flat" curve isn't flat all and has all kinds of muddying to make it "easier" to use.

P1/C1 has recommended that Scientific shouldn't be used except for calibration purposes, sensor profiling, or extreme situations where raw data accuracy was needed. Even for document imaging and repro, they recommend "Linear Response".

In the end, histograms are still not accurate, guaranteed, or standardized. There's too much "magic" behind the scenes that we don't know about for each manufacturer to make their tool "the better choice". It really doesn't matter how it's done or with what tool, as long as it's something that you understand and can get repeatable results. With that said, I still prefer to use an instrument designed for giving me repeatable, standard values - a light meter.

 

[Godfrey]

Nobody suggested this is happening in the field, so indeed it's happening (at least in my case) at home and then use one or another tool to determine which is the best one to pick.

Shall I get you a bucket?

Don't know what I'd do with a bucket, but a bearer to carry all that gear would be handy. LOL!

So what you're saying is that all this analysis work is post-exposure time, during the selection and rendering process. Hmm. I've just learned to read what the LR or other tools are telling me when I tweak the images I'm selecting from in terms of what I can get. I'm not sure I really understand the point of analyzing all the exposures to such excruciating detail ... the results in processing are quite obvious to my eyes.

And remember: It's not the raw data from the capture that's the critical point, it's the response curve of the displays and/or printing papers that are the target of the rendering which present the ultimate limits to the rendering qualities. And those limits are severely more constrained than the range of the digital data itself.

I'll keep on bracketing exposures when I feel it's needed...

G

 

[pegelli]

Don't know what I'd do with a bucket, but a bearer to carry all that gear would be handy. LOL!

I thought you needed a bucket because the emoticon showed you were throwing up

So what you're saying is that all this analysis work is post-exposure time, during the selection and rendering process. Hmm. I've just learned to read what the LR or other tools are telling me when I tweak the images I'm selecting from in terms of what I can get. I'm not sure I really understand the point of analyzing all the exposures to such excruciating detail ... the results in processing are quite obvious to my eyes.

I think what Erik is showing is that the default conversion of a raw converter can show clipping when none of the channels are clipped and my experience is that the reverse can also happen (raw converter doesn't show clipping when one or more raw channels are clipped). If you already know that (and I guess you do) that isn't a surprise but for some mebers who are lower on the experience curve it can be useful information. Btw,I find RawDigger extremely user friendly to determine if there are blown channels and select the best shot from a bracketing sequence (if I indeed bracketed) and don't consider it "analyzing all the exposures to such excruciating detail"

I'll keep on bracketing exposures when I feel it's needed...

Excellent strategy

 

[Alan]

Now, looking at the same image in Capture One at defaults the clouds seem to be clipped...

What profile did you have selected in your C1 process recipe? What did you have clipping indicators set at?

 

[jng]

Nobody suggested this is happening in the field, so indeed it's happening (at least in my case) at home and then use one or another tool to determine which is the best one to pick.

Shall I get you a bucket?

LOL, actually there are a few of us off our rockers enough to do (almost) just that , although I only do some quick checks on the histogram and picker tool, no deep dives with software like Rawdigger.

In any case, bracketing is a good strategy although when stitching under rapidly changing light and/or for long exposures, this is not always an option. Best to get the exposure as close to where one wants it.

John

 

[Godfrey]

I thought you needed a bucket because the emoticon showed you were throwing up

I think what Erik is showing is that the default conversion of a raw converter can show clipping when none of the channels are clipped and my experience is that the reverse can also happen (raw converter doesn't show clipping when one or more raw channels are clipped). If you already know that (and I guess you do) that isn't a surprise but for some mebers who are lower on the experience curve it can be useful information. Btw,I find RawDigger extremely user friendly to determine if there are blown channels and select the best shot from a bracketing sequence (if I indeed bracketed) and don't consider it "analyzing all the exposures to such excruciating detail"

Excellent strategy

LOL! Yeah, GetDPI icons are oddly chopped up a good bit of the time.

And yeah ... I guess I spend a good bit of time studying everything about my cameras' behaviors, learning to read what they're telling me, and just playing/experimenting with settings and results that using analytic tools seems way over the top. RawDigger latest version seems fine (downloaded a copy for testing), I just don't find it particularly important or necessary. (I've been doing photography for 55+ years, and playing with digital capture since the middle 1980s ... I usually set the exposure by eye and experience, then meter, then analyze and make a bracketed set. On post analysis, I am usually much closer to correct than the darn meters were and have to factor in their error when I use them. Such it is. )

G

 

[ErikKaffehr]

Does this imply that everyone carries a laptop or tablet with them as they make their exposures and downloads each exposure to run software and analyze its channel signature? Whew. That adds a lot of time and equipment to a 'go make some photos' adventure.

Why not just bracket a few shots where the dynamic range is large and do all that later when selecting what to render?

G

Hi,

Using RawDigger can be a learning experience

LOL! Yeah, GetDPI icons are oddly chopped up a good bit of the time.

And yeah ... I guess I spend a good bit of time studying everything about my cameras' behaviors, learning to read what they're telling me, and just playing/experimenting with settings and results that using analytic tools seems way over the top. RawDigger latest version seems fine (downloaded a copy for testing), I just don't find it particularly important or necessary. (I've been doing photography for 55+ years, and playing with digital capture since the middle 1980s ... I usually set the exposure by eye and experience, then meter, then analyze and make a bracketed set. On post analysis, I am usually much closer to correct than the darn meters were and have to factor in their error when I use them. Such it is. )

G

Hi,

My point was not that a raw analyzer is a necessary tool. Rather what I wanted to demonstrate was that histograms in raw developers show processed data. Some Phase One models have optional raw histograms and any user using any normal raw converter may get confused when Capture One shows overexposure. On the other hand, Lightroom may hide some clipping as it applies significant highlight reconstruction by default.

I don't see RawDigger as a field usable tool, rather as a learning tool.

Normally, I don't bracket a lot. Foremost, I really hate culling images.

But, with bracketed images I find FastRawReader to be a usable tool. It also operates on raw data and it is very usable for fast evaluation of exposure and sharpness.

Best regards
Erik

 

[Godfrey]

... Using RawDigger can be a learning experience ...

My point was not that a raw analyzer is a necessary tool. Rather what I wanted to demonstrate was that histograms in raw developers show processed data. Some Phase One models have optional raw histograms and any user using any normal raw converter may get confused when Capture One shows overexposure. On the other hand, Lightroom may hide some clipping as it applies significant highlight reconstruction by default.

I don't see RawDigger as a field usable tool, rather as a learning tool.

Understood now.

Perhaps what I was reacting to is that this statement of the intent of your piece should have been the first thing in it, rather than something which is added quite a lot later. As a (retired) technical writer, my work and user feedback taught me that it is good editorial practice to set up the intent and expectations, the reason, for the ensuing discussion at the beginning of a piece rather than after comments about it have provided evidence that the intent wasn't clear.

With this addition to your article, please consider my comments as constructive criticism on the article...

G

 

[davidrm]

As mentioned above, CaptureOne shows clipping based on output rather than input. So, if your current recipe profile happens to be set to sRGB you’ll see a more clipping than if it is set to ProPhoto, say.

Personally I find C1’s clipping indicators to be rather pessimistic and not too reliable. I’ve noticed cases where, say, I drag the exposure slider to a point where clipping warnings just turn off, I can then nudge it back up quite some way before they turn on again.

Usually I just turn them off and keep an eye on the histogram...

 

[SrMphoto]

As mentioned above, CaptureOne shows clipping based on output rather than input. So, if your current recipe profile happens to be set to sRGB you’ll see a more clipping than if it is set to ProPhoto, say.

Personally I find C1’s clipping indicators to be rather pessimistic and not too reliable. I’ve noticed cases where, say, I drag the exposure slider to a point where clipping warnings just turn off, I can then nudge it back up quite some way before they turn on again.

Usually I just turn them off and keep an eye on the histogram...

When only one channel has blown highlights, software like LrC can reconstruct the blown channel using the other two channels. That is why it is easier to "recover" highlights with color sensors than monochrome sensors, where blown highlights are not recoverable. The disadvantage of mentioned highlight recovery is that the color may change in the recovered highlights as the software is guessing the blown channel's value.

 

[TimoK]

What profile did you have selected in your C1 process recipe? What did you have clipping indicators set at?

As mentioned above, CaptureOne shows clipping based on output rather than input. So, if your current recipe profile happens to be set to sRGB you’ll see a more clipping than if it is set to ProPhoto, say.

Personally I find C1’s clipping indicators to be rather pessimistic and not too reliable. I’ve noticed cases where, say, I drag the exposure slider to a point where clipping warnings just turn off, I can then nudge it back up quite some way before they turn on again.

Usually I just turn them off and keep an eye on the histogram...

I did check my settings in C1 for clipping warnings. There was (in RGB values from 0 to 255) 6 and 250. I don't remember if those were C1 defaults or my own custom settings. Not pessimistic but very safe for print production. There is still much room in tones beyond 250 before the real clipping. I think it's better to raise those limits than leave them out of sight.

It seems the forum software don't let me quote third time. So netx sentence is quoted from Erik's post:
"My point was not that a raw analyzer is a necessary tool. Rather what I wanted to demonstrate was that histograms in raw developers show processed data. Some Phase One models have optional raw histograms and any user using any normal raw converter may get confused when Capture One shows overexposure. On the other hand, Lightroom may hide some clipping as it applies significant highlight reconstruction by default.
I don't see RawDigger as a field usable tool, rather as a learning tool."

I think RawDigger is an overkill for what it is usable. Why to buy a special program only to look at raw histogram and maybe learn something?

When I most often use RawTherapee to process my hobby landscapes, I can toggle between RGB histogram ( output) and RAW histogram (input). I see the clipping. So I can do my work that in mind.I use RT's raw histogram as a tool. For example to choose the right exposure from my bracketed files. It's a part of my workflow, not anything special.
When working with my files I can possibly at the same time learn something.
And RT is free.