Another comparison of Sony A7rII raw formats

[ErikKaffehr]

Hi,

I am somewhat skeptical, about the benefits of the uncompressed Sony raw format. I have seen some cases where the delta coding showed up, like the demos on DPReviews and the start tracks image that Diglloyd has shown. Those artefacts are facts in my view. It is a good thing Sony got rid of that.

But I don't really think there are other benefits of the uncompressed raw, or if there are they are not obvious to me.

I set up a difficult target, intended to use all of the dynamic range and have a lot of smooth graduations. So I have porcelain coffeecup and a black one over anthracite and a reflection in the glass behind.

Some crops are here: http://echophoto.dnsalias.net/ekr/Articles/SonyRAWCompression/

The raw images are here: http://echophoto.dnsalias.net/ekr/Articles/SonyRAWCompression/RAWS/

This crop is a highlight area, one stop below saturation:

Uncompressed:

Compressed:

This one is near black uncompressed:

And compressed:

As a side note, importing the files as DNG in Lightroom gives comparable file sizes.

So, what is my take on this? Mostly that I don't see any obvious artefacts. There is a small difference in tonal scale. It could be a variation of flash output, but I don't think that is the cause.

From what I have seen here the compressed and the uncompressed file are very similar.

Best regards
Erik

 

[ErikKaffehr]

Hi,

A small update:

In the first posting I tried to adjust one image to taste and keep the two in sync. After that I tried to just do a +5EV exposure push in Lightroom. Here I have noticed a colour shift towards brownish on the compressed image while the uncompressed stayed neutral. Pictures coming tomorrow. Time to go and earn some money to pay for all that stuff.

This crop is uncompressed:

And this is compressed:

I have checked this in both Lightroom, using different profiles, and also in Capture One. Results were always similar.

Best regards
Erik

 

[Quentin_Bargate]

Hi,

A small update:

In the first posting I tried to adjust one image to taste and keep the two in sync. After that I tried to just do a +5EV exposure push in Lightroom. Here I have noticed a colour shift towards brownish on the compressed image while the uncompressed stayed neutral. Pictures coming tomorrow. Time to go and earn some money to pay for all that stuff.

Best regards
Erik

Erik

I can see no material difference. I'm thinking of switching back to compressed RAW.

 

[iiiNelson]

Hi,

I am somewhat from Alabama, that is skeptical, about the benefits of the uncompressed Sony raw format. I have seen some cases where the delta coding showed up, like the demos on DPReviews and the start tracks image that Diglloyd has shown. Those artefacts are facts in my view. It is a good thing Sony got rid of that.

But I don't really think there are other benefits of the uncompressed raw, or if there are they are not obvious to me.

I set up a difficult target, intended to use all of the dynamic range and have a lot of smooth graduations. So I have porcelain coffeecup and a black one over anthracite and a reflection in the glass behind.

Some crops are here: http://echophoto.dnsalias.net/ekr/Articles/SonyRAWCompression/

The raw images are here: http://echophoto.dnsalias.net/ekr/Articles/SonyRAWCompression/RAWS/

This crop is a highlight area, one stop below saturation:

Uncompressed:

Compressed:

This one is near black uncompressed:

And compressed:

As a side note, importing the files as DNG in Lightroom gives comparable file sizes.

So, what is my take on this? Mostly that I don't see any obvious artefacts. There is a small difference in tonal scale. It could be a variation of flash output, but I don't think that is the cause.

From what I have seen here the compressed and the uncompressed file are very similar.

Best regards
Erik

One thing of note that's been discussed in the past. Never convert your ARW files to DNG in Lightroom. It throws away some data and compresses the file. I know on the A7r the files go from about 37mb to 31mb.

Oh and I agree there isn't much of a difference but then I never saw any artifacts on my end in normal usage when I did my part to take the picture.

 

[dmward]

I am somewhat from Alabama, that is skeptical, about the benefits of the uncompressed Sony raw format. ...

Erik, is this a reference to the "show me state" if so, its Missouri.

Which tone curve did you use for these tests?

If you used the Adobe Standard or any of the camera profiles intended to reproduce the in-camera JPG processing the tone curve has a toe and shoulder applied that compress highlights and shadows near the clipping points. That, in my view, hides the potential loss of detail caused by lossy file compression.

That's why I used an outdoor scene with very gradual tonal changes from 255 to at least 235 and 0 to 20. Those are the portions of the tonal range that are important for ensuring detail and most likely to be lost by lossy compression. Those ranges are the portion of the tonal scale from the white square on a Color Checker to specular and from the black square to black-no detail.

 

[Wayne Fox]

Erik

I can see no material difference. I'm thinking of switching back to compressed RAW.

For me the problem is I don’t trust compressed, because even though 99% of the time there seems to be no visible difference and there are plenty of examples where it doesn’t (in fact most tests show that), there is enough anecdotal information that it can happen at least on occasion.

And since Murphy seems to rule most of the time, it would probably end up happening on some terrific capture ...

 

[ErikKaffehr]

Hi,

I absolutely agree, that is a very good point.

Best regards
Erik

For me the problem is I don’t trust compressed, because even though 99% of the time there seems to be no visible difference and there are plenty of examples where it doesn’t (in fact most tests show that), there is enough anecdotal information that it can happen at least on occasion.

And since Murphy seems to rule most of the time, it would probably end up happening on some terrific capture ...

 

[ErikKaffehr]

Hi,

Adobe standard but also Capture One. Not sure I agree on tone and shoulder compression, more like an S-curve. That obviously also compress toe and shoulder, of course.

On the other hand, I was using highlight slider and shadow slider quite extensively.

My processing was pretty much what I do on any raw. Compress highlights and lift shadows, but I made a significant effort to get a very large scene illumination range under controlled conditions.

Best regards
Erik

Erik, is this a reference to the "show me state" if so, its Missouri.

Which tone curve did you use for these tests?

If you used the Adobe Standard or any of the camera profiles intended to reproduce the in-camera JPG processing the tone curve has a toe and shoulder applied that compress highlights and shadows near the clipping points. That, in my view, hides the potential loss of detail caused by lossy file compression.

That's why I used an outdoor scene with very gradual tonal changes from 255 to at least 235 and 0 to 20. Those are the portions of the tonal range that are important for ensuring detail and most likely to be lost by lossy compression. Those ranges are the portion of the tonal scale from the white square on a Color Checker to specular and from the black square to black-no detail.

 

[k-hawinkler]

Clearly what we need is lossless compression!

 

[mjm6]

Can you take the images and do a subtraction of the two in PS? You will surely see noise, but I suspect you will also see an outline of the cups that reflects the differences associated with the compression. Be careful not to do any image file format conversions in LR or PS before you do this of possible to avoid introducing another layer of potential sources of differences.

Just because you have not hit the threshold for obvious image degradation does not mean that the image is not being altered. Just a question of whether it is significant to result in a problem in your images, which it clearly is for star shooters.

I suspect that the compression is meaningless for a lot of images, hence why it has taken so long for the resolution of this problem.


---Michael

 

[mjm6]

One thing of note that's been discussed in the past. Never convert your ARW files to DNG in Lightroom. It throws away some data and compresses the file. I know on the A7r the files go from about 37mb to 31mb.

Oh and I agree there isn't much of a difference but then I never saw any artifacts on my end in normal usage when I did my part to take the picture.

Can you provide a citation for evidence that converting to DNG causes the file to contain less meaningful data? I appreciate that the file may be smaller, but it may not contain less usable information, depending on how it is compressed. I'm trying to figure out my long term storage options, and don't want to be throwing out any meaningful data along the way...

---Michael

 

[ErikKaffehr]

Hi,

My guess is that DNG uses Huffman coding or ZIP-algorithm. So I don't think they loose any bit, but they will probably/possibly not include all vendor tags. That probably doesn't matter if you use Lightroom or any raw converter properly supporting DNG as I think raw converters are pretty generic and probably don't use a lot of vendor specific tags.

If you use vendor programs, like Capture One or Canon's raw converter, those may make use of vendor tags possibly not included in DNG.

What is included in DNG is decided by the program doing the DNG conversion.

Best regards
Erik

Can you provide a citation for evidence that converting to DNG causes the file to contain less meaningful data? I appreciate that the file may be smaller, but it may not contain less usable information, depending on how it is compressed. I'm trying to figure out my long term storage options, and don't want to be throwing out any meaningful data along the way...

---Michael

 

[ErikKaffehr]

Hi,

I did do that. I aligned the two exposures before subtraction. The resulting image was black.

After that I pushed exposure 5 stops:

Here is a detail of the pushed image:

Raw images are here: http://echophoto.dnsalias.net/ekr/Articles/SonyRAWCompression/RAWS/

Best regards
Erik

Can you take the images and do a subtraction of the two in PS? You will surely see noise, but I suspect you will also see an outline of the cups that reflects the differences associated with the compression. Be careful not to do any image file format conversions in LR or PS before you do this of possible to avoid introducing another layer of potential sources of differences.

Just because you have not hit the threshold for obvious image degradation does not mean that the image is not being altered. Just a question of whether it is significant to result in a problem in your images, which it clearly is for star shooters.

I suspect that the compression is meaningless for a lot of images, hence why it has taken so long for the resolution of this problem.


---Michael

 

[thompsonkirk]

I've read that DNG conversion is virtually lossless; so you can shoot the large uncompressed files, convert them as lossless DNGs, and have the option that Sony left out, in between lossless uncompressed and lossy compressed.

I haven't had a chance to check this out in practice (except to note that DNG files are considerably smaller). But in the short run I've been shooting everything in uncompressed, using these files for solo shots, but converting files to DNGs in instances when I want to stitch them.

Kirk

 

[algrove]

One thing of note that's been discussed in the past. Never convert your ARW files to DNG in Lightroom. It throws away some data and compresses the file. I know on the A7r the files go from about 37mb to 31mb.

Oh and I agree there isn't much of a difference but then I never saw any artifacts on my end in normal usage when I did my part to take the picture.

So to be clear for 'ol me, what is the best process from your experiences to take ARW files into LR?

 

[dmward]

... Adobe standard but also Capture One. Not sure I agree on tone and shoulder compression, more like an S-curve. ...

Erik,
That is what I was pointing out. The benefit of digital is that we can get the entire tonal range, without a toe or shoulder compressing the extremes. Then apply the tonal curve characteristics we want in processing.

Lightroom, Capture and others, use tonal curves and toe and shoulder compression without letting us know what and how its applied. When combined with lossy compression that also affects the toe and shoulder regions of the tonal curve, it means a greater extent of alteration outside our control.

Linear tone curve inherent in a raw file should be preserved into the software so we have total control over how shoulder and toe region of tone curve is applied on an image by image basis.

It appears that its possible with both Lightroom (via a custom camera profile) and Capture via the tone curve options to have the raw data imported with a linear tone curve.

In my view, that is the starting point for evaluating how Sony lossy compression affects the extremes of the tonal range. If the software is applying its shoulder and toe compression it hides the lossy compression impact.

 

[mjm6]

Erik,

Thanks for doing the subtraction...

It appears that the image you have doesn't have enough contrast in it to cause visible artifacts for the most part, as the subtraction image took a huge push to get visible differences... However, the compression artifacts aren't an intentional image adjustment like edge enhancement or unsharp masking, so I would expect the difference to be pretty subtle for sure.

Possibly, you are shooting at a low enough ISO that the DR of the camera is capable of covering the DR of the scene acceptably. I'm wondering what would happen to a similar image if you shot it at 3200 with and without compression. Based on the graphs I've seen, the DR drops by 4 stops by 3200, and combining this with high contrast conditions might be a possible source for problems.

Ultimately, I agree, that there are only very subtle (at most) differences in the two images you provided, and the calculated difference image bears that out.


---Michael

 

[ErikKaffehr]

Hi,

Using high ISO is essentially under exposure. The Sony A7rII is said to make use of an Aptina patent to improve SNR at 640 ISO and above, essentially by reducing full well capacity (*).

I did develop the images in DCRaw and into linear gamma space, and that didn't change the conclusion.

What I noticed is that with exposure pushed 5 EV the dark areas turn brownish in the compressed image while in the uncompressed they stay neutral. This also applies to the DCRaw conversion. I would also say a significant deviation, perhaps not visible in real world images, but very much observable in experiments.

I cannot explain the colour shift but it seems to be related to compression.

The reason I don't see artefacts on edges is probable that edge contrast is not high enough. The delta compression is lossless under a wide variety of conditions, but it will have artefacts if contrast is high enough, like on the star trails often shown.

I will check out 3200 as I still have the setup standing.

Best regards
Erik

(*) According to the said Aptina patent the photodiode is often connected to a capacitor to increase full well capacity (FWC). The voltage from the pixel is proportional to captured photons / FWC. In the Aptina patent the condenser is connected to the pixel trough a transistor that can disconnect the capacitor at a certain ISO, thus raising the output voltage from the cell.

Erik,

Thanks for doing the subtraction...

It appears that the image you have doesn't have enough contrast in it to cause visible artifacts for the most part, as the subtraction image took a huge push to get visible differences... However, the compression artifacts aren't an intentional image adjustment like edge enhancement or unsharp masking, so I would expect the difference to be pretty subtle for sure.

Possibly, you are shooting at a low enough ISO that the DR of the camera is capable of covering the DR of the scene acceptably. I'm wondering what would happen to a similar image if you shot it at 3200 with and without compression. Based on the graphs I've seen, the DR drops by 4 stops by 3200, and combining this with high contrast conditions might be a possible source for problems.

Ultimately, I agree, that there are only very subtle (at most) differences in the two images you provided, and the calculated difference image bears that out.


---Michael

 

[GrahamWelland]

I've read that DNG conversion is virtually lossless

thats an oxymoron I'm afraid ... :facesmack:

Do or not do, there is no try (virtual lossless). :ROTFL:

 

[k-hawinkler]

+1