Compressed VS Uncompressed DNGs - A quick test

[jaknight]

Thank you for this interesting discussion.
I have always chosen not to compress files: partly because I have been concerned that there may be some loss of information from the files; and partly because I have been concerned that as time goes on, and software evolves, it may become harder to decompress/read those files. I am not a software wizard and could be quite wrong, but it just seems a bit safer.

 

[Bob]

Compression etc:
There are all sorts of compression schemes, some are lossy, some are not.
zip compression, for example, is loss-less. Imagine what would happen if random bits suddenly changed in the software you just downloaded.
jpeg compression, is usually lossy, although there is a loss-less variant in the jpeg2000 standard.

In general, lossy compression achieves smaller files however what we want as photographers is to have "perfect" images and maximum storage card capacity.

Sometimes selection of one or the other is a balance between card capacity and write-time. Depending on the relative write speed of the card/camera with the time it takes to compress the image, the rate at which images are stored are sometimes slowed by compression, OTOH, for most current cameras it may actually be improved.

My test for lossless compression is to always compare what would the bits be if compressed vs un-compressed. All true lossless compression schemes will yield the exact same result, although the path to that result may be different.

I once tested Nikon's lossless compression and found that to be true..
I have no reservation about using lossless compression other than the usual unpredictable result on the display of remaining card capacity. With lossless compressed files, the number of images remaining is usually understated.
-bob

 

[MaxKißler]

Just wanted to let you know: The M9 (and I guess the same applies to other M cameras) uses indeed lossless compression. My local Leica dealer confirmed this today. The option to shoot uncompressed is to maximize compatibility with several raw developers that cannot open compressed raw files. So if you're using software that can open compressed raws you don't need to worry.

Cons:
Camera takes slightly longer to write to card.
Is it future-proof?

Pros:
Shorter import times / Backing up files on an external HD is also faster.
Pretty obvious, almost twice as many images go on one sd card.

 

[MirekE]

Though I admit, it doesn't make much of a difference to using Adobe RGB. Before printing, the files get translated to the according color spaces anyway which happen to be a lot smaller than any of the former ones.

Inkjet printers like the popular Epson 3880 definitely have gamut with colors that extend beyond AdobeRGB. In the diagram below, the color polygon is AdobeRGB and the white is a printer profile for luster paper for 3880. There are printers with even larger gamut so it makes sense to use color spaces larger than AdobeRGB.

 

[MirekE]

Max, if you use an unnecessarily large gamut like ProPhotoRGB, you can simply be throwing away data. I never simply default to that unless it is clear AdobeRGB is not large enough. I have yet to find AdobeRGB too small for what I shoot. Also, if you are letting your CM system simply rescale one gamut into another, then you are also transforming the image. Another reason I like AbobeRGB as it is closer to the print space.

Is the AdobeRGB really big enough for you? The following diagrams show pixels from a picture of an ordinary tree with green leaves plotted into the CIE chromaticity "horseshoe". The three triangles are gamuts of sRGB, AdobeRGB and ProPhoto RGB. One diagram is exported as ProPhoto and one as AdobeRGB. As you can see, lots of the greens had to be remapped from its original color to fit inside AdobeRGB. So using AdobeRGB meant in this particular case throwing away data - something that you apparently like to avoid...

 

[MirekE]

Just wanted to let you know: The M9 (and I guess the same applies to other M cameras) uses indeed lossless compression. My local Leica dealer confirmed this today.

Your shots look like there is no difference. But document on a Leica site says "choice of uncompressed or slightly compressed (by non-linear reduction of color depth)" - go to Leica Camera AG - Photography - M9 & M9-P and click on the Technical Data link. Non linear reduction of color depth means lossy.

 

[MaxKißler]

Your shots look like there is no difference. But document on a Leica site says "choice of uncompressed or slightly compressed (by non-linear reduction of color depth)" - go to Leica Camera AG - Photography - M9 & M9-P and click on the Technical Data link. Non linear reduction of color depth means lossy.

Too bad, I guess there is no free lunch today. Thanks for the info Mirek. So back to uncompressed it is.

 

[MaxKißler]

Is the AdobeRGB really big enough for you? The following diagrams show pixels from a picture of an ordinary tree with green leaves plotted into the CIE chromaticity "horseshoe". The three triangles are gamuts of sRGB, AdobeRGB and ProPhoto RGB. One diagram is exported as ProPhoto and one as AdobeRGB. As you can see, lots of the greens had to be remapped from its original color to fit inside AdobeRGB. So using AdobeRGB meant in this particular case throwing away data - something that you apparently like to avoid...

^^This looks like a translation from ProphotoRGB to AdobeRGB with a relatively colorimetric rendering intend. Using a preceptive rendering intend would have slightly moved all the colors. Though I'm no expert on this topic and I admit I might very well be wrong...
You have to admit that this is all a bit theoretical. According to the images it looks like it has a larger impact on a print than it usually does. "Usually" because I personally am a fan of muted colors and I rarely print highly saturated images. The images from the test are just my attempt to cause banding but I never increase saturation or vibrance.

But you are right, I thrive for ultimate quality so I'll continue using uncompressed raws and my color management with ProphotoRGB.


Regards

 

[thedigitalbean]

Your shots look like there is no difference. But document on a Leica site says "choice of uncompressed or slightly compressed (by non-linear reduction of color depth)" - go to Leica Camera AG - Photography - M9 & M9-P and click on the Technical Data link. Non linear reduction of color depth means lossy.

Yep, here's an excerpt from the M page on Facebook, posted by Zalman Stern (a friend and colleague):

Zalman Stern (I am the former tech lead on Camera Raw at Adobe.)

The M Type 240 implements DNG Lossless Compression, which is in fact lossless. (The method used is Lossless JPEG as in many vendor proprietary raw formats.) I do not know why Leica left in the option for Uncompressed DNG. Perhaps they use it in debugging or there is a customer built workflow tool that required uncompressed DNG. I would be in favor of the option being removed, especially if DNG Proxy format support is added in the future. (DNG Proxy allows for lossy compressed and subsampled raw files, similar to e.g. mRAW and sRAW on Canon cameras, though significantly better in quality for a given file size.)

Compressed DNG on the M8, M9, and M-E series of cameras is lossy, though it is hard to make a case that it is visually detectable. The method used transforms the linear raw data via a perceptual curve and quantizes to 8-bit. Hence the constant output size and minimal visual degradation.

One can take Uncompressed DNGs from any M and reconvert them with the Adobe DNG Converter to get approximately a factor of 2 smaller file with no loss of visual information or metadata. Even if one wants to preserve exactly the bits from the camera in one's permanent archive, this is extremely useful for keeping more files on a laptop or for extra backup copies.

Slightly related to the discussion, if one has lens detection turned on for cameras before the Type 240, vignette correction is done destructively to the raw data. I do not know of a way to have lens metadata written to the file without vignette correction being applied. Ideally this would be done by writing DNG lens correction metadata (opcodes) to the DNG file instead of via modifying the raw samples themselves. Perhaps it is done this way in the M Type 240. I have not had an opportunity to investigate.

 

[thedigitalbean]

By the way I should add that what Zalman suggests is what I used to do with the M9. Shoot with it uncompressed on camera and then convert to DNG on a computer getting the lossless compression.

 

[Shashin]

Is the AdobeRGB really big enough for you? The following diagrams show pixels from a picture of an ordinary tree with green leaves plotted into the CIE chromaticity "horseshoe". The three triangles are gamuts of sRGB, AdobeRGB and ProPhoto RGB. One diagram is exported as ProPhoto and one as AdobeRGB. As you can see, lots of the greens had to be remapped from its original color to fit inside AdobeRGB. So using AdobeRGB meant in this particular case throwing away data - something that you apparently like to avoid...

It depends on the rendering.

You also still missed my point of the coordinate space. You get no more possible colors in a color space. You see the image does not fill the colors spaces. The larger the color space, the greater the distance between the colors. A larger colors space will have to do more binning (and as you illustrate, ProPhotoRGB will always have empty coordinates as simply it has "colors" that cannot be perceived). And ultimately you will need to fit your image to a print space. So even with the image you are using, I could still end up with less data using ProPhotoRGB.

Now, if you read my post, what I said was not to automatically default to ProPhotoRGB. I never said not to use it.

 

[Shashin]

^^This looks like a translation from ProphotoRGB to AdobeRGB with a relatively colorimetric rendering intend. Using a preceptive rendering intend would have slightly moved all the colors. Though I'm no expert on this topic and I admit I might very well be wrong...
You have to admit that this is all a bit theoretical. According to the images it looks like it has a larger impact on a print than it usually does. "Usually" because I personally am a fan of muted colors and I rarely print highly saturated images. The images from the test are just my attempt to cause banding but I never increase saturation or vibrance.

But you are right, I thrive for ultimate quality so I'll continue using uncompressed raws and my color management with ProphotoRGB.


Regards

And how you determine "quality"? By looking at the results? In this post you claim you cannot see the difference between compressed and uncompressed files. And this is an important point. A photograph is made for a viewer. The numbers and specs do not indicate if an image is pleasing--"pleasing" is a technical term meaning that it looks good to a viewer. And you should be judging a photograph by its appearance--seeing is believing in photography (if it looks bad, it is). Since a photograph is a visual stimuli, how does the technical specifications converge and diverge from that experience? Photography, while it tries to imitate human vision, it does it in an artificial way. Photographers like to argue over the numbers, but the results are what are important. Which is probably why we argue over them so much--MFD is king, the D800 is the MFD killer, m4/3 is better than FF, etc. So, were does "quality" come from?

Now, if you do like more of a muted palette, ProPhotoRGB will use fewer coordinates. A smaller gamut will use more. The greatest thing about RAW is that you can open a file into any gamut you want and as many as you want. Perhaps try opening a file in both ProPhotoRGB and AdobeRGB and process both and see how they are. If you use fairly complex curves to process, you may find one color space better than the other.

I guess for me, the color space is just another choice on the way to the final image. Just like exposure, lighting, and a whole host of other factors.

 

[Bob]

Shooting raw, colorspace is just a metadata tag.
That is why I shoot raw and nothing else (well, occasionally I will shoot both simultaneously).
Processing raw into a working space that is as large as possible then matching colorspace post-edit to the output device.
It is usually best to minimize the number of colorspace conversions. Conversion from large to small and back again is a sure way of losing information.
-bob

 

[MirekE]

^^This looks like a translation from ProphotoRGB to AdobeRGB with a relatively colorimetric rendering intend. Using a preceptive rendering intend would have slightly moved all the colors. Though I'm no expert on this topic and I admit I might very well be wrong...
You have to admit that this is all a bit theoretical. According to the images it looks like it has a larger impact on a print than it usually does. "Usually" because I personally am a fan of muted colors and I rarely print highly saturated images. The images from the test are just my attempt to cause banding but I never increase saturation or vibrance.

But you are right, I thrive for ultimate quality so I'll continue using uncompressed raws and my color management with ProphotoRGB.


Regards

These diagrams are reaction to a post suggesting that using ProPhoto RGB can lead to data loss. I wanted to demonstrate that actually using AdobeRGB leads to data loss. There are clearly colors outside of AdobeRGB in a trivial shot of a tree that must be remapped.

With regards to your suggestion to use perceptual intent, LR does not allow that and I was told in a color management forum recently that there is no such thing as perceptual intent between these color spaces, as they do not have LUTs. In any case, what you see here is not a conversion from ProPhoto to AdobeRGB, these are both direct conversions from raw.

Whether this is only theoretical or not - this is the picture in ProPhoto, if you want to soft proof it on your own:

I use Lightroom, so yes, it is theoretical for me, because I don't store images in their cooked form anymore. I just think it is good to know about the limitations. AdobeRGB can clip greens, on the other hand, some photographers claim that ProPhoto caused posterization in skin tones in their studio shots. Pick your own poison...

 

[MirekE]

Too bad, I guess there is no free lunch today. Thanks for the info Mirek. So back to uncompressed it is.

Some people said they had visible loss of detail in busy areas of their images, so you may want to do some test on subjects that have lots of color detail. I used uncompressed with M9, because it is safer and does not cost anything.

 

[MirekE]

You also still missed my point of the coordinate space. You get no more possible colors in a color space. You see the image does not fill the colors spaces. .

No, I did not. I did not comment on it because I don't have the knowledge to comment. I have never seen posterization with 16bit ProPhoto. Some people claim it exists though and I am not going to say it does not exist.

 

[Duane Pandorf]

So what I've learned in this thread, and if I'm wrong please tell me, but I've been shooting in uncompressed DNG on my M9. So I just experimented with Lightroom's "Convert to DNG" on import with a couple of my M9 files. In the past I've just used the "Copy" to new location option. Well I after import using the Convert to DNG option I'm saving quite a bit of space. 36mb files down to 18 - 21mb files.

So this will be my new way to import as long as I understand that by converting to DNG a second time I've created a lossless DNG that has not lost any of the original "RAW" data the camera created.

 

[MaxKißler]

And how you determine "quality"? By looking at the results? In this post you claim you cannot see the difference between compressed and uncompressed files. And this is an important point. ...

Now, if you do like more of a muted palette, ProPhotoRGB will use fewer coordinates. A smaller gamut will use more. The greatest thing about RAW is that you can open a file into any gamut you want and as many as you want. Perhaps try opening a file in both ProPhotoRGB and AdobeRGB and process both and see how they are. If you use fairly complex curves to process, you may find one color space better than the other.

...

An idea comes first, this should go without saying (content beats technique - always). My definition of quality was in regard to the technical aspects of a photograph. Having a vast color palette and getting the maximum achievable resolution is of importance to me.

I know I said that I do not see any difference whether I shoot compressed or uncompressed DNGs. And admittedly, I never process a file as intensely as in the test above so that I should never see a difference. But knowing that I'm basically throwing away color information in a compressed raw is something I cannot deal with. It's the fact that there is the eventuality that a certain image could be "better" that is very unapppealing to me.


"ProPhotoRGB will use fewer coordinates. A smaller gamut will use more." This should not affect anything. ProphotoRGB is a larger color space but that doesn't change the location of particular tones in the entire color space.

 

[Bob]

I just think it is good to know about the limitations. AdobeRGB can clip greens, on the other hand, some photographers claim that ProPhoto caused posterization in skin tones in their studio shots. Pick your own poison...

I have never seen a hint of that assuming a good workflow. Converting a jpeg shot in sRGB to pro photo, working on it for awhile, then moving back to sRBG, yes that would tend to cause problems.
-bob

 

[MaxKißler]

These diagrams are reaction to a post suggesting that using ProPhoto RGB can lead to data loss. I wanted to demonstrate that actually using AdobeRGB leads to data loss. There are clearly colors outside of AdobeRGB in a trivial shot of a tree that must be remapped.

With regards to your suggestion to use perceptual intent, LR does not allow that and I was told in a color management forum recently that there is no such thing as perceptual intent between these color spaces, as they do not have LUTs. In any case, what you see here is not a conversion from ProPhoto to AdobeRGB, these are both direct conversions from raw.

Whether this is only theoretical or not - this is the picture in ProPhoto, if you want to soft proof it on your own:

I use Lightroom, so yes, it is theoretical for me, because I don't store images in their cooked form anymore. I just think it is good to know about the limitations. AdobeRGB can clip greens, on the other hand, some photographers claim that ProPhoto caused posterization in skin tones in their studio shots. Pick your own poison...

You're right, there is a visible difference in the greens. Especially in the green of the pine trees. I said it was theoretical because in a print it will hardly make a difference as most printer's color spaces are much smaller anyway. So both ProphotoRGB and AdobeRGB would get scaled down to match that particular printer's color space.

If these are both conversions from raw data it shouldn't make a difference which color space is assigned to the image. As long as there have not been made any permanent conversions to a certain color space it will only make a difference in the way it's being displayed. That's why LR doesn't have internal rendering specifications/preferences.

----


My rule of thumb is: Use the largest possible color space and use the right rendering intent when scaling down to the output color space. That's why it's important to me to know whether shooting compressed raw yields the same quality as uncompressed raw. If the (color) information isn't there in the first placce, using a large color space such as ProphotoRGB won't make much sense.