Some insights in Sony raw compression

[Vivek]

Tim, i would like to know the filters used. i have all sorts of polarizers (among a cubic meter box of various filters), including IR and UV polarizers.

 

[jrp]

Perhaps I am being naive, but is it not the case that water reflections have only a limited palette to start with, the more so if they are polarised?

 

[tashley]

Lloyd's article says that a Zeiss polariser was used, though not necessarily at maximum. It might well have been a significant factor in causing the problem.

 

[Vivek]

Thanks. I still have a D300 somewhere and i am positive that i can produce 14bit NEF files that would show similar effects. If i locate the Nikon gear, i will shoot some and make the NEFs available.

 

[tashley]

Thanks. I still have a D300 somewhere and i am positive that i can produce 14bit NEF files that would show similar effects. If i locate the Nikon gear, i will shoot some and make the NEFs available.

I believe Amin is trying something similar. I'm off to the Adriatic next week, blue seas a plenty, and will (nervously) pack a polariser too...

 

[mjm6]

Perhaps I am being naive, but is it not the case that water reflections have only a limited palette to start with, the more so if they are polarised?

Well, water does reflect what we perceive as a 'white' reflection, so it will reflect through a variety of wavelengths. It's not a perfectly uniform reflector, but not too bad, actually.

However, the polarizer does have two possible impacts. First, some of them tend to have a slight green hue to them, which is effectively taking away magenta (red channel primarily) information.

Second, if the water has a peculiar color that is outside the normal gamut, as I believe this image shows, there will be relatively little information in the file other than the RGB values that are 'pegged' at the place nearest to where the gamut of the camera is capable of representing the color. When the polarizer is employed, it will eliminate reflections that are actually beneficial to adding some color variability to the file be reducing the purity of the color and bringing some of it back into gamut, where the file will then be able to record differences.

This is all made much worse by the conversions to other color gamuts (AdobeRGB, and then presumably sRGB for the JPG), as they are smaller gamut spaces and it's the stuff at the fringes that all get clumped together in most cases, unless you use perceptual rendering when doing the color gamut conversions. Even then, who knows what's under the hood?

As I said in the other post, this is not new stuff, and someone else pointed out that this can occur with some flowers as well.


---Michael

 

[ErikKaffehr]

Hi Tim,

I also respect your opinion.

Best regards
Erik

Erik, whilst I respect your opinions may I yet again draw attention to the very carefully written post that you quote but do not address in which I make it really very, extremely clear, that the raw file itself, to which I have access, as does Iliah, his colleague, and of course Lloyd himself, is posterised. Iliah says so, so does everyone who has seen it. Yes it's made worse by colour space mapping but Iliah is very very clear that a good JPEG could not be made from that file, whatever form of colour space mapping or other PP juju, was performed. In other words the posterisation is beyond redemption. It's not the end of the world, the sky is not falling in, the earth keeps turning and the camera will make millions of exceptionally fine images and I am certain that you will enjoy it very much. But that particular image was posterised, before Lloyd even opened the file, beyond any hope of redemption.

As for 'good science' and sharing the file, he has shared it with two of the best brains in the business. Scientists share with relevant peers first, not necessarily with a whole lot of people they don't know the credentials of. In any event, Lloyd, with whom I have no affiliation whatsoever other than as a subscriber, is not a scientist, he's a reviewer and he will live and die by the quality of his reviews.

So in the final analysis it really doesn't matter, at all, that colour space mapping makes it worse. A dead horse is dead.

:deadhorse::deadhorse::deadhorse:

 

[tashley]

;-)

Hi Tim,

I also respect your opinion.

Best regards
Erik

 

[timparkin]

Hi Tim,

I also respect your opinion.

Best regards
Erik

Whilst I respect everyones opinion (at least those whose opinions I respect anyway) I do wish to offer an alternative to the 'posterised' diagnosis.

1) The areas in the water are all quite out of focus.

2) The Lens has some pretty awful bokeh (see foreground where double lines occur etc). Checking other images these double lines are also evident in the background. Here's a link to what nisen bokeh can do to transitions (http://bit.ly/1KGPhyB)

3) Water itself has incredibly 'harsh' colour transitions, particularly in the conditions shown (see below for further discussion)

Because of this, I'm not sure if it's actually possible to distinguish posterisation from real world out of focus colour without resorting to numerical analysis. Combing of the histogram itself doesn't imply visible posterisation and so the numerical analysis needs to be on the 2D image surface.

I've been trying to think how to filter out these issues and the only things I can think of are

1) remove the red channel and see if we still see the same colour boundaries. If so it's not posterisation

2) add a small amount of noise and see if it disappears (not so sure on this one).

3) 'slice' the red channel at certain brightness levels and check for large, connected areas of the same tone.

As for assessing posterisation, just like Erik has posted elsewhere, raw converters and noise reduction can have massive influences. Here's a shot from this weekend

Only iridient seems to provide 'unposterised' colour. Not the

Have our 'testers' used iridient?

p.s Here's a shot from Lloyd's trip where it looks like there is obvious posterisation but obviously not as it wasn't mentioned.

The large patches of colour are 'wind calm', isolated areas where wind flow is laminar over the water surface - when this laminar flow starts to break down, waves are formed which either reflect from the sky (intense, dark blue) or are transparent and show the colour of the water (intense green). The transitions between these two may be smaller than a pixel at this distance but once you add lens blur the transitions can look very different, from very smooth (adopised aperture) to harsh transitions (nissen bokeh) with potentially very little luminosity variation.

Tim

 

[Iliah Borg]

Dear Tim,
You can use RPP (current version is http://www.raw-photo-processor.com/RPP/RPP64_1748Beta.zip), switch off the colour management (output to raw RGB tiff, the choice is to the left of the Save button), set it to gamma 2.2 curve Curve type, and use the low acutance AHDMF demosaicing. If the result is visibly posterized, it is from raw and from nothing else.

Edit: that is, visibly posterized in per-channel view, to eliminate monitor issues and colour management completely.

 

[timparkin]

Dear Tim,
You can use RPP (current version is http://www.raw-photo-processor.com/RPP/RPP64_1748Beta.zip), switch off the colour management (output to raw RGB tiff, the choice is to the left of the Save button), set it to gamma 2.2 curve Curve type, and use the low acutance AHDMF demosaicing. If the result is visibly posterized, it is from raw and from nothing else.

Edit: that is, visibly posterized in per-channel view, to eliminate monitor issues and colour management completely.

Thanks Iliah - Doing my own tests with a blue filter at the moment..

What does the file look like if you ditch the red channel?

Tim

 

[ErikKaffehr]

Hi,

The star tracks artefacts are caused by the "delta compression", no questions about that. It has nothing to do with combing, gaps, tonal curve etc. You take out the delta compression and those artefacts go away.

Yes, Sony should remove the delta compression.

Best regards
Erik

The Star trail compression artefacts (example image available for download in RAWDigger site) are quite visible in my calibrated 27" screen at 50% viewing and modest (and quite necessary) push on the file. The gradient on the background gets broken so the artefacts are quite visible.

The same photographer has posted a few more samples into our local forum that show artefacting on quite small magnifications. For example very very low res pic (1100 pixels on long edge) behind link below shows the artefacts if one zooms to 200% with browser. High quality A4/A3 print would be quaranteed to show the same thing; the photographer who took the pics has printed them on different sizes and confirms this.

http://digikamera.net/keskus/viewtopic.php?t=12734522&start=0&sid=ece7ec3c79ce2f3d70b8f48682ba6a87


1) We cannot be sure how good a result would a full 14-bit file without any gapping give us since we do not have one. But the logic is quite simple: too few values in the Sony file ==> posterization. Now if we get a bit over 2x values common sense should tell us there is at least less posterization, right?
2) LLoyd shoots with only CA correction on. Also, if one is using Adobe products the CA correction cannot be avoided. I shoot with all corrections "off" in camera. If I import a pic taken with Batis 85/1.8 into Lightroom CC the software automatically extracts a build in CA correction profile from the RAW File and applies it, even if I do not check "Enable profile corrections"

The other gaps (the few nearly completely nonpopulated values inside the Red chunk in histogram) on the file were quite strange; they were actually present in all color channels in exact same spot so if this is caused by any lens correction something is obviously broken with that.

Personally, I do not think A7R II has anything nearly what one would call a real "posterization issue". But this is one example where a full, 14-bit lossless compressed, RAW would a bit tiny bit more robust. Throwing away data during compression/processing just is not a free lunch.

Again, I think these discussions have been very positive: I've learned new things about exposure & color filtering during exposure and color management I can potentially use in the future. My E Mount cameras (currently own 3 and an RX100 MKIII) are no worse than they were before these discoveries.

 

[tn1krr]

Hi,

The star tracks artefacts are caused by the "delta compression", no questions about that. It has nothing to do with combing, gaps, tonal curve etc. You take out the delta compression and those artefacts go away.

Yes, Sony should remove the delta compression.

The star trail example was there just to answer the question "will this show up in a reasonable sized print?". It is often falsely claimed (in that other forum where shooting the messenger is the number one pastime ) that you need huge print or 200% magnification to see the artefacts when they can in fact show up in a small web size photo. The good thing is that outside star trails and few few very specific cases (I have a few high-DR night photos, well lit bridge on near-black background that are somewhat PP-limited by the artefacts) it is very very hard for anything to surface. But still, lossless compression (and non-filtered 13/14-bit bulb) please