The GetDPI Photography Forum

Great to see you here. Join our insightful photographic forum today and start tapping into a huge wealth of photographic knowledge. Completing our simple registration process will allow you to gain access to exclusive content, add your own topics and posts, share your work and connect with other members through your own private inbox! And don’t forget to say hi!

Some insights in Sony raw compression

ErikKaffehr

Well-known member
Hi,

Lot of stuff written about Sony raw conversion recently. Here are some articles actually describing it:

A similar compression scheme is used on Nikons, here is Emil Martinec opinion about it:

NEF "lossy" compression is clever - Open Photography Forums

This article explains it a bit more:

How Many Bits to Fully Encode My Image | Strolls with my Dog

Jim Kasson has built a model of Sony's raw compression.

This is image the difference of the uncompressed image and the compressed image with no delta coding and dithered by shot noise at 100 ISO. The difference image is pushed 10 stops.


Now, if there was no photon noise present, there would be artefacts, the dithering effect of photon noise makes this compression scheme workable.


But, Sony has also a delta encoding scheme, handling a 16 pixel block of data and using a 7 bit coding of the range of numbers within that block. So, minimum value is saves as 11 bit code and the offset is coded by 7 bits. This compression is lossless unless the gradient is to large. This can cause artefacts:


The images above come from a series by Jim Kasson, here is a link to one of the latest articles in the series:
Removing the delta modulation component from Sony raw compression | The Last Word

Finally, here is an article on RawDigger discussing the issue:
RawDigger: detecting posterization in SONY cRAW/ARW2 files | RawDigger

This is the "star trail image"


And this is the effect of the delta coding, heavily enhanced:


Note the similarity to Jim Kasson's image, with delta compression enabled and photon noise on.

Best regards
ERik
 
Last edited:

GrahamWelland

Subscriber & Workshop Member
Very interesting.

The real question is what would you actually see without the aid of a microscope in a real A4/A3 or even 36in print? :chug:
 

ErikKaffehr

Well-known member
Hi,

I wanted to present the technical aspects. What my be less obvious that the images by Jim Kasson has been extremely pushed. He substracts the compressed image from the uncompressed, resulting in a "black" image than he pushes the image to make the differences observable. This is all well explained in Jim's articles, but it may not be obvious from the excerpts.

Commenting on the issue, I dont have an A7rII, but a Sony A99. I have not observed the things Lloyd sees. On the other hand I have seen some issues with colour management. A digital camera is not a very well defined colorimetric device, Bruce Frazer used to call them colour mixing devices. So converting from a large gamut colour mixing device to a small gamut file has it's complexities.

Best regards
Erik



Very interesting.

The real question is what would you actually see without the aid of a microscope in a real A4/A3 or even 36in print? :chug:
 

tashley

Subscriber Member
Hi,

I wanted to present the technical aspects. What my be less obvious that the images by Jim Kasson has been extremely pushed. He substracts the compressed image from the uncompressed, resulting in a "black" image than he pushes the image to make the differences observable. This is all well explained in Jim's articles, but it may not be obvious from the excerpts.

Commenting on the issue, I dont have an A7rII, but a Sony A99. I have not observed the things Lloyd sees. On the other hand I have seen some issues with colour management. A digital camera is not a very well defined colorimetric device, Bruce Frazer used to call them colour mixing devices. So converting from a large gamut colour mixing device to a small gamut file has it's complexities.

Best regards
Erik
The current orthodoxy has become to characterise the problems with Lloyd's file as being gamut and management issues, which may or may not be your implication here. For the avoidance of doubt I'd like to draw attention to two quotes from Iliah on 'the other forum', both posted since he and Lloyd clarified what colour management process Lloyd used to present his web jpegs from their original raw source (which AFAIK only Lloyd, Iliah and his colleague Alex, and I have seen).

1)

Statement by other contributor: "This is not posterization issue it is a color management issue ..."
Response from Iliah: "It is posterization issue too all right. When you have as few as 46 levels in the 44-138 range in the red channel - and that is over 1.8 million red pixels - it is too few"

2)
Other contributor: I don't have access to the original RAW file, but I bet I could make a good looking JPEG from the original file, assuming I didn't go from RAW to ProPhoto to Adobe or sRGB.

Iliah: Oh yes, with a little painting you can.

-------

The file is posterised. I have seen it myself and played with it in a thousand ways. We can argue about what has caused that posterisation, whether that be data schema, compression, decompression, raw pre-cooking, exposure, tidal forces or the wrath of Gaia but it is as posterised as an Art Deco billboard advert and no amount of rage or disbelief will ever change that.

Just wanted to be clear about that.
 
V

Vivek

Guest
An open question to all: given the pains to analyze a file and compression effects and having been convinced of its ills, do any of you see a business opportunity to hack Sony's FW and come up with a new one that would offer lossless files?

In other words, what is the purpose of this?
 

Amin

Active member
Statement by other contributor: "This is not posterization issue it is a color management issue ..."
Response from Iliah: "It is posterization issue too all right. When you have as few as 46 levels in the 44-138 range in the red channel - and that is over 1.8 million red pixels - it is too few"
You noticed the "too" in Illiah's response, right? From the very beginning of the discussion, Iliah said this was an issue of subject, color management, exposure, and sensor. He added the polarizer as a fifth contributor after he was made aware of it.


...both posted since he and Lloyd clarified what colour management process Lloyd used to present his web jpegs from their original raw source
Here is where I think you're (unintentionally) misleading folks, Tim. Prior to any post Iliah made in that thread, he duplicated Lloyd's findings by processing to Adobe RGB. It's not like Lloyd did something special here. He used what Iliah very recently referred to in the other thread as "default color management" (after another member depicted it as "bad color management"). With that full understanding, which was never altered by any subsequent clarification, Iliah stated:

"No, it is a mix of three things, exposure, raw format, and colour management; with colour management adding the most to the perceived damage."

and

"... converting to Adobe RGB shows exact same issues, while in ProPhoto red channel is much smoother and cleaner"


The current orthodoxy has become to characterize the A7RII as having a "posterization issue". It should be pointed out that every camera with a Bayer sensor has a "posterization issue". This idea that the A7RII will do this under certain very rare circumstances should be understood in context of the fact that a camera like the D810 might do it under even rarer circumstances, but they all can and will do it when the light captured is of a certain quality and quantity. My first Canon 5D had 12-bit RAW and was one of the most popular cameras in the world. Can't ever recall there being a stir made about posterization.


A couple questions have not been answered:

1) Would the greater number of red levels in the red channel be enough to completely avoid posterization under these particular capture conditions if the camera offered uncompressed (or loslessly compressed) raw?

2) Iliah noted other gaps in the raw file, which he thinks could be from lens compensation settings. The EXIF shows that Lloyd had lens compensation on. Would the posterization have been different or avoided without lens compensations enabled?
 
Last edited:

GrahamWelland

Subscriber & Workshop Member
The current orthodoxy has become to characterize the A7RII as having a "posterization issue". It should be pointed out that every camera with a Bayer sensor has a "posterization issue". This idea that the A7RII will do this under certain very rare circumstances should be understood in context of the fact that a camera like the D810 might do it under even rarer circumstances, but they all can and will do it when the light captured is of a certain quality and quantity.
Indeed. Let's face it most (2/3rds I believe) of the colour in a demosaic'd raw is interpolated colour information and not real recorded data. The fact that deBayering produces believable colour is in itself a great feat of computation and we shouldn't be surprised if when pushing boundary conditions that any system will produce artifacts of some kind or another.
 

tn1krr

New member
Very interesting.

The real question is what would you actually see without the aid of a microscope in a real A4/A3 or even 36in print? :chug:
The Star trail compression artefacts (example image available for download in RAWDigger site) are quite visible in my calibrated 27" screen at 50% viewing and modest (and quite necessary) push on the file. The gradient on the background gets broken so the artefacts are quite visible.

The same photographer has posted a few more samples into our local forum that show artefacting on quite small magnifications. For example very very low res pic (1100 pixels on long edge) behind link below shows the artefacts if one zooms to 200% with browser. High quality A4/A3 print would be quaranteed to show the same thing; the photographer who took the pics has printed them on different sizes and confirms this.

http://digikamera.net/keskus/viewtopic.php?t=12734522&start=0&sid=ece7ec3c79ce2f3d70b8f48682ba6a87

Amin said:
A couple questions have not been answered:

1) Would the greater number of red levels in the red channel be enough to completely avoid posterization under these particular capture conditions if the camera offered uncompressed (or loslessly compressed) raw?

2) Iliah noted other gaps in the raw file, which he thinks could be from lens compensation settings. The EXIF shows that Lloyd had lens compensation on. Would the posterization have been different or avoided without lens compensations enabled?
1) We cannot be sure how good a result would a full 14-bit file without any gapping give us since we do not have one. But the logic is quite simple: too few values in the Sony file ==> posterization. Now if we get a bit over 2x values common sense should tell us there is at least less posterization, right?
2) LLoyd shoots with only CA correction on. Also, if one is using Adobe products the CA correction cannot be avoided. I shoot with all corrections "off" in camera. If I import a pic taken with Batis 85/1.8 into Lightroom CC the software automatically extracts a build in CA correction profile from the RAW File and applies it, even if I do not check "Enable profile corrections"

The other gaps (the few nearly completely nonpopulated values inside the Red chunk in histogram) on the file were quite strange; they were actually present in all color channels in exact same spot so if this is caused by any lens correction something is obviously broken with that.

Personally, I do not think A7R II has anything nearly what one would call a real "posterization issue". But this is one example where a full, 14-bit lossless compressed, RAW would a bit tiny bit more robust. Throwing away data during compression/processing just is not a free lunch.

Again, I think these discussions have been very positive: I've learned new things about exposure & color filtering during exposure and color management I can potentially use in the future. My E Mount cameras (currently own 3 and an RX100 MKIII) are no worse than they were before these discoveries.
 

Guy Mancuso

Administrator, Instructor
The polarizer being in that shot is a major contributer for throwing this file into posterization with the extremely heavy blue channel. It seems to be overlooked a great deal. The file would not been in that position to cause trouble before the gamut issues in the first place . Than along with other factors it went south. Just wanted to make that point. I'm going to stay out of this conversation but bottom line this is a anomaly and not a everyday issue. At least with this file. Sure we have other issues at hand that need to be addressed but this is not pinned down to one individual issue.

I'm going to go work on my tests which I will post today
 

Amin

Active member
We cannot be sure how good a result would a full 14-bit file without any gapping give us since we do not have one.
The odd gaps are unexplained. We don't know if those are related to a camera setting or not.


Now if we get a bit over 2x values common sense should tell us there is at least less posterization, right?
Sure, but how much less? Would the file have been usable? As I mentioned, all these cameras can have posterization.


Also, if one is using Adobe products the CA correction cannot be avoided. I shoot with all corrections "off" in camera. If I import a pic taken with Batis 85/1.8 into Lightroom CC the software automatically extracts a build in CA correction profile from the RAW File and applies it, even if I do not check "Enable profile corrections"
The way certain corrections can't be disabled (like distortion correction for Micro 4/3 lenses and the CA correction you mention) is my second least favorite thing about Lightroom.
 
Last edited:

tashley

Subscriber Member
An open question to all: given the pains to analyze a file and compression effects and having been convinced of its ills, do any of you see a business opportunity to hack Sony's FW and come up with a new one that would offer lossless files?

In other words, what is the purpose of this?

That, my friend, is a very fine question!

To give personal answers to your two questions:

1) That would be fun, I'd probably give it a go if someone did it but there might be too many people worried about voiding their warranties to make it commercial.


2) I think there are several points to all this, and they include in no particular order

* The abstract desire to understand the causes of a phenomenon

* The abstract desire to agree or disagree with theories about the causes of a phenomenon (or indeed whether or not the phenomenon is in fact a phenomenon at all) according to personal, emotional, cultural and psychological tendencies

* The desire, once having satisfied oneself as to causes, to see whether any particular shooting technique or processing methods might help prevent one experiencing the phenomenon oneself

* The desire to identify shortcomings, oversights, wool-pullings, or mis-allocation of priorities on the part of a manufacturer so as to create pressure on them to ameliorate the phenomenon via firmware, or in future models (see also: shutter shock)

* The desire to protect or advance the status of any of the parties to the debate

More, anyone?
 
V

Vivek

Guest
Hi Tim,

I have 3 E mount cams at the moment without warranty because they were hardware hacked, knowingly. I use them (A7 UV for UV captures, NEX-C3 for UV or IR, NEX-5NM, monochrome only).

I have no worries on warranties. :)

FW hack, if anyone does it, I am willing to try it at my own risk! :)
 

tashley

Subscriber Member
Hi Tim,

I have 3 E mount cams at the moment without warranty because they were hardware hacked, knowingly. I use them (A7 UV for UV captures, NEX-C3 for UV or IR, NEX-5NM, monochrome only).

I have no worries on warranties. :)

FW hack, if anyone does it, I am willing to try it at my own risk! :)
You, my friend, are a VERY NAUGHTY BOY! :ROTFL::ROTFL::ROTFL:
 

Iliah Borg

New member
My very first post on the topic here:
===
The raw data is posterized OK (partially because the exposure could be 1 stop higher), but also a lot of damage is done in conversion from ProPhoto RGB to output space.

It is not a problem with the new camera, it is the result of several things, not in the least that all the cameras from whatever maker, not just SONY, lack proper exposure feedback, and that straightforward colour management is prone to issues. And of course SONY may want to look into revising their output raw format.
===
http://www.dpreview.com/forums/post/56349196

I maintained from the very beginning that the issue is multi-faceted, and that it starts with the problems introduced in raw. To me, it is obvious that if you just convert from ProPhoto to Adobe RGB and look at the red channel, the problems mount. But that does not mean it is the only problem.
 

tn1krr

New member
The odd gaps are unexplained. We don't know if those are related to a camera setting or not.
I can tell they are not due to Lens compensation. My "14-bit" (that are actually 13-bit) RAW files have them too; exact same locations (14, 66, 120...) all color channels. I was testing the hot pixel stuff the other night and checked those files in Rawdigger. Selected logarithmic Y-Axis for easier readability. Screenshot below, I have all the lens compensations off.



Sure, but how much less? Would the file have been usable? As I mentioned, all these cameras can have posterization.
It would have been slightly more usable. Would it print beautifully wall-sized without excessive PP? Most propably not, but IMO it is sort of irrelevant. Full 14-bit lossless has that tiny bit of extra in some of these very extreme cases and that is why I'd like that from Sony.
 

Iliah Borg

New member
I can tell they are not due to Lens compensation. My "14-bit" (that are actually 13-bit) RAW files have them too; exact same locations (14, 66, 120...) all color channels. I was testing the hot pixel stuff the other night and checked those files in Rawdigger. Selected logarithmic Y-Axis for easier readability. Screenshot below, I have all the lens compensations off.




It would have been slightly more usable. Would it print beautifully wall-sized without excessive PP? Most propably not, but IMO it is sort of irrelevant. Full 14-bit lossless has that tiny bit of extra in some of these very extreme cases and that is why I'd like that from Sony.
It may be as simple as ISO calibration, those regular gaps. However on Lloid's shot there is a void:

This void is 3 data numbers missing, and after white balance application (multiplication by 2.4 for this shot) it will be 7 or 8 data numbers (depends on rounding). That is a huge void. Consider how demosaicking would act over such an artifact.
 

tashley

Subscriber Member
You noticed the "too" in Illiah's response, right? From the very beginning of the discussion, Iliah said this was an issue of subject, color management, exposure, and sensor. He added the polarizer as a fifth contributor after he was made aware of it.
It would be correct to assume that, given the fact that I chose to quote it, I had noticed what it said. To be helpful I will quote the section again:

Statement by other contributor: "This is not posterization issue it is a color management issue ..."
Response from Iliah: "It is posterization issue too all right. When you have as few as 46 levels in the 44-138 range in the red channel - and that is over 1.8 million red pixels - it is too few"


Here is where I think you're (unintentionally) misleading folks, Tim. Prior to any post Iliah made in that thread, he duplicated Lloyd's findings by processing to Adobe RGB. It's not like Lloyd did something special here. He used what Iliah very recently referred to in the other thread as "default color management" (after another member depicted it as "bad color management"). With that full understanding, which was never altered by any subsequent clarification, Iliah stated:

"No, it is a mix of three things, exposure, raw format, and colour management; with colour management adding the most to the perceived damage."
Thank you for that kind understanding, I am flattered not to be accused of intentionally misleading anyone! However I do hope that neither of us is doing so unintentionally either. Regarding Iliah's quote that you have reproduced directly above I think it is in fact true to say that Iliah's subsequent two quotes did in fact offer significant additional drill down detail and clarification. Again: "It is posterization issue too all right. When you have as few as 46 levels in the 44-138 range in the red channel - and that is over 1.8 million red pixels - it is too few"

So I think we can agree that overall and including everything Iliah has so far said publicly that we have both read, he is clear that the RAW file is posterised.

I would also like again to remind you that I have the file, and can play with it in my evil laboratory and look at its EXIF for hours on end whenever I choose.

When that file is opened in Lightroom, it looks horribly posterised.
When that file is opened in C1 it looks horribly posterised.

Let's think about that.

LR uses AdobeRGB for some things but, for display in the Develop Module, it uses ProphotoRGB and, with soft proofing enabled, allows a variety of further colour spaces to be used (simulated in fact)
C1 uses "A very large colour space" (to quote them) which I will assume, since they are not more specific, is at least as large as Adobe RGB and probaly Prophoto but it also offers the possibility of viewing in other spaces.

So before there has been anything non-default done to the file, before any other colour space decisions or actions have been made, the file looks horribly posterised in both programs. Because it is.

Of course different monitors have different gamuts relative to a particular colour space so in effect and by default, anyone who looks at the file on any monitor will have effectively chosen a colour space in which to view it, when they purchased their monitor, which is why I use several, amongst which is an Eizo with 97% of Adobe RGB. From memory, Macbook and Macbook Pro colour spaces cover just under 90% of Adobe RGB. Very few monitors manage to cover 100% and none that I am aware of get anywhere close to Prophoto. So pretty much everyone who sees the file will, without taking any action in C1 or LR to change colour space either to view or to output the file, NOT see the entire range of colours the file contains. In fact, the camera outputs files in an uncalibrated color space unless you shoot JPEG.

In other words, the question of what Lloyd did or did not do on output is entirely moot. Iliah himself has said that the information in the file is posterised and, to clarify, when someone said to him "I don't have access to the original RAW file, but I bet I could make a good looking JPEG from the original file, assuming I didn't go from RAW to ProPhoto to Adobe or sRGB." Iliah replied "Oh yes, with a little painting you can."

I take his response (I have several Russian photographer friends) to be a typically Russian piece of humour, somewhere between gentle sarcasm, gentle mockery and irony, and that it means "Not without resorting to a paintbrush".

Iliah reads this forum, he can certainly correct me if I am wrong about that - but let's not forget that he has already said "It is posterization issue too all right". And yes, I did notice the 'too'.

About that 'too' - what I hope we have established beyond much sensible doubt now is that the file is posterised (and in my view, horribly) BEFORE anything is 'done' to it other than choosing which monitor and reputable RAW converter to open it in. That's where all the stuff about histograms and colour channel comes in, and we can all read that again at our leisure. But the file itself is posterised.


and

"... converting to Adobe RGB shows exact same issues, while in ProPhoto red channel is much smoother and cleaner"
Certainly true, much smoother and cleaner. But not smooth or clean enough, because as the RawDigger histos show and as Iliah has clearly stated, the file is posterised to the extent that starting with it, no one could make a good looking JPEG out of it without a paintbrush.

The current orthodoxy has become to characterize the A7RII as having a "posterization issue". It should be pointed out that every camera with a Bayer sensor has a "posterization issue". This idea that the A7RII will do this under certain very rare circumstances should be understood in context of the fact that a camera like the D810 might do it under even rarer circumstances, but they all can and will do it when the light captured is of a certain quality and quantity. My first Canon 5D had 12-bit RAW and was one of the most popular cameras in the world. Can't ever recall there being a stir made about posterization.
Yes, thanks for helping point out the quantisations involved in de-bayerisation and most importantly for agreeing that there are indeed circumstances under which the A7RII will do it. We can agree that most, even all, cameras could be subject to it. But in the case we are discussing what we are all trying to understand is whether that is MORE likely to happen with the A7RII camera than with one of its peers, such as the MF cameras, the new Canon, or the Nikon D810/A. In order to do that we need to fully understand what is causing it. Focussing on the colour management issues later in the imaging chain might help explain why it gets horrendous rather than the horrible it started out as, so let's focus on what might be causing it and whether it is more likely to happen with this particular camera than with its peers end even, God forbid, whether there are any shooting habits we can adopt that might ameliorate it. Such as, not using a polariser, or exposing as far to the right as we can, or using a Cyan filter (not for me, that one, I had enough of remedial filters with the M8!) but NONE of this has anything to do with the management of output color space.


A couple questions have not been answered:

1) Would the greater number of red levels in the red channel be enough to completely avoid posterization under these particular capture conditions if the camera offered uncompressed (or loslessly compressed) raw?

2) Iliah noted other gaps in the raw file, which he thinks could be from lens compensation settings. The EXIF shows that Lloyd had lens compensation on. Would the posterization have been different or avoided without lens compensations enabled?
I can't answer the first one but I do subscribe to the 'no free lunch' theory on this, and I do find the files less robust in PP than those from a D810.

The second one, to clarify, the EXIF shows distortion correction: None and Vignetting Correction: OFF. However, Chromatic aberration corrections are set to Auto and, not having the 85mm Batis, I don't know whether the camera allow that setting to be set to off. There are, later in the EXIF, parameters for these CA corrections but I don't know if that means that they were necessarily used - parameters are also given for distortion corrections and we know that that was set to OFF.
 

Amin

Active member
My very first post on the topic here:
===
The raw data is posterized OK (partially because the exposure could be 1 stop higher), but also a lot of damage is done in conversion from ProPhoto RGB to output space.

It is not a problem with the new camera, it is the result of several things, not in the least that all the cameras from whatever maker, not just SONY, lack proper exposure feedback, and that straightforward colour management is prone to issues. And of course SONY may want to look into revising their output raw format.
===
Re: A7RII posterization?: Sony Alpha Full Frame E-mount Talk Forum: Digital Photography Review

I maintained from the very beginning that the issue is multi-faceted, and that it starts with the problems introduced in raw. To me, it is obvious that if you just convert from ProPhoto to Adobe RGB and look at the red channel, the problems mount. But that does not mean it is the only problem.

Thank you for clarifying that your initial points were not based on some sort of misunderstanding of Lloyd's color management that was subsequently understood differently.

With that, I take my leave of this discussion, which from what I can tell has devolved into recurrent argumentation.
 

ErikKaffehr

Well-known member
Hi,

"Conventional wisdom" says that shot noise will dither the signal so no posterisation will arise. So, here Diglloyd's interpretation contradicts "conventional wisdom" which happen to be know as signal processing theory. The original intent of this thread was to point to a few articles that analyse/describe the algorithms involved.

Now, I understand that Iliah Borg happens to believe that shot noise (that is a fact of nature) and readout noise is not sufficient to dither out posterisation with the sampling used by raw compression. For that part Iliah may be right, I am not an expert on signal processing theory in contrary to those folks I referred to in the original posting.

I have something like four/five issues with Lloyd's article:

1) He discusses the tonal compression and ignores the delta compression. The delta compression has demonstrable artefacts, at least in some cases, namely the star tracks artefacts.

2) He systematically ignores mentioning any articles/posting contradicting his findings. That is bad science.

3) He does not publish full information, including raw files. For more information you need to go to his pay site. I am subscribing, but I still find this bad science.

4) As we have seen Iliah suggest that colour space mapping is the major contributor to the issue. Yes he says that the raw file contributes.

5) It seems that someone over at Luminous Landscape asked Joseph Holmes, who has cooperated with Lloyd discussing the A7r vibration issues, and he plainly said that Lloyd is wrong.

Now, the A7r shutter vibration is an interesting thing. Lloyd expected Sony to release a firmware upgrade with delaying the shutter release. Good suggestion, except that Lloyd ignored the facts. A real engineer has actually measured the vibrations trough the exposure cycle and found that the release of the first shutter curtain was the dominant factor. So, Sony could not fix anything with a firmware release. They needed to redesign the shutter and add electronic first shutter curtain to the sensor. They did it both on the A7rII. The reason I did not buy into the A7r at that time was that I felt the sensor was an old design, with no EFCS (Electronic First Cutter Shutter Curtain) and no on sensor phase detection, both features I had on my Sony Alpha 99 SLT.

Sony added that stuff on the A7rII, but I am still waiting on delivery. When I have that camera I will publish both raw files and findings.

Best regards
Erik




It would be correct to assume that, given the fact that I chose to quote it, I had noticed what it said. To be helpful I will quote the section again:

Statement by other contributor: "This is not posterization issue it is a color management issue ..."
Response from Iliah: "It is posterization issue too all right. When you have as few as 46 levels in the 44-138 range in the red channel - and that is over 1.8 million red pixels - it is too few"




Thank you for that kind understanding, I am flattered not to be accused of intentionally misleading anyone! However I do hope that neither of us is doing so unintentionally either. Regarding Iliah's quote that you have reproduced directly above I think it is in fact true to say that Iliah's subsequent two quotes did in fact offer significant additional drill down detail and clarification. Again: "It is posterization issue too all right. When you have as few as 46 levels in the 44-138 range in the red channel - and that is over 1.8 million red pixels - it is too few"

So I think we can agree that overall and including everything Iliah has so far said publicly that we have both read, he is clear that the RAW file is posterised.

I would also like again to remind you that I have the file, and can play with it in my evil laboratory and look at its EXIF for hours on end whenever I choose.

When that file is opened in Lightroom, it looks horribly posterised.
When that file is opened in C1 it looks horribly posterised.

Let's think about that.

LR uses AdobeRGB for some things but, for display in the Develop Module, it uses ProphotoRGB and, with soft proofing enabled, allows a variety of further colour spaces to be used (simulated in fact)
C1 uses "A very large colour space" (to quote them) which I will assume, since they are not more specific, is at least as large as Adobe RGB and probaly Prophoto but it also offers the possibility of viewing in other spaces.

So before there has been anything non-default done to the file, before any other colour space decisions or actions have been made, the file looks horribly posterised in both programs. Because it is.

Of course different monitors have different gamuts relative to a particular colour space so in effect and by default, anyone who looks at the file on any monitor will have effectively chosen a colour space in which to view it, when they purchased their monitor, which is why I use several, amongst which is an Eizo with 97% of Adobe RGB. From memory, Macbook and Macbook Pro colour spaces cover just under 90% of Adobe RGB. Very few monitors manage to cover 100% and none that I am aware of get anywhere close to Prophoto. So pretty much everyone who sees the file will, without taking any action in C1 or LR to change colour space either to view or to output the file, NOT see the entire range of colours the file contains. In fact, the camera outputs files in an uncalibrated color space unless you shoot JPEG.

In other words, the question of what Lloyd did or did not do on output is entirely moot. Iliah himself has said that the information in the file is posterised and, to clarify, when someone said to him "I don't have access to the original RAW file, but I bet I could make a good looking JPEG from the original file, assuming I didn't go from RAW to ProPhoto to Adobe or sRGB." Iliah replied "Oh yes, with a little painting you can."

I take his response (I have several Russian photographer friends) to be a typically Russian piece of humour, somewhere between gentle sarcasm, gentle mockery and irony, and that it means "Not without resorting to a paintbrush".

Iliah reads this forum, he can certainly correct me if I am wrong about that - but let's not forget that he has already said "It is posterization issue too all right". And yes, I did notice the 'too'.

About that 'too' - what I hope we have established beyond much sensible doubt now is that the file is posterised (and in my view, horribly) BEFORE anything is 'done' to it other than choosing which monitor and reputable RAW converter to open it in. That's where all the stuff about histograms and colour channel comes in, and we can all read that again at our leisure. But the file itself is posterised.




Certainly true, much smoother and cleaner. But not smooth or clean enough, because as the RawDigger histos show and as Iliah has clearly stated, the file is posterised to the extent that starting with it, no one could make a good looking JPEG out of it without a paintbrush.



Yes, thanks for helping point out the quantisations involved in de-bayerisation and most importantly for agreeing that there are indeed circumstances under which the A7RII will do it. We can agree that most, even all, cameras could be subject to it. But in the case we are discussing what we are all trying to understand is whether that is MORE likely to happen with the A7RII camera than with one of its peers, such as the MF cameras, the new Canon, or the Nikon D810/A. In order to do that we need to fully understand what is causing it. Focussing on the colour management issues later in the imaging chain might help explain why it gets horrendous rather than the horrible it started out as, so let's focus on what might be causing it and whether it is more likely to happen with this particular camera than with its peers end even, God forbid, whether there are any shooting habits we can adopt that might ameliorate it. Such as, not using a polariser, or exposing as far to the right as we can, or using a Cyan filter (not for me, that one, I had enough of remedial filters with the M8!) but NONE of this has anything to do with the management of output color space.




I can't answer the first one but I do subscribe to the 'no free lunch' theory on this, and I do find the files less robust in PP than those from a D810.

The second one, to clarify, the EXIF shows distortion correction: None and Vignetting Correction: OFF. However, Chromatic aberration corrections are set to Auto and, not having the 85mm Batis, I don't know whether the camera allow that setting to be set to off. There are, later in the EXIF, parameters for these CA corrections but I don't know if that means that they were necessarily used - parameters are also given for distortion corrections and we know that that was set to OFF.
 

tashley

Subscriber Member
Hi,

"Conventional wisdom" says that shot noise will dither the signal so no posterisation will arise. So, here Diglloyd's interpretation contradicts "conventional wisdom" which happen to be know as signal processing theory. The original intent of this thread was to point to a few articles that analyse/describe the algorithms involved.

Now, I understand that Iliah Borg happens to believe that shot noise (that is a fact of nature) and readout noise is not sufficient to dither out posterisation with the sampling used by raw compression. For that part Iliah may be right, I am not an expert on signal processing theory in contrary to those folks I referred to in the original posting.

I have something like four/five issues with Lloyd's article:

1) He discusses the tonal compression and ignores the delta compression. The delta compression has demonstrable artefacts, at least in some cases, namely the star tracks artefacts.

2) He systematically ignores mentioning any articles/posting contradicting his findings. That is bad science.

3) He does not publish full information, including raw files. For more information you need to go to his pay site. I am subscribing, but I still find this bad science.

4) As we have seen Iliah suggest that colour space mapping is the major contributor to the issue. Yes he says that the raw file contributes.

5) It seems that someone over at Luminous Landscape asked Joseph Holmes, who has cooperated with Lloyd discussing the A7r vibration issues, and he plainly said that Lloyd is wrong.

Now, the A7r shutter vibration is an interesting thing. Lloyd expected Sony to release a firmware upgrade with delaying the shutter release. Good suggestion, except that Lloyd ignored the facts. A real engineer has actually measured the vibrations trough the exposure cycle and found that the release of the first shutter curtain was the dominant factor. So, Sony could not fix anything with a firmware release. They needed to redesign the shutter and add electronic first shutter curtain to the sensor. They did it both on the A7rII. The reason I did not buy into the A7r at that time was that I felt the sensor was an old design, with no EFCS (Electronic First Cutter Shutter Curtain) and no on sensor phase detection, both features I had on my Sony Alpha 99 SLT.

Sony added that stuff on the A7rII, but I am still waiting on delivery. When I have that camera I will publish both raw files and findings.

Best regards
Erik

Erik, whilst I respect your opinions may I yet again draw attention to the very carefully written post that you quote but do not address in which I make it really very, extremely clear, that the raw file itself, to which I have access, as does Iliah, his colleague, and of course Lloyd himself, is posterised. Iliah says so, so does everyone who has seen it. Yes it's made worse by colour space mapping but Iliah is very very clear that a good JPEG could not be made from that file, whatever form of colour space mapping or other PP juju, was performed. In other words the posterisation is beyond redemption. It's not the end of the world, the sky is not falling in, the earth keeps turning and the camera will make millions of exceptionally fine images and I am certain that you will enjoy it very much. But that particular image was posterised, before Lloyd even opened the file, beyond any hope of redemption.

As for 'good science' and sharing the file, he has shared it with two of the best brains in the business. Scientists share with relevant peers first, not necessarily with a whole lot of people they don't know the credentials of. In any event, Lloyd, with whom I have no affiliation whatsoever other than as a subscriber, is not a scientist, he's a reviewer and he will live and die by the quality of his reviews.

So in the final analysis it really doesn't matter, at all, that colour space mapping makes it worse. A dead horse is dead.

:deadhorse::deadhorse::deadhorse:
 
Top