Testing Protocols for the D800?

[glenerrolrd]

With literally thousands of posts regarding the D800 and how it compares to almost anything , it seems that some agreement on a valid testing protocol would be useful . Even with the best efforts here and on other test sites I consistently see issues that make it hard to get value out of the tests . I fully understand that nothing substitutes for getting your hands on a camera and running your own tests. Here are two that I see repeatedly .

1. Lack of a clearly visible plane of focus . I like to see that the area just before and after the target is in the plane of focus . A very slight front or back focus can destroy the value of any test and if you can t see it you don t know . Shooting a flat surface makes it easy to see edge sharpness but how many times is it a miss focus ?

2. Consistent processing of raw files . Right out of the camera doesn t mean much to me . How many comparisons have you seen where one file is cold and one warm ? I want to see a file processed to a standard . Camera calibration,wb,exposure ,scene contrast should be the same . Sharpening,noise reduction etc should be tuned to the camera . An M9 does t benefit much from aggressive sharpening ..its a CCD without the AA filter that creates very sharp and high contrast files . A nikon is a CMOS and some have AA filters . It needs deconvolution sharpening to offset the blur of the AA filter . It will take more sharpening than an M9 file . An S2 produces a linear contrast tone curve ..add a preset to pull down the curve and you have significant pop in the image.

I have a few others but this could be a start ....agree ..disagree .... have other issues with most testing protocols. Waste of time ?

 

[johnnygoesdigital]

I agree...Many of these images are landscape "wides" with significant crops. Perhaps as suggested a more pronounced POF, as particular lenses behave much differently at various f-stops. Diffraction alone is not a constant, so must be consider with each lens used. Even such things as monitor calibration and color profile should be standardized, so the most accurate results from the d800 can be obtained.

I'm not sure any camera model has been so throughly scrutinized, (at least not a recent 35mm), as the D800. It's direct comparison to MFD are obvious, so a standard protocol is a must...good call.

Let's also shoot some fashion with lots of sheer fabric and shimmering water, using a tighter portrait lens with the D800e, with no crops.

 

[Shashin]

You are tilting at windmills. Too many variables and you really don't need it. The current image examples show exactly what the camera does. You can even go to DPreview if you want a fixed/standardized target.

 

[Jack]

Roger,

I agree. I routinely shoot with ANY new lens at a static, 3-dimensional subject at wide aperture to establish the lens' PoF profile, usually aiming down slightly at a large expanse of lawn where I can follow the PoF in the blades of grass. You'd be amazed at how many contemporary lenses have pronounced 2nd order curvature or even 3rd order mustache-shaped PoF's. These mitigate as you stop down, but if you want to eek everything out of your camera, it helps to know how they behave.

Processing is one of my pet peeves -- processing files from different cameras identically tells us almost nothing. Processing each file optimally makes more sense, but then it takes some time to learn how to work each ideally -- and then one persons ideal is often different from yours or mine. I tend to focus my efforts on maximizing micro detail and letting other nits (like noise) fall where they may.

To date, I think PhotoZone does the best job with their standard targets for categorizing lens traits; it gives us a base image to compare differing options on.

 

[Shashin]

processing files from different cameras identically tells us almost nothing..

Doesn't that tell us how the camera sees in comparison to another camera? Which is kind of the point in a D800/D800E comparison. I usually can look at a file an see how much further I can go with processing. I do like to see unprocessed images--things can be masked by good processing or emphasized by bad, and vicky verky.

But I am with you that an image is not a final answer. It is just a point along the way. And even unprocessed images do not show a great difference between these cameras. And then there is the experience of the folks trying to make a conclusion--absolutes do not really work well in photography, even Black & White has shades of gray.

Still, I am probably being far too reasonable if I were trying to pick a "winner."

 

[tashley]

I'm slightly in the 'too many variables' camp and I totally agree with Jack about processing: whatever the lens and camera combination, processing is capable of bringing out almost anything you hope to find. There are plenty of people who think the DPR test targets aren't useful, that DXO mark is too narrow or favours certain kinds of pre-cooking etc etc. So even if we got the world's ten best sensor, software and lens designers together they'd not agree on protocols.

I always test to pin down the one thing that I am trying to assess, whether it's edge to edge sharpness of a lens, or shadow noise, or whatever. Trying to test for too many things at once dooms one to endless criticism and failure!

So I think we have the perfect test bed here: a community of people who know each other somewhat, understand the same questions and ways of answering them, and are amenable to doing their own tests and sharing them or doing a 'could you please' test for other forum members.

So I propose the 'GetDPI Protocol': test, share, listen, test again, send the gear back for exchange... :ROTFL:

 

[Vivek]

So I propose the 'GetDPI Protocol': test, share, listen, test again, send the gear back for exchange... :ROTFL:

+1

 

[Shashin]

In all honesty, how big of a "mistake" will you make if you choose the "wrong" model? Photography is an art, hobby, or business, not a psychiatric disorder.

 

[Guy Mancuso]

One thing that helps is I almost always provide the raws so others can process with there favorite raw converter. I think we all would agree that so many variables come into play especially when you try to compare say a D800 and a phase back which are two completely diffrent animals. The big trick as Tim mentioned is try not to compare too much in one series of images and try to keep the variables down to a minimum. But end of day there are no absolutes here either. I know a lot of science involved on some things but it is art and that leaves it up to variables as well. Most of us are not scientist and really go by what our eyes see which can be different among ourselves. I look for clues in almost everything I have read on the D800 tests in particular. Not one single person testing is going to give you a 100 percent answer on everything. It's a puzzle that just needs to be put together.

 

[paulraphael]

Photography is ... not a psychiatric disorder.

Are you sure?

 

[ustein]

>So I propose the 'GetDPI Protocol': test, share, listen, test again, send the gear back for exchange.

+2

>In all honesty, how big of a "mistake" will you make if you choose the "wrong" model? Photography is an art, hobby, or business, not a psychiatric disorder.

+2

I like what I get from the D800 but it did not make me better artistic photos (yet ?). But I think more pixels never hurt.

 

[glenerrolrd]

Roger,

I agree. I routinely shoot with ANY new lens at a static, 3-dimensional subject at wide aperture to establish the lens' PoF profile, usually aiming down slightly at a large expanse of lawn where I can follow the PoF in the blades of grass. You'd be amazed at how many contemporary lenses have pronounced 2nd order curvature or even 3rd order mustache-shaped PoF's. These mitigate as you stop down, but if you want to eek everything out of your camera, it helps to know how they behave.

Processing is one of my pet peeves -- processing files from different cameras identically tells us almost nothing. Processing each file optimally makes more sense, but then it takes some time to learn how to work each ideally -- and then one persons ideal is often different from yours or mine. I tend to focus my efforts on maximizing micro detail and letting other nits (like noise) fall where they may.

To date, I think PhotoZone does the best job with their standard targets for categorizing lens traits; it gives us a base image to compare differing options on.

Wow we even use the same test . I learned this with my M s which I am never sure are calibrated perfectly . Throw down a baseball hat on the lawn and at least you can see the point of focus and the impact of field curvature .

I like your terms IDEALLY verse IDENTICALLY .

I don t have much trouble testing a lens . I tend toward Leica glass and I ve found something to like about all of them ..just a matter of trade offs and matching the glass to the shooting requirements .

Camera sensors I am much less confident and this is where processing standards really come in.

 

[Steen]

(...) Photography is an art, hobby, or business, not a psychiatric disorder.

I am ready to believe that this "E vs. non-E" does not have any significant practical importance for print.

But if someone has put a blurring filter on my sensor, I probably could not keep me from constantly looking for influences by zooming into the capture, like when the tongue revolves around a hollow tooth ...

 

[Shashin]

I am ready to believe that this "E vs. non-E" does not have any significant practical importance for print.

But if someone has put a blurring filter on my sensor, I probably could not keep me from constantly looking for influences by zooming into the capture, like when the tongue revolves around a hollow tooth ...

And so you buy the E and you're happy and the world seems fresh and new every day. There really is no need to get stressed out over the choice. It is a marvelous camera, just like the non-E version.

 

[glenerrolrd]

In all honesty, how big of a "mistake" will you make if you choose the "wrong" model? Photography is an art, hobby, or business, not a psychiatric disorder.

Is that why you look at the tests ? When anything new is released by Leica,Nikon etc that fits my shooting requirements and looks to have potential ..I just order it several places . Otherwise I might not get it for a very long time . Amazon,BH and Adorama all have liberal return policies so I will not lose more than freight charges .

What I look for are the performance characteristics ....things I might find out through frequent use but its helpful to see what others are saying . People kick DxO mark around because they don t see the value in a composite score ...I agree . But go down a few layers and you can learn things that take a lot of time to test . Here is an example ....the impact on loss of DR from just one EV in increased ISO exceeds the difference between the newest Nikon s and the original D3 . So I learned ...all things being even try to shoot at the native ISO and give ground slowly when you need a better F stop or shutter speed.

My points are actually much simpler ...when comparing try to get the files somewhat close in wb,exposure ,contrast ..so we are not looking at imagined resolution due to contrast . They don t have to be perfect ..they just should be close enough to allow for a casual observer to sort of get the point .

Plenty of places to get ahold of raw files but I would rather see what Jack does with them .

Purely a subjective POV on my part no right or wrong ..just suggesting a few things to make the testing better ..hopefully without making it a lot harder.

 

[Vivek]

.. like when the tongue revolves around a hollow tooth ...

That is a beauty, Steen!

 

[glenerrolrd]

You can see an excellent example of a test that doesn t meet the standards I proposed . Diglloyd is normally very careful to create tests that bring out the differences in performance of new gear from exiting standards . On his latest test of the D800 verse the D800E ..IMO ..he rushed it .

Identical framing of a scene with terrific detail . He used the identical Zeiss 21/2.8 ZF.2 lens on both bodies . Here are the differences that cloud any conclusions on my part .

1. The sample have slightly different focus points resulting in one being slightly front and one slightly back. Sort of kills any edge comparisons .

2. Lighting changed and one set was under cloud cover and the other was in diffused sunlight . Big difference in the contrast of the light .

3. Color balance ..one of my favorites ...one is cool the other warm ..guess which one looks sharper.

4. Post processing ..no evidence of tuning the sharpening etc to the two different files .

His conclusion is that the D800E is noticeably better ..and it looks that way in the test images ...this is why quick and dirty tests are hard to use . I am sure he will create bullet proof tests when he has access to a D800E body .