Sony lenses on Roger's optical bench...compared.

[uhoh7]

Extremely interesting post here:

LensRentals.com - Sony E Mount Lens Optical Bench Tests

What stunned me was the copy variation.......of the Otus!! And other lenses also. FE 90/2.8 is not quite the dragonslayer I thought, and has huge copy variation.

Also of note is the Sony lenses need optical glass in the path to work right......

I hope to see something besides WO, or am I missing a link?

 

[dandrewk]

Lens manufacturers design and produce lenses for their own camera, or at least for specific cameras.

IMHO, removing the camera and sensor from lens tests negates any real value from those tests. Is it any wonder that LR's results differ significantly from just about every other test?

And how are we to interpret having the necessary optical glass for Sony's lenses? I understand the necessity, but how is it then fair to compare results with other systems?

 

[turtle]

If we ignore the fact that the 90mm Macro tested did not perform as well with Lensrentals as elsewhere, I can only agree on the variance issue. Removing the camera and sensor has no bearing on the fact that with some of Sony's lenses on the optical bench, the various sides and corners are very different.

Inconsistency in performance (and symmetry) with some FE lenses is frankly staggering. It took me five lenses to get a very good 55mm (second copy great, the first showing the worst decentering I have ever seen in a lens) and one 35mm FE (third copy, the first two having soft sides) that still has one lesser extreme corner but is otherwise excellent. That's disgraceful. I returned as many Sony Zeiss FE lenses in two months as I have Canon, Zeiss, Leica and Pentax lenses in ten years. My Sony FE 28-70mm kit lens is also clearly decentered (horrible lower right quadrant until f11, at which point it is OK), but I simply could not be bothered to continue the mail ping pong. Thankfully my 70-200 f4G was perfect on the first copy, but I bought that from a local dealer at a higher price specifically because I could exchange it same day if I had any problems. Thankfully I did not have to. However, the end result is three good lenses out of seven copies.

I have read quite a few comments about decentered 90mm Macro G and Zony 35mm f1.4 lenses. Great copies are very impressive, it seems, but my impression is that one encounters users commenting about exchanging copies much more often than with, say the Canon 100mm f2.8 IS L macro, which surely has sold vastly more copies. The only thing making the Zony 35mm f1.4 look less than perfect is the new Canon 35mm f1.4 L II; however, that lens is ridiculously good and even more expensive.

For some time I have gotten the impression that Canon has been producing the most consistent and some of the best mass produced lenses at the moment and Lensrentals shows that Canon is indeed at the top of the pile, especially with recent models. Now all they need to do is sort their sensors out! While Roger's report shows up the shortcomings of Sony, it does at least show how well Canon is doing, so its not all bad news.

 

[ErikKaffehr]

Hi,

They know what they are doing? One difference is that Lensrentals only test at full aperture normally. That is logical as much of their testing is intended as quality assurance of the lenses going out to customers.

But, the downside is that maximum aperture is seldom where lenses perform best.

The optical industry lives by MTF and it is the generally accepted measure of lens sharpness. For instance, Leica, Zeiss, Hasselblad, Schneider and Rodenstock present MTF data for their lenses in a pretty similar format to LR.

But the manufcturers use two sets of curves, one for maximum aperture and one stopped down.

The MTF curves essentially show sharpness, albeit astigmatism, field curvature and lateral chromatic aberration may be deduced from MTF curves.

Especially for Sony users, MTF testing adds some benefits, as the new FE mount lenses can use all kinds of lenses from Canon, Nikon, Zeiss you name it.

When testing lenses on camera it is not possible to separate camera from lens. A given lens from Zeiss will perform better on the Nikon D810 than on the Canon 5DIII, as the Nikon has higher resolution. If you test on the 5Ds the Canon version will be superior. And the 5DsR will give superior results to the 5Ds.

You see this pretty clear in the DxO tests, all of the top rated lenses are tested on Nkon D8x0 and Sony A7r. Once they start testing on Canon 5DsR all winners will be Canons.

Another factor is that MTF values measured on an optical bench are not affected by sharpening, but some sharpening is generally involved in camera based testing.

Anyway, lens rentals is not a lens testing outfit. They test lots of lenses as a part of their daily operations and they share some of their findings.

Best regards
Erik

Lens manufacturers design and produce lenses for their own camera, or at least for specific cameras.

IMHO, removing the camera and sensor from lens tests negates any real value from those tests. Is it any wonder that LR's results differ significantly from just about every other test?

And how are we to interpret having the necessary optical glass for Sony's lenses? I understand the necessity, but how is it then fair to compare results with other systems?

 

[ohnri]

Extremely interesting post here:

LensRentals.com - Sony E Mount Lens Optical Bench Tests

What stunned me was the copy variation.......of the Otus!! And other lenses also. FE 90/2.8 is not quite the dragonslayer I thought, and has huge copy variation.

Except that it is a Dragonslayer in real usage. And the lens testing is done under conditions that are impossible to achieve when actually taking a photo. And the copy variation, which is calculated in a nearly opaque and arbitrary manner, may or may not have anything to do with the final image quality anyway.

Unfortunately, I have now seen a number of people report reconsidering purchasing some really excellent lenses because of this blog post.

The increasing trend toward quantifying the performance of every camera and lens into one or a small handful of numbers based on bench testing is not, in my mind, always beneficial.

Fortunately, the option of looking at the results of the work of many photographers and reading the reviews from a variety of sources is more available than ever.

I am reminded of two quotes:

There are three kinds of lies: lies, damned lies and statistics.
And
A picture is worth a thousand words.

In this case, I'll go with the picture.

Or, rather, I'll go with the thousands of pictures over the statistics. It is the pictures I am going to want to look at anyway.

-Bill

 

[turtle]

You seem to be ignoring the (too many) irritated owners who have had to send back one or more Sony/Zesiss FE lenses. It has nothing to do with opaque and arbitrary numbers and everything to do with sloppy assembly and poor quality control compared to other major manufacturers.

Internet pictures don't tell you much about copy to copy variation. The thousands of pictures you speak of may give you an idea of the general characteristics of the lens and whether you like the look, but it will tell you nothing about the probability of your newly ordered $1500 35mm Distagon arriving with a soft left side.

... And the copy variation, which is calculated in a nearly opaque and arbitrary manner, may or may not have anything to do with the final image quality anyway.

Or, rather, I'll go with the thousands of pictures over the statistics. It is the pictures I am going to want to look at anyway.

-Bill

 

[dandrewk]

You seem to be ignoring the (too many) irritated owners who have had to send back one or more Sony/Zesiss FE lenses. It has nothing to do with opaque and arbitrary numbers and everything to do with sloppy assembly and poor quality control compared to other major manufacturers.

Internet pictures don't tell you much about copy to copy variation. The thousands of pictures you speak of may give you an idea of the general characteristics of the lens and whether you like the look, but it will tell you nothing about the probability of your newly ordered $1500 35mm Distagon arriving with a soft left side.

Define "too many"? What number is that? How does one determine a statistic or percentage based on what someone might or might not post on an internet forum? If they should post, how certain are we that their evaluation or methodology is valid? e.g. One person may see a "defect", and another one does not.

The point is forum posts are purely anecdotal, and often biased. We also have to consider that problems are much more likely to be publicized than no-problems.

From looking at posts in Sony forums, here and elsewhere, there is a good deal of overlap on folks reporting issues. Some are quite vocal and post several times, often with increased rhetoric.

I see, hear and read issues with QC with other makes. Just briefly peruse Nikon, Canon forums and you will always see issues, and believe me those that have those issues are quite upset about it.

To be clear, I am not saying there aren't QC issues with Sony. But nobody will know if they are better/worse than any other manufacturer. It's all conjecture.

And I repeat my earlier post, I don't put much stock in tests that don't include the mated camera. If I care to consider MTF charts, -at all-, I'll consider methods that have been tried/tested/proven for decades.

 

[Vivek]

Unfortunately, I have now seen a number of people report reconsidering purchasing some really excellent lenses because of this blog post.


-Bill

I am not one of them. I am not persuaded or dissuaded by that testing and/or any mis representations of the results there.

The 24-70 Zony zoom I bought and returned (2 samples) had nothing to do with lens rentals. However, the 3rd sample that I never bought and will not buy has everything to do with what they showed- ie. how it is put together.

Not worth my cash.

How about Sony increasing the quality of their products and stop peddling the Zeiss logo for no good reason other than profits?

 

[ShooterSteve]

I'm not sure you can really expect mass produced AF optics to ever had the quality control that will provide 100% defect free lenses every time at the prices people are willing to pay. In thirty years plus of professional shooting and owning almost every brand of camera from 35 - 8x10, I can only remember one or two instances where I felt the lens I purchased was not up to expectations. I returned them and remedied the situation.

It's amazing to me that any large aperture AF lenses can maintain perfect optical alignment through production and wear and tear for very long.

If you really want to make sure you invest your money in only superb lenses, buy Leica C or Zeiss Ultra Prime cine lenses. Plan on spending about $20k per lens, and you are are assured of perfect quality every time . I use them on a RED occasionally that they will satisfy the most critical optical snobs, although many only cover APS-C (S-35 format).

In comparison, I think Zeiss is doing a very good job with their new lenses and is probably striking a perfect balance between quality and price. And until Sony zooms prove themselves to be up to at least the standards of the latest Canon L lenses, I'll probably hold off on those, although that 16-35 is very tempting

I was a bit disappointed that Sony didn't release or announce more new lenses at Photo Expo last week, of did I miss something?

 

[turtle]

We are rarely in command of fully researched and robust data that is beyond reproach, but isn't that typical? We all have to make our own 'assessment' and yours is that there is insufficient data to reach a conclusion that Sony/Zony lenses have QC issues that neccessrily have any effect on imaging performance. Mine is that there is ample data for me to make a confident personal assessment that Sony Zeiss has issues. That a personal call, but it comes from:

1. Direct personal experience (yes I could be a statistical anomoly, but see next point)
2. Having read lots of reports of people ending multiple copies back one after the other (which I have very rarely heard about with larger manufacturers which have a LOT more lenses out there)
3. The Lensrentals optical bench.

Sure, the optical bench does not tell us for sure what the impact would be 'on sensor', but it does tell us which manufacturers have the greater variances in resolution across the lens. Considering the demands of pixel rich sensors, its not hard to see how substantial variances would at least translate into some kind of variance on sensor. Yes, this could be a gross error, but interestingly, this marries up with what a substantial number of users are saying. To assume that the sensor would somehow mask measured decentering (on the optical bench) is much more of a stretch than the chain of reason outlined above.

I feel this is a classic case of it looks like a duck, walks like and duck and quacks like a duck, it is probably a duck. So far, all the information in this chain is congruous.... No, this does not mean it is impossible that a false conclusion has been drawn and we are in fact looking at a Kangaroo, however, based on available information would appear more likely to be a duck.

Define "too many"? What number is that? How does one determine a statistic or percentage based on what someone might or might not post on an internet forum? If they should post, how certain are we that their evaluation or methodology is valid? e.g. One person may see a "defect", and another one does not.

The point is forum posts are purely anecdotal, and often biased. We also have to consider that problems are much more likely to be publicized than no-problems.

From looking at posts in Sony forums, here and elsewhere, there is a good deal of overlap on folks reporting issues. Some are quite vocal and post several times, often with increased rhetoric.

I see, hear and read issues with QC with other makes. Just briefly peruse Nikon, Canon forums and you will always see issues, and believe me those that have those issues are quite upset about it.

To be clear, I am not saying there aren't QC issues with Sony. But nobody will know if they are better/worse than any other manufacturer. It's all conjecture.

And I repeat my earlier post, I don't put much stock in tests that don't include the mated camera. If I care to consider MTF charts, -at all-, I'll consider methods that have been tried/tested/proven for decades.

 

[ohnri]

You seem to be ignoring the (too many) irritated owners who have had to send back one or more Sony/Zesiss FE lenses. It has nothing to do with opaque and arbitrary numbers and everything to do with sloppy assembly and poor quality control compared to other major manufacturers.

Internet pictures don't tell you much about copy to copy variation. The thousands of pictures you speak of may give you an idea of the general characteristics of the lens and whether you like the look, but it will tell you nothing about the probability of your newly ordered $1500 35mm Distagon arriving with a soft left side.

When people actually show an image that leads them to return a lens - that is exactly the data I would look at.

I agree that it has nothing to do with arbitrary numbers.

Can you really make a fair comparison to other manufacturers from this data?

Hardly. Read a little of Thom Hogan's discussion of Nikon's QC issues for some perspective. Getting my D800 and 85/1.4 D to be sharp was always an adventure.

Or look at Leica. I have used Rangefinder Leica's for nearly 50 years. Talk about focus issues! A minor bump can so easily lead to miscalibration and to misfocus. Or, with digital M's, worrying about whether my camera was set for the optimal distance of focus for each lens - rah, rah focus shift. Basically, you almost had to mate a specific lens to a body for critical close to medium focus at wide aperatures.

Or, try to critically Manually Focus any DSLR at f/1.4 through the viewfinder. Hah! It is almost not possible due to the optical design of modern viewfinders.

I have a higher percentage of critically in focus, wide aperature shots from my 35/1.4 and 90/2.8 using the Sony A7r2 and Eye Detect than I ever got from other systems. So, now which lens is better? The ones I get critical focus from or the ones I don't ? The whole imaging chain is important.

And looking at many internet pictures from many lenses would seem to me to be a perfectly reasonable way to make a rough guess at whether the lens I am going to order will fulfill my expectations.

If Lens Rentals took their many copies of lenses and shot many types of images while looking for many different image qualities it would mean so much more. But, that would also be so much more difficult and expensive and time consuming.

That is probably why no one does it.

-Bill

 

[Vivek]

If Lens Rentals took their many copies of lenses and shot many types of images while looking for many different image qualities it would mean so much more. But, that would also be so much more difficult and expensive and time consuming.

That is probably why no one does it.

-Bill

Any idea what a Zheiss QC certificate entails?

 

[turtle]

People do show images that cause them to send lenses back...

I think you are conflating issues:

1. D800 + lens issues were in many cases due physical focus problems, lens compatibility and possibly pixel pitch.... not optics.
2. The Leica M issues you speak of are due to RF calibration.... not optics.
3. Manual focus of DSLR lenses vs. live view EVFs is about systems.... nothing to do with the optics themselves.

Whether or not a lens meets your expectations has nothing to do with whether or not there are optical quality control issues at Sony/Zony. Surely it just means they are below your tolerance threshold, or do not matter in the context of the other benefits the system brings you?

Regarding your last point, I think you are missing the point. You say "If Lens Rentals took their many copies of lenses and shot many types of images while looking for many different image qualities it would mean so much more. But, that would also be so much more difficult and expensive and time consuming". This statement is unconnected to what I have been saying or what Lensrentals has been saying (optical consistency within lenses and across samples i.e. decentering). You appear to be talking about something quite different i.e. 'why don't they look at factors other than optical bench measurements and cross frame and batch consistency, to produce a more rounded profile/evaluation of the lens.' The answer is 'because they aren't, don't pretend to be and this discussion is about what they are showing and not what they aren't.' If you don't think the evidence supports there being optical QC issues with Sony/Zony, that's fair enough. If you don' think they matter (to you), that's also fair enough. If you prefer to give consideration to optical qualities other than even performance across the frame, that is of course also understandable. However, confusing the various issues is, well, confusing.

When people actually show an image that leads them to return a lens - that is exactly the data I would look at.

...
Hardly. Read a little of Thom Hogan's discussion of Nikon's QC issues for some perspective. Getting my D800 and 85/1.4 D to be sharp was always an adventure.

Or look at Leica. I have used Rangefinder Leica's for nearly 50 years. Talk about focus issues! A minor bump can so easily lead to miscalibration and to misfocus. Or, with digital M's, worrying about whether my camera was set for the optimal distance of focus for each lens - rah, rah focus shift. Basically, you almost had to mate a specific lens to a body for critical close to medium focus at wide aperatures.

Or, try to critically Manually Focus any DSLR at f/1.4 through the viewfinder. Hah! It is almost not possible due to the optical design of modern viewfinders.

I have a higher percentage of critically in focus, wide aperature shots from my 35/1.4 and 90/2.8 using the Sony A7r2 and Eye Detect than I ever got from other systems. So, now which lens is better? The ones I get critical focus from or the ones I don't ? The whole imaging chain is important.

And looking at many internet pictures from many lenses would seem to me to be a perfectly reasonable way to make a rough guess at whether the lens I am going to order will fulfill my expectations.

If Lens Rentals took their many copies of lenses and shot many types of images while looking for many different image qualities it would mean so much more. But, that would also be so much more difficult and expensive and time consuming.

That is probably why no one does it.

-Bill

 

[Jack]

I have said it before, but it bears repeating: E V E R Y lens is subject to sample variation, and the only way you will know if yours is good (enough for you) is to test it (on your typical subjects at your typical apertures).

rant This is where I have issues with the dedicated brand adherents -- fanboyz if you prefer -- when it comes to anybody's camera or lens line. Loyalism is one thing, but uninfomred loyalism as respects quality and performance is simply sophomoric. /rant

 

[ohnri]

People do show images that cause them to send lenses back...

I think you are conflating issues:

1. D800 + lens issues were in many cases due physical focus problems, lens compatibility and possibly pixel pitch.... not optics.
2. The Leica M issues you speak of are due to RF calibration.... not optics.
3. Manual focus of DSLR lenses vs. live view EVFs is about systems.... nothing to do with the optics themselves.

Whether or not a lens meets your expectations has nothing to do with whether or not there are optical quality control issues at Sony/Zony. Surely it just means they are below your tolerance threshold, or do not matter in the context of the other benefits the system brings you?

Actually, that is part of my my point. I care about my final image. What is being tested appears to have zero application to my final image. I am not going to claim Sony has a giant QC issue when I get better images with it than I do with other systems. So, yes, if a lens and camera imaging chain meets people's expectations it has everything to do with whether QC is sufficient or not.

If others, such as yourself, have found inconsistency such that your final images are not up to your desires then I understand your frustration. If you can produce the images you want consistently with other cameras and lenses from other companies then that becomes more interesting in terms of whether Sony has an issue or not.

I will certainly not claim Sony is worse than other companies based on this data. There is no basis for stating Sony lens/camera IQ is worse than that of Nikon or Canon based on anything Lens Rentals is doing.

Many people think that the D610 and D810 exist solely to fix QC issues with the D600 and D800.

Even if Canon is a paragon of consistency they don't make a camera I want or even a camera sensor I want. So I can consistently get images I don't want. Or use adapters, which apparently have their own problems. So maybe Canon is guilty of sloppy design. Putting mediocre sensors in big bodies that don't even do 4K video.

I will, finally, repeat what I have said before. Tests done on a lens, alone, in one setting and wrapping the results in opaque and arbitrary math is not going to be high on my list of criteria for choosing a camera/lens combination to shoot with.

And, finally, finally, I am finished with this back and forth. I believe you are fully entitled to you opinion and that it is valid for you. I believe that Sony might really have issues with lens consistency that probably fall below the threshold of the vast majority of users notice. I believe that if only a tiny percentage of users have a problem with consistency then it is not really a problem. I believe it is impossible to divine anything meaningful about Sony vis a vis other companies regarding image quality from any readily available lens testing. I believe in images more than Frankenstein test rigs.

I believe you are sincere in feeling Sony has a problem. That is fair.

I believe Roger is sincere in his tests and feeling they are helpful. Maybe they are.

I believe people are using Sony cameras and praising their lenses because they are getting amazing results.

I believe Sony can probably do even better and feedback is an important part of that.

I believe I have had great results with every camera system I have ever used as well as real frustrations with each one. That includes my current Sony system.

I believe I have no more time to spend on this issue at the moment but may return to it down the road.

Take care,

Bill

 

[tn1krr]

Lensrentals also tests these things with camera/sensor inequation, with Imatest. Here's what Roger had to say about FE lenses on Imatest

http://www.dpreview.com/forums/post/56675717

Also, it seems that manufacturers including Sony respect Roger's work and he works with them quite closely and some parties even contract Lensrentals lab for testing purposes only.

http://www.dpreview.com/forums/post/56675944

As for lens variation issue overall I have one question: how many people over here have exchanged a "pure" Zeiss lens for decentering etc. issues. Forums are full of reports of how many tries it took to get a good copy of a Sony Zeiss lens yet I'm yet to see a report about decentered Loxia or Batis despite those lenses selling well enough for them to be constantly backordered. Coincidentally the pure Zeiss lenses (ZE/ZF, no numbers available for Loxia/Batis yet) tend to score very highly on Lensrentals variation tests and the pure Zeiss lenses are very adjustable/correctable if they get out parameters

Lensrentals WA copy variation tests for DSLR lenses

http://www.lensrentals.com/blog/2015/07/variation-measurements-for-wide-angle-lenses

And please do not take me as hater, I have never owned a non-Sony system camera and currently own a pile of Sony Zeiss lenses. I love my FE 90/2.8 for macro, it is stellar in short range both optically and in useability, but I have not trouble admitting that my Batis 85 looks sharper than 90 at infinity, just like dxomark (short distance test) and lensrentals test bench (infinity) suggest.

 

[Guy Mancuso]

Not questioning Roger here and we all know sample variances exist. Trust me I had my share in every brand and several lenses in Sony for sure. Granted I don't know enough to understand they way he is testing these Sony lenses but taking the lens off the body makes me nervous. Even though it could be correct still makes me nervous . Also to Rogers credit he did put in his warning on it. Now I know for a fact I had a bad 35 2.8 and 2 bad 24-70 and I know I have a very slight corner issue with my 35 1.4 but I actually consider mine. Very good copy. 1.4 is a tough beast to test wide open and corners are never perfect in any brand. I had my issues with Nikon, canon and others as well. This test just makes me a little nervous. Not saying it's wrong, right or anything else. Again I'm back to the puzzle and this is just another piece to consider

 

[uhoh7]

I have said it before, but it bears repeating: E V E R Y lens is subject to sample variation, and the only way you will know if yours is good (enough for you) is to test it (on your typical subjects at your typical apertures)

Yes every lens has copy variations. How much they vary, that is the question. Roger, who I cannot thank enough for all the time he puts into this, now has the numbers to show which are the worst offenders:

"The FE 35mm f/1.4 ZA lenses are all over the place. It actually is a bit worse than the graphs look because a lot of the variance is WITHIN a copy, not just copy-to-copy. None of the 10 copies we tested had even corners. And I'll editorialize and say that none of the dozens we've tested on Imatest had even corners either. If you use this lens for centered objects, you'll be happy. If you want 4 sharp corners, it's not likely to happen unless your standards for equal sharpness are pretty low."
Roger.

So testing "in your usual shooting method" is not going to cut it really for some of us, who like Steve Huff, are all about the central subject, and don't really look hard at the edges much. But given the choice I imagine Steve would like the best copy too.

I suggest anyone who cares about Sony read that blog several times word for word. It's a bombshell if you really care about quality.

Not just Sony, but you see Zeiss, Sigma, Canon, Nikon variations from a large sample of copies. Hopefully we will see some figures for M as well.

Also the F/4 and 5.6 and f/8 numbers will come eventually.

 

[vjbelle]

As for lens variation issue overall I have one question: how many people over here have exchanged a "pure" Zeiss lens for decentering etc. issues.

I have....... it took 3 Zeiss Otus 55mm lenses before I got one that really lived up to the hype..... all went back to Zeiss and I have the paperwork to back this up. I will say that they are really above board to work with - actually sending me a new copy before they got the old one back from me.

Victor

 

[Vivek]

So the QC certificate with some guy's signature does not mean much? :bugeyes: