LensRentals.com

Sony E Mount Lens Optical Bench Tests

Posted by

It has taken a long time, the sacrifice of a couple of cameras and lenses, and a lot of soldering, but we have finally managed to rig up a powered Sony mount to our optical bench so that we can test the new full-frame Sony and Sony-Zeiss E-mount lenses.

For those of you who haven’t understood why there has been a problem, I’ll explain it for you. The optical bench tests lenses at infinity focus. Most lenses have a mechanical focusing ring. You set it to infinity, confirm it (because some lenses can actually focus past infinity, which screws up the test results), put them on the bench and test them. A few lenses are ‘focus-by-wire’; they are focused by the electric AF motor even when you turn the focusing ring manually. Most of these are no trouble either. You put them on the camera, focus to infinity, take them off and the focus stays where it last was.

Sony’s newer lenses (and some micro 4/3 lenses) are different. They are focused by an electromagnet. You can put them on the camera and focus to infinity, but when you take them off the camera the focusing group just falls back to wherever it wants to be, which is never infinity focus.

So in order to test Sony lenses on the optical bench we had to find a way to keep the lens hooked to the camera, but at the same time have it off the camera because it has to be mounted to the bench’s Sony test plate. Our solution isn’t particularly elegant, but it works. We took apart a random lens and camera to get a lens mount and camera mount. We modified one of the Optical Bench mounting plates so it contained a Sony camera mount. We then wired that to an empty lens mount. We place the test lens in the mounting plate (red line), put the empty lens mount into the camera, and we can control the lens focus by using the camera while the lens is on the optical bench.

 

Yeah, it looks crude. But this is version 3.0; the first two attempts were much cruder. Before you make fun remember we sacrificed an A7, an 18-200 lens, and a $900 optical test plate just to get you some numbers. OK, we really did it because it sounded fun and because nobody else in the Known Universe can test Sony lenses on an optical bench right now and we can. So there!

Remember This is Beta Testing

The goal and objective is to compare Sony lenses both to each other and to other E-mount lenses from Zeiss, Sigma, and whomever. But I know people are also going to compare them to Canon, Nikon, and other lenses. I want to emphasize that we’re learning as we go and I’m writing as we learn. So two months from now I may be writing one of those articles that tell you we’ve found an error in our technique or a better way to do it. I don’t think so, we’ve been experimenting for a while now, but that’s what happens when you do new things that nobody else is doing.

More importantly, there are some differences in the testing methods for the Sony-mount lenses compared to the Canon and Nikon mount tests we’ve been doing. We want to be as transparent as possible, so I’m going to identify all of those differences I can think of.

There is Glass in the Path

First, unlike most of the Canon and Nikon mount tests we’ve already done, we’ve added 2mm of optical glass to the testing path of these lenses. Why? Because we ran lots of trial tests and found that at least the first Sony lenses we tested performed much better with that amount of glass in the optical path. It made a big difference with the Sony lenses, while it did not make much difference for most of the Canon and Nikon lenses we tested. Why are Sony lenses more sensitive? My best guess, and it’s just a guess, is that their exit pupil is further to the rear than some of the other lenses. It’s also very possible that the shorter flange-to-sensor distance (18mm for Sony, 44mm for Canon) makes the lenses more sensitive to glass in the optical pathway. (For both of you who want to know we tested both 3 and 1mm and 2mm gave the best results.)

We ‘See’ Fewer Corners

With all of our other lenses the optical bench gives us a complete open circle. We test 4 quadrants and see results out to the edge of the frame (20mm of sensor distance) in all 4 directions. Our electronic Sony mount includes the rectangular light baffle that the Sony camera mount has. This means at certain rotations (top to bottom in camera terms) our view is cut off at roughly 15mm, the distance of the top or bottom of the sensor from the center. The result is our numbers for a Canon or Nikon lens measure each lens all the way to 8 edges (2 edges for each of 4 rotations). With the Sony lenses we only measure 6, the top and bottom are cut off. You can make an argument that this might make the variation appear slightly larger (there are few numbers in the mathematically average), or slightly smaller (we might not ‘see’ a bad corner) than it would be if we could measure all 4 corners. I think the only important point is that we should be conservative when making comparisons to Canon and Nikon tests. The tests are slightly different.

There May Be a Bigger Difference Between These Tests and Pictures You Take

We know that Sony ‘cooks the RAW’ at least a bit, doing some in-camera modifications to raw files. I don’t know exactly what or to what degree. They aren’t alone in this, it seems to be the wave of the future to do some in-camera correction for at least distortion. But it appears they are the only full-frame cameras to do in-camera RAW correction at this moment.

What that means, though, is that tests of just the lens without a camera body, like we’re doing here, may be quite a bit different than what comes out of the camera. For example, our tests of the FE 35mm f/1.4 ZA show a LOT more distortion than tests done on a camera body using Imatest or DxO optics; we show 4% distortion where most of the on-camera testing shows 1% or less. This probably means that the camera is processing the distortion out of the image. (It could also mean that the distortion is very different focused up close, where DxO and Imatest work, then at infinity, where the optical bench works.) Depending upon your point of view that may be good, bad, or make no difference to you at all.

The same thing may (or may not) apply to sharpening. If there’s a little sharpening going on in the raw image, then you may think “Roger you said that lens isn’t so sharp, but the RAW images look very sharp, and the report from this other site using Imatest says it’s very sharp, too”.  To take it a step further, sharpening may be applied more to the corners and edges, or vignetting correction applied, or other stuff. (Again, I don’t know if it is or not.)

Because so many people struggle with this idea, I’ll repeat what I say over and over: I’m testing the lens alone. (You would not believe how many people ask what camera these tests are done on.) Other sites are testing the image that has been processed by the camera. Since you’re using the camera, their results will look more like your results and you’ll not see much of the distortion in your images. About 90% of you are going to think ‘why would I care’, and you’d be right. What comes out of the camera is more important.

But if you’re a lens geek like me, what the lens is actually providing TO the sensor is important, if only for esoteric reasons. Or maybe not just esoteric reasons. If the lens is inferior, but the image looks better from in-camera processing, you may not be able to manipulate the RAW as much as you would like, because it’s already been manipulated once. A lot of people are very excited about Sony uncompressed raw because they will be able to do more intense post-processing. But uncompressed doesn’t mean unmodified. If the raw is still modified before you get it, post-processing may still be somewhat limited.

Of course, the vast majority of photographers could care less, but then they probably don’t read this blog 🙂

Some Test Results

We have lots of Sony and Sony mount lenses to test and it’s going to take a while to get them all done. Today I’ll present 3 of the lenses many people have wanted to see: the Sony FE 35mm f/1.4 ZA, the Sony FE 55mm f/1.8 ZA, and the Sony FE 90mm f/2.8 Macro OSS. We’ll show you the MTF curves and copy-to-copy variation graphs for each, comparing them to some of the Canon and Nikon mount lenses we’ve tested. Again, please remember that testing techniques are slightly different for the Sony lenses as explained above.

Sony FE 35mm f/.4 ZA

(Note: this post originally published too early with an incorrect MTF graph for the Sony FE 35mm f/1.4 ZA lens. It has been corrected as of 4:20 pm CST. I apologize for the error, I was trying to get the post up as I was leaving for New York.)

We’ll compare the Sony to several other 35mm lenses. The Sigma and Canon are amazingly good 35mm lenses. The Sony is better than the more average Nikon 35mm f/1.4.  It’s certainly not quite as good as either the Canon or the Sigma as far as resolution goes.

 

Roger Cicala and Aaron Closz, Olaf Optical Testing, 2015

 

When we look at copy-to-copy variation, something we’ve long thought was the case becomes documented.

Roger Cicala and Aaron Closz, Olaf Optical Testing, 2015

 

I’m going to avoid comparing the consistency numbers, because as we’ve mentioned the testing for the Sony lenses is a bit different. But the variation graphs show what our photographic experience with this lens has suggested. The FE 35mm f/1.4 ZA lenses are all over the place. It actually is a bit worse than the graphs look because a lot of the variance is WITHIN a copy, not just copy-to-copy. None of the 10 copies we tested had even corners. And I’ll editorialize and say that none of the dozens we’ve tested on Imatest had even corners either. If you use this lens for centered objects, you’ll be happy. If you want 4 sharp corners, it’s not likely to happen unless your standards for equal sharpness are pretty low.

Sony FE 55mm f/1.8 ZA

Going into these tests, I had expected the 55mm f/1.8 ZA to do better than the 35mm f/1.4 and it did. Compared to a sampling of other 50-something lenses, the Sony 55 does very well on the MTF charts. Its center resolution is superb and it maintains sharpness very well to the edges. The Nikkor 58mm and Zeiss Otus 55m lenses are being tested here at f/1.4, which gives the Sony lens a bit of an unfair advantage, but it’s still excellent and holding it’s own against the best lenses in this range at the very least.

 

Roger Cicala and Aaron Closz, Olaf Optical Testing, 2015

 

The copy-to-copy variation graphs for the 55mm lens show it has a lot better consistency than the 35mm, too. There’s some variation, but it’s similar to most of the other 50ish lenses we’ve tested. The Canon 50mm f/1.8 is very much the exception, being the only 50mm lens we’ve tested that has almost no copy-to-copy variation.

Roger Cicala and Aaron Closz, Olaf Optical Testing, 2015

 

Sony FE 90mm f/2.8 OSS Macro

The FE 90mm Macro shows a good, not great MTF chart. It’s certainly not bad, but not something to get overly excited about. Remember, though, that Macro lenses often perform better close up, where they are designed to work, than at infinity focus, where this test is done. If you’re using this as a long portrait or short telephoto lens, these results should reflect what you see (other than if there’s sharpening done in RAW). At macro distances, it may perform differently.

 

Roger Cicala and Aaron Closz, Olaf Optical Testing, 2015

Things aren’t quite so good when we look at copy-to-copy variation, though. Medium focal length macro lenses are usually very consistent. In this case, the Sony has a lot of variation compared to the others. The type of variation we’re seeing is also quite different from the Sony FE 35mm f/1.4. That lens has a lot of corner variation in every copy, but the center remains is consistently sharp comparing one lens to another. With the Macro, there’s a lot of center variation and it’s not particularly worse in the corners. This suggests there is significant copy-to-copy variation in overall sharpness, rather than individual lenses have a bad corner.

 

Roger Cicala and Aaron Closz, Olaf Optical Testing, 2015

 

So What Can We Take Away Today?

Mostly we are just starting with FE lens testing. Over the next few weeks, we should get a good handle on how the Sony lenses compare to each other.

I think there’s no question we’ve seen support for what a lot of people are claiming: the 35mm f/1.4 ZA lens is very likely to have some tilt with one corner or side out of the plane of focus. Depending on what you photograph this may make no difference to you, or may be a huge issue. But I doubt sending copy after copy back is going to make much difference (you may trade a soft right upper corner for a slightly soft left side, etc.). We’ve taken a couple of these apart and there’s not a ton of optical adjustments that can be made: there are a set of shims behind the front group that can be modified a bit, but it’s a crude and very time-consuming adjustment, so I don’t think it’s generally going to be something that can be fixed.

The 55mm f/1.8 is really a good lens, very sharp and consistently made. The 90 mm f/2.8 is a decent lens, reasonably sharp, but there seems to be a fair bit of copy-to-copy variation in overall sharpness.

All that being said, these lenses may look better on-camera than on the optical bench because there seems to be at least some modification of files as the raw is written. We’ll get a better handle on that as we test more lenses. And let me emphasize, the only clear evidence we see for that is with distortion correction. There may be nothing more than that going on.

 

Roger Cicala and Aaron Closz

Lensrentals.com

October, 2015

  • Damir ?olak

    Thank you Roger. I went through three samples, they all had right 1/3 of the picture blurry. If I put focus on the right side, left 2/3 of the picture was blurry.

    No other lens I have exhibits the same behaviour.

    I would’ve been more than happy to have had just one of the corners out of focus. :-))

    It’s good to know there are samples out there that are better than those I had, I’ll keep searching for the “better one.”

  • I know some people are happy with theirs. Speaking for optical bench testing, we haven’t found one that was perfect on all 4 corners, but the optical bench can see flaws cameras may not see.

  • Damir ?olak

    I just returned the lens for the third time. Did anyone ever find FE 35 1.4 that was ok?

    All three had the right 1/3 of picture out of focus.

  • AaronClosz

    Not silly at all!

    Testing at 5 meters should not make much of a difference for a 35mm lens, as the focusing element would not likely have moved much from its infinity position. There would certainly be a big difference at 5 feet however. Combining near and far test results for lens variance is something we intend to do soon enough. As for your alternative, the variance may appear to be worse with the Sigma, but I think this has more to do with the shape of the curve. The score indicates that they are quite even in terms of variance.

  • François

    Hello !

    I just found this article while looking for info about copy variance for Sony EF lenses. I have a question. When buying a lens like the 35mm F1.4, it’s not really to use it for landscape photos. It’s mostly to take shots of the subject at a shorter distance. And mostly at big apertures. So corner sharpness is VERY rarely a problem, but sometimes it’s nice to be able to put the subject on the side of the frame. And using my Contax Zeiss 35mm F1.4, I’m finding it REALLY great for this kind of photo (people at 2 to 5 meters, as well as close up details), but less convincing for landscape photos. Do you think that doing your tests at 5 meters instead of infinite would make a difference ? I would be curious to know your opinion about that.

    And talking about variance, the other alternative I have for a bit aperture 35mm is the Sigma lens, now that an adapter will be available. But if I read your charts correctly, the Sigma is even worse for copy variance, am I right ?

    I hope the questions are not too silly ! Thanks in advance !

  • Taki

    Thanks, look forward to it.

  • AaronClosz

    Taki,

    We do have the capability to measure MTF at finite distances and hope to begin incorporating those results very soon.

  • Taki

    Hi Roger

    Apologies for not seeing the other 432 replies! Is it not possible to test macro lenses at their intended focus range? I’d be really interested to see what difference there would be in the results.

  • DrJon

    I haven’t seen anything more on these lenses (did I miss something?) so can I ask if the Panasonic 42.5mm issues were within the lenses or between lenses – so is simply getting one with 4 even corners all you need, or a choice of 18-odd examples, some lens testing machines and a spare week?

  • Roger Cicala

    Hakan, I meant that a copy is likely to have a soft side or a soft corner, that it’s not even across the frame.

  • Hakan

    Hello Roger,
    thanks for yet another compherensive study for us, gear freaks…
    I’d like to understand your statement about FE35 F1.4, ” a lot of the variance is WITHIN a copy, not just copy-to-copy” .. What do you mean by variance within a copy? Do you mean, with the same lens, some shots show better corner eveness but some shows worse…I think, this is the situation with my FE35…
    many thanks for clarification…

  • JohnDizzo

    The optical issues in the copy of the 35/1.4 I just received last week is the reason I even began googling for information on this topic which is what has led me here. I have been on the fence for a few days now about whether to send it back or to ask Sony to check it out. Based on what I have read here though, it appears to be an issue with the lot of them and I should just figure out whether or not softness in the areas of the copy I have are acceptable. lol.

    Anyhow, as usual, thank you guys for all the work/testing and insight.

  • Hope I’m wrong, but my speculation is there could be some unhappy Batis owners if and when LR tests the 25mm and 85mm … especially variance between lenses reflecting q/a.

  • Roger Cicala

    Taki, I’ll repeat for the 432 time: This test is done at infinity on an optical bench. The other test is done at much closer focusing distance with a marcro backlit chart on a camera. If they were the same you should be baffled.

  • Regarding the results of the 90mm macro, these results are even more baffling than the DxO discrepancy, when compared to your own previous test of the 90mm http://wordpress.lensrentals.com/2015/05/sony-fe-90mm-f2-8-g-oss-resolution-test
    I’m seriously baffled here???

  • Brandon

    Scott,

    The 42.5 is superb across the frame at f/4. The 17.5 improves less so but is “decent.”

    Brandon

  • scott kirkpatrick

    Brandon, thanks for the information about the superfast M43’s. Were the Voightlanders sharp as well as consistent once you got them to, say, f/2.8 or 4.0? Did you write this up and post it somewhere?

    scott

  • Roger Cicala

    Andrew, we tried, the extension tubes loosened around the solders with repetetive movements of the bench.

  • Andrew Dodd

    FYI,you probably could’ve manufactured your mount breakouts using extension tubes instead of cannibalizing cameras:

    http://www.dpreview.com/forums/post/56168165

  • Rob

    Roger, do you know if interlens variability and iffy construction applies to the 35/2 sonnar lens in the Sony RX1/R? I have noticed my copy to have a tilted focus plane.

    Rob

  • Brandon

    Scott,

    I finished three of M4/3 lenses over the summer. The panasonic 42.5/1.2 as well as the Voigtlander 17.5 and 42.5mm f/0.95 lenses. As far as consistency is concerned, the panasonic is an absolute trainwreck and the voigtlanders are excellent.

    Regards,
    Brandon

  • Xpanded

    Great article and great sense of humour – thanks Roger. I cannot quite decide if it will be of any help to you, but if you need peers to discuss you may want to contact Christian Nilsson at http://www.lensfreaks.com. He has been testing lenses on the Hasselblad Lab’s test benches for a decade or more. They are measuring real MTF-values and not just whatever a shelf piece of software (however good) says. He is a great guy too. Always helpful and friendly.