Lenses and Optics

Sony E Mount Lens Optical Bench Tests

Published October 20, 2015

It has taken a long time, the sacrifice of a couple of cameras and lenses, and a lot of soldering, but we have finally managed to rig up a powered Sony mount to our optical bench so that we can test the new full-frame Sony and Sony-Zeiss E-mount lenses.

For those of you who haven’t understood why there has been a problem, I’ll explain it for you. The optical bench tests lenses at infinity focus. Most lenses have a mechanical focusing ring. You set it to infinity, confirm it (because some lenses can actually focus past infinity, which screws up the test results), put them on the bench and test them. A few lenses are ‘focus-by-wire’; they are focused by the electric AF motor even when you turn the focusing ring manually. Most of these are no trouble either. You put them on the camera, focus to infinity, take them off and the focus stays where it last was.

Sony’s newer lenses (and some micro 4/3 lenses) are different. They are focused by an electromagnet. You can put them on the camera and focus to infinity, but when you take them off the camera the focusing group just falls back to wherever it wants to be, which is never infinity focus.

So in order to test Sony lenses on the optical bench we had to find a way to keep the lens hooked to the camera, but at the same time have it off the camera because it has to be mounted to the bench’s Sony test plate. Our solution isn’t particularly elegant, but it works. We took apart a random lens and camera to get a lens mount and camera mount. We modified one of the Optical Bench mounting plates so it contained a Sony camera mount. We then wired that to an empty lens mount. We place the test lens in the mounting plate (red line), put the empty lens mount into the camera, and we can control the lens focus by using the camera while the lens is on the optical bench.


Yeah, it looks crude. But this is version 3.0; the first two attempts were much cruder. Before you make fun remember we sacrificed an A7, an 18-200 lens, and a $900 optical test plate just to get you some numbers. OK, we really did it because it sounded fun and because nobody else in the Known Universe can test Sony lenses on an optical bench right now and we can. So there!

Remember This is Beta Testing

The goal and objective is to compare Sony lenses both to each other and to other E-mount lenses from Zeiss, Sigma, and whomever. But I know people are also going to compare them to Canon, Nikon, and other lenses. I want to emphasize that we’re learning as we go and I’m writing as we learn. So two months from now I may be writing one of those articles that tell you we’ve found an error in our technique or a better way to do it. I don’t think so, we’ve been experimenting for a while now, but that’s what happens when you do new things that nobody else is doing.

More importantly, there are some differences in the testing methods for the Sony-mount lenses compared to the Canon and Nikon mount tests we’ve been doing. We want to be as transparent as possible, so I’m going to identify all of those differences I can think of.

There is Glass in the Path

First, unlike most of the Canon and Nikon mount tests we’ve already done, we’ve added 2mm of optical glass to the testing path of these lenses. Why? Because we ran lots of trial tests and found that at least the first Sony lenses we tested performed much better with that amount of glass in the optical path. It made a big difference with the Sony lenses, while it did not make much difference for most of the Canon and Nikon lenses we tested. Why are Sony lenses more sensitive? My best guess, and it’s just a guess, is that their exit pupil is further to the rear than some of the other lenses. It’s also very possible that the shorter flange-to-sensor distance (18mm for Sony, 44mm for Canon) makes the lenses more sensitive to glass in the optical pathway. (For both of you who want to know we tested both 3 and 1mm and 2mm gave the best results.)

We ‘See’ Fewer Corners

With all of our other lenses the optical bench gives us a complete open circle. We test 4 quadrants and see results out to the edge of the frame (20mm of sensor distance) in all 4 directions. Our electronic Sony mount includes the rectangular light baffle that the Sony camera mount has. This means at certain rotations (top to bottom in camera terms) our view is cut off at roughly 15mm, the distance of the top or bottom of the sensor from the center. The result is our numbers for a Canon or Nikon lens measure each lens all the way to 8 edges (2 edges for each of 4 rotations). With the Sony lenses we only measure 6, the top and bottom are cut off. You can make an argument that this might make the variation appear slightly larger (there are few numbers in the mathematically average), or slightly smaller (we might not ‘see’ a bad corner) than it would be if we could measure all 4 corners. I think the only important point is that we should be conservative when making comparisons to Canon and Nikon tests. The tests are slightly different.

There May Be a Bigger Difference Between These Tests and Pictures You Take

We know that Sony ‘cooks the RAW’ at least a bit, doing some in-camera modifications to raw files. I don’t know exactly what or to what degree. They aren’t alone in this, it seems to be the wave of the future to do some in-camera correction for at least distortion. But it appears they are the only full-frame cameras to do in-camera RAW correction at this moment.

What that means, though, is that tests of just the lens without a camera body, like we’re doing here, may be quite a bit different than what comes out of the camera. For example, our tests of the FE 35mm f/1.4 ZA show a LOT more distortion than tests done on a camera body using Imatest or DxO optics; we show 4% distortion where most of the on-camera testing shows 1% or less. This probably means that the camera is processing the distortion out of the image. (It could also mean that the distortion is very different focused up close, where DxO and Imatest work, then at infinity, where the optical bench works.) Depending upon your point of view that may be good, bad, or make no difference to you at all.

The same thing may (or may not) apply to sharpening. If there’s a little sharpening going on in the raw image, then you may think “Roger you said that lens isn’t so sharp, but the RAW images look very sharp, and the report from this other site using Imatest says it’s very sharp, too”.  To take it a step further, sharpening may be applied more to the corners and edges, or vignetting correction applied, or other stuff. (Again, I don’t know if it is or not.)

Because so many people struggle with this idea, I’ll repeat what I say over and over: I’m testing the lens alone. (You would not believe how many people ask what camera these tests are done on.) Other sites are testing the image that has been processed by the camera. Since you’re using the camera, their results will look more like your results and you’ll not see much of the distortion in your images. About 90% of you are going to think ‘why would I care’, and you’d be right. What comes out of the camera is more important.

But if you’re a lens geek like me, what the lens is actually providing TO the sensor is important, if only for esoteric reasons. Or maybe not just esoteric reasons. If the lens is inferior, but the image looks better from in-camera processing, you may not be able to manipulate the RAW as much as you would like, because it’s already been manipulated once. A lot of people are very excited about Sony uncompressed raw because they will be able to do more intense post-processing. But uncompressed doesn’t mean unmodified. If the raw is still modified before you get it, post-processing may still be somewhat limited.

Of course, the vast majority of photographers could care less, but then they probably don’t read this blog 🙂

Some Test Results

We have lots of Sony and Sony mount lenses to test and it’s going to take a while to get them all done. Today I’ll present 3 of the lenses many people have wanted to see: the Sony FE 35mm f/1.4 ZA, the Sony FE 55mm f/1.8 ZA, and the Sony FE 90mm f/2.8 Macro OSS. We’ll show you the MTF curves and copy-to-copy variation graphs for each, comparing them to some of the Canon and Nikon mount lenses we’ve tested. Again, please remember that testing techniques are slightly different for the Sony lenses as explained above.

Sony FE 35mm f/.4 ZA

(Note: this post originally published too early with an incorrect MTF graph for the Sony FE 35mm f/1.4 ZA lens. It has been corrected as of 4:20 pm CST. I apologize for the error, I was trying to get the post up as I was leaving for New York.)

We’ll compare the Sony to several other 35mm lenses. The Sigma and Canon are amazingly good 35mm lenses. The Sony is better than the more average Nikon 35mm f/1.4.  It’s certainly not quite as good as either the Canon or the Sigma as far as resolution goes.


Roger Cicala and Aaron Closz, Olaf Optical Testing, 2015


When we look at copy-to-copy variation, something we’ve long thought was the case becomes documented.

Roger Cicala and Aaron Closz, Olaf Optical Testing, 2015


I’m going to avoid comparing the consistency numbers, because as we’ve mentioned the testing for the Sony lenses is a bit different. But the variation graphs show what our photographic experience with this lens has suggested. The FE 35mm f/1.4 ZA lenses are all over the place. It actually is a bit worse than the graphs look because a lot of the variance is WITHIN a copy, not just copy-to-copy. None of the 10 copies we tested had even corners. And I’ll editorialize and say that none of the dozens we’ve tested on Imatest had even corners either. If you use this lens for centered objects, you’ll be happy. If you want 4 sharp corners, it’s not likely to happen unless your standards for equal sharpness are pretty low.

Sony FE 55mm f/1.8 ZA

Going into these tests, I had expected the 55mm f/1.8 ZA to do better than the 35mm f/1.4 and it did. Compared to a sampling of other 50-something lenses, the Sony 55 does very well on the MTF charts. Its center resolution is superb and it maintains sharpness very well to the edges. The Nikkor 58mm and Zeiss Otus 55m lenses are being tested here at f/1.4, which gives the Sony lens a bit of an unfair advantage, but it’s still excellent and holding it’s own against the best lenses in this range at the very least.


Roger Cicala and Aaron Closz, Olaf Optical Testing, 2015


The copy-to-copy variation graphs for the 55mm lens show it has a lot better consistency than the 35mm, too. There’s some variation, but it’s similar to most of the other 50ish lenses we’ve tested. The Canon 50mm f/1.8 is very much the exception, being the only 50mm lens we’ve tested that has almost no copy-to-copy variation.

Roger Cicala and Aaron Closz, Olaf Optical Testing, 2015


Sony FE 90mm f/2.8 OSS Macro

The FE 90mm Macro shows a good, not great MTF chart. It’s certainly not bad, but not something to get overly excited about. Remember, though, that Macro lenses often perform better close up, where they are designed to work, than at infinity focus, where this test is done. If you’re using this as a long portrait or short telephoto lens, these results should reflect what you see (other than if there’s sharpening done in RAW). At macro distances, it may perform differently.


Roger Cicala and Aaron Closz, Olaf Optical Testing, 2015

Things aren’t quite so good when we look at copy-to-copy variation, though. Medium focal length macro lenses are usually very consistent. In this case, the Sony has a lot of variation compared to the others. The type of variation we’re seeing is also quite different from the Sony FE 35mm f/1.4. That lens has a lot of corner variation in every copy, but the center remains is consistently sharp comparing one lens to another. With the Macro, there’s a lot of center variation and it’s not particularly worse in the corners. This suggests there is significant copy-to-copy variation in overall sharpness, rather than individual lenses have a bad corner.


Roger Cicala and Aaron Closz, Olaf Optical Testing, 2015


So What Can We Take Away Today?

Mostly we are just starting with FE lens testing. Over the next few weeks, we should get a good handle on how the Sony lenses compare to each other.

I think there’s no question we’ve seen support for what a lot of people are claiming: the 35mm f/1.4 ZA lens is very likely to have some tilt with one corner or side out of the plane of focus. Depending on what you photograph this may make no difference to you, or may be a huge issue. But I doubt sending copy after copy back is going to make much difference (you may trade a soft right upper corner for a slightly soft left side, etc.). We’ve taken a couple of these apart and there’s not a ton of optical adjustments that can be made: there are a set of shims behind the front group that can be modified a bit, but it’s a crude and very time-consuming adjustment, so I don’t think it’s generally going to be something that can be fixed.

The 55mm f/1.8 is really a good lens, very sharp and consistently made. The 90 mm f/2.8 is a decent lens, reasonably sharp, but there seems to be a fair bit of copy-to-copy variation in overall sharpness.

All that being said, these lenses may look better on-camera than on the optical bench because there seems to be at least some modification of files as the raw is written. We’ll get a better handle on that as we test more lenses. And let me emphasize, the only clear evidence we see for that is with distortion correction. There may be nothing more than that going on.


Roger Cicala and Aaron Closz


October, 2015

Author: Roger Cicala

I’m Roger and I am the founder of Lensrentals.com. Hailed as one of the optic nerds here, I enjoy shooting collimated light through 30X microscope objectives in my spare time. When I do take real pictures I like using something different: a Medium format, or Pentax K1, or a Sony RX1R.

Posted in Lenses and Optics
  • Raoul

    Hi Roger and Aaron
    Interesting test, once again.
    I was just looking at your variation figures:
    It looks to me like it is giving more importance to the higher curves (Black and red). Am I right ?
    When you look at the Canon and Nikon 50mm, there is not much difference except in the higher curves… But the figures are very different.
    Is is meant so or is it a side effect ?

    Have a good day

  • Brian

    You want decentering? Try testing the new Batis line of lenses. The 25mm and 85mm both have centering issues.

  • Neil Holmes

    Did you test the Canon and Sigma 35s using an adapter on the same rig?

    If not that would be interesting to see if the results are different.

    Thanks for the tests.

  • Roger Cicala

    D. Garlans, because we had a banged up one available that was too rough to rent.

  • Roger Cicala

    Tuco, you may be right on Leica, and some do let you set for in-camera lens correction.

  • tuco

    You said Sony is the only FF tweaking the RAW with in-camera lens corrections. I can’t say for sure but doesn’t the Leica M9 also do the same? I mean, if you put some 3rd party lenses on a M9 and don’t manually set the camera to a good Leica version of the focal length you can get some terrible results like color and vignetting. It feels like the camera is making some corrections here too.

  • It’s “couldn’t care less”, Roger!

    See this instructional video from the Queen (by way of David Mitchell):


  • Brandon


    Several manufactures already cook distortion correction into the raws, see e.g. Photozone’s findings in the micro 4/3 realm.

    Distortion is caused by an asymmetry of lens power about the aperture stop. Internal focusing lenses do have their distortion change as they focus, as moving an element changes the power of that element and the one behind it, but it should never be as extreme as 4x. Even 2x would be extreme except for a macro lens, as macro lenses focus down to one of the special cases of image and object distance relationships.

    Something this wide would also always test to have more distortion at near distances, as rectilinear distortion would be recorded by imatest or other software as distortion, when really the center of the chart is just appreciably closer than the corners.


  • obican

    Is there a way we can mount the 90mm on a Sony camera and run an IMA test but without connecting it to the camera electronicaly? That way we could find out whether something is done to RAW after recognizing the lens or not.

    Btw, I’ve noticed that A7’s Auto WB was consistently off when testing a Canon 70-200/2.8L IS via a Commlite adapter and Minolta 70-210/4 via LA-EA4. Camera would always dial some more magenta tint to Minolta.

  • L.P.O.

    thanks again for superb research. Even though I don’t own Sony equipment, nor do I plan to at the moment, it is good to be in the know.

    A few words regarding cooking of RAW, though. While I believe things like vignetting correction to be very possible to cook into RAW images, I find it a bit hard to believe that they would really do distortion correction. That’d be a pretty intensive (not to speak of lossy) operation with RGBG mosaic data. I don’t say it isn’t happening, but the Occam’s razor principle makes me find it much more likely that the lens distortion characteristics just change with focusing distance.

    Have a nice Back to the Future day!

  • D. Garlans

    Just out of curiosity, why did you sacrifice an a7 instead of something dirt cheap like an nex3 or c3 of f3? The E mount can still control FE lenses I believe, so it probably would have worked. Or are the a7’s easier to get into?

  • obican

    Let’s see. Maybe the 90 and 35 scoring so badly is due to fact they were optimised for closer distances and maybe, again just maybe, the camera can make a pretty good image with those lenses due to microlenses and software and such.

    Could those microlenses and software and such have an effect on lens variance too? Maybe they’re working amazingly efficient at correcting the lens to bring it on the same level as top lenses in their class, so that they can fix a lot of that invariance too. Could this be?

    Naah, still doesn’t explain why people are complaining about not getting a good copy on the first three tries. 🙂

  • Roger Cicala

    David, we plan on doing that complete kind of testing. It took several hundred hours to get just the Canon primes all done, and the Sony lenses are going to take a while, but I hope to have a full database by the end of the year.

  • Thanks for starting these tests, which are very interesting to me.

    One request, keeping in mind that you are beta testing:

    I understand that lots of people prioritize wide-open lens performance, but obviously that’s not always the best indicator of overall lens sharpness or quality.

    For instance, in some online tests, the Sony 90mm FE macro is just competent wide open, but blazingly sharp stopped down.

    It would be great to see some of the Sony and Sony-Zeiss lenses tested at their optimum apertures in the future, for those of us who use lenses that way.


  • Roger Cicala

    Adrian, you have just figured out what version 1.0 was. Didn’t work out with the solders very well.

  • Your tests and observations are always interesting and illuminating for me. None of what you have found in testing these lenses seems unreasonable. Sony always brings out the fear of the dark side in me now. I was a devoted acolyte for many years and the disillusion came hard. I look forward to your updates.

  • A

    Wouldn’t it have been cheaper and easier to hack an extension tube for the Sony mounts?

  • Many thanks for this. The decentering of Sony lenses was something that had been reasonably proved by many, but your bench and the capability of testing batches of lenses gives us a robust data set. Thanks also for the consideration about the scarce margins for improvement… for some time I guessed whether it would make sense to ask for the lab to tweak my lenses and reduce the decentering, but now I suppose it wouldn’t bring much benefits. Longing for you to test the Sony lenses I own.

  • Roger Cicala

    Maya, I’m not sure about it. I can see situations where it might. I’m totally speculating but lets say distortion correction, localized sharpening and vignetting correction take place in all 4 corners, but the lens is tilted so one corner is out of focus. It may be that the other corners improve a lot but the out of focus not at all, making the difference seem greater, not smaller.

  • Roger Cicala

    David, that’s a really good point and one I didn’t make for the 35mm like I did for the 90mm. This is what makes comparing this kind of testing with DxO or Imatest interesting. Is it the focusing distance? Something the camera is doing? Something else? It fascinates me, trying to figure out what is actually going on.

  • Roger Cicala

    Obican, it may well be at Macro distances, and it may be that camera-lens combination is better (sensor microlenses and other stuff can play a part). I can just say it isn’t at infinity and without a camera.

  • Roger Cicala

    CarVac, we didn’t, we need glass slightly larger than sensor size. The best we can do is try various thicknesses of optical glass and see the effect. In this case it was pretty clear that 2mm of glass was very close to correct. It’s intersting, with Canon and Nikon lenses we didn’t see much difference in the first half-dozen lenses we did so we left the glass out (testing is quicker and easier without it), but looking back on things I wonder with a couple of lenses if it might make a difference. We’re doing some retesting to make sure with them. But with FE 2mm is clearly needed.

  • MayaTlab

    Haha, I can’t handle that level of geekiness (emphasised by the most appropriate reverential capitalisation). But dare I suggest that walking is SO 4 ABY ? Now you’re hip if you ROLL.

    On a more serious note, would the raw cooking you think is applied affect sample variation as well as average performances ?

  • David Braddon-Mitchell

    Variation in these lenses is a real worry; my first 90 was badly skewed: the second fairly symmetrical. I got a pretty good 35 1.4 that was a little worse on the bottom right, but kept it thinking that my odds of a better one were low.

    I’ve never used the 90 at infinity, but one thing about he 35 1.4 is that it seems significantly worse at infinity than at closer distances (almost like the earlier Sigma 50 1.4 in this respect). Perhaps it’s optimised for environmental portrait distances, which kind of makes sense for a 35 1.4 (not the sort of lens I’d use for most infinity purposes except astro)

  • obican

    Oh, expect some angry comments here because everyone has been saying that Sony 90mm Macro is better than Canon’s IS L since some website claimed so.

  • stu

    I’m so glad you guys are doing this because the greater amount of scrutiny could be an incentive for Sony to improve the consistency of its lenses through greater quality control. Indeed, it seems like decentering is an issue that is frequently mentioned for Sony lenses. It is somewhat scary to spend money on a product that performs underwhelmingly due to poor quality control and I have pushed back buying some Sony lenses due to this alone. I really don’t want to have to go through a few copies to finally get a good one…

  • CarVac

    Just curious, could you take an AA filter stripped from a dead camera and see the effect it has on delivered MTF versus a plain piece of glass?

  • Wally

    Thanks for going to the trouble of setting up for FE lenses. If I remember correctly the 90mm f/2.8 had pretty good numbers in the earlier Imatest results, perhaps infinity is to far for this macro lens. The tests also back what I hear on the on-line forums about various lenses, there is just too much variation among the new Sony lenses, some people get great lenses and some dogs.

  • Roger Cicala

    Maya, I think if we could make it walk around, it wouldn’t look out of place in The Trailer. I think the cover of the bench is Duraplast and there’s certainly some transparasteel in there.

  • MayaTlab

    Thanks a lot ! Perhaps it’s just because today is trailer day, but this bench looks like it was built with a hydrospanner.

Follow on Feedly