fbpx
Geek Articles

Testing Lenses: Stopped-Down MTF Curves

For quite a while now we’ve been publishing our MTF curves. They’ve always been the same thing; at least ten copies of each lens, in 4 rotations, and measured at the widest aperture. The reasons for that are (to us) very straightforward. Our purpose in doing the tests is to establish reasonable expectations, learn what type of optical adjustments are possible, and figure out how much copy-to-copy variation to expect. We often publish the results so you can see them, too.

There are other things that photographers and videographers would like to see us do; field curvature, stopped-down MTFs, high-frequency testing, etc. We’re in kind of a catch-22. The thing we’ve considered most important to show; variation between copies; requires we test at least ten copies of each lens before publishing results. Testing 10 copies mean we just don’t have time to do those other tests.

We’re rewriting our proprietary software and exploring some interesting ways to present data. While we’re doing that, I thought we’d revisit our ’10-copy’ rule and consider presenting some more data for you all to look over. One of those things is stopped down MTF. We don’t have time to test 10 copies at every aperture stop. But we can test 10 copies wide-open, pick out one average copy, and test that. It’s not full 10-copy data, but at least that copy will be representative of the norm.

Today, I’m going to present data from Canon, Sigma, Rokinon, and Zeiss Cinema primes. Why? Because those are the same as the photo primes optically, so what happens with the Cine lenses is representative of what will happen with photo lenses. We get a two-for-one special. (The one exception I’m aware of is that the Canon CN-E 35mm T1.5 lens is optically the same as the Canon 35mm f/1.4L original version, not the Mk II version.)

I’m not going to try every aperture for these lenses, but I will compare them wide-open and at T4. In the future, we’ll probably add T5.6 and perhaps T8 data (I’m open to suggestions) for most lenses. But that will take time, and this is what I have right now. And this is meant as an introduction to some of the other things we’re testing that we haven’t shown yet.

T4 MTF Curves

I’ll just go down the list, by focal length. For each lens, we’ll show an average copy wide open on the left and stopped down to T4 on the right. Remember the starting apertures are different. We would expect the jump from T1.5 to T4 to be more impressive than that from T2.1 to T4.

85mm

Yes, everything is sharper at T4. But the 85mm graphs, I think, may surprise some of you. They did me, at least a bit.

The Canon and Xeen are basically soft-focus lenses wide open, and jump to quite good and certainly adequate, respectively, at T4. The Sigma is already sharp wide open and just breathtakingly sharp at T4. The two Zeiss are nearly identical at T4 (as they should be, they’re very similar lenses). From a resolution standpoint, though, at T4 neither is nearly as sharp as the Sigma, nor quite as sharp as the Canon. (That last part surprised me a bit.)

Canon CN-E 85mm T1.3

Olaf Optical Testing, 2017

Sigma Cine 85mm FF T1.5 

Olaf Optical Testing, 2017

 

Rokinon Xeen 85mm T1.5

Olaf Optical Testing, 2017

Zeiss CP.2 85mm Super Speed T1.5

Olaf Optical Testing, 2017

Zeiss CP.2 85mm T2.1

Olaf Optical Testing, 2017

50mm

At 50mm things are slightly different. The Sigma 50mm is again the sharpest of the wide-aperture lenses wide open, and the sharpest stopped down. The Canon and Zeiss lenses all sharpen up a lot in the center (especially the Zeiss) but still stay somewhat soft in the outer half of the image. The Xeen was the most significant surprise here, going from the lowest MTF wide open to being quite good at T4. In the outer half of the image, it’s the second best lens at T4.

Canon CN-E 50mm T1.3

Olaf Optical Testing, 2017

Sigma Cine 50mm FF T1.5

Olaf Optical Testing, 2017

Rokinon Xeen 50mm T1.5

Olaf Optical Testing, 2017

Zeiss CP.2 50mm Super Speed T1.5

Olaf Optical Testing, 2017

Zeiss CP.2 50mm T2.1

Olaf Optical Testing, 2017

Looking at these 50mm MTFs my first question is whether the Zeiss lenses really get worse off axis when the aperture is stopped down. That makes no sense. So what is happening? For now, I’ll throw out a couple of possibilities, because the next post in this series will get into this more, and also show you some useful new tools. Things to consider are focus shift and field curvature.

The takeaway messages are in two parts. First, standard MTF curves don’t show everything, and occasionally can be a bit misleading. Second, if we delve into them a little more in-depth (which we will in the next few articles) they actually give a lot more information than the standard curves show. Much of that is useful information.

35mm

There’s a bit more uniformity at 35mm with all of the lenses sharpening up nicely at T4. The Zeiss lenses don’t get quite as sharp in the center but have smoother performance away from the center. The Xeen has a lot of astigmatism-like separation of sagittal and tangential lines away from center even stopped down. That can actually be astigmatism, or it could be lateral color. Which is it? That’s another thing we can look at with more detailed testing in the next post.

Canon CN-E 35mm T1.5

Olaf Optical Testing, 2017

Sigma Cine 35mm FF T1.5

Olaf Optical Testing, 2017

Rokinon Xeen 35mm T1.5

Olaf Optical Testing, 2017

Zeiss CP.2 35mm T2.1

Olaf Optical Testing, 2017

 

Wider Lenses

There’s nothing spectacular going on here. Some of the patterns are the same. The Xeen, like the other Xeens, is barely usable wide open but is not only quite good at T4; it’s probably the best off-axis. The old-designed Zeiss 21mm looks terrific at T2.9 and even better at T4. The Canon serves as an excellent demonstration that testing a single lens sometimes results in jerkier graphs than our usual averages of multiple copies. The Sigma raises a similar question to what the Zeiss 50mm lenses did. Is that weakness off-axis at T4 field curvature or is the lens just not sharpening up because its aberrations are not very aperture sensitive?

Canon CN-E 24mm T1.5

Olaf Optical Testing, 2017

 

Sigma Cine 24mm FF T1.5

Olaf Optical Testing, 2017

 

Rokinon Xeen 24mm T1.5

Olaf Optical Testing, 2017

 

Zeiss CP.2 28mm T2.1

Olaf Optical Testing, 2017

 

Zeiss CP.2 21mm T2.9

Olaf Optical Testing, 2017

 

So What Did We Accomplish Today?

Well, I think it’s evident that the idea of testing multiple copies wide open and then doing more extensive tests on a ‘typical’ copy is worthwhile. As we all knew, stopping down improves all of the lenses, but it doesn’t do so equally. I’d think we should have added T2.9 comparisons, too. The Xeens, in particular, have changed so dramatically at T4 that I want to know how they look at T2.9.

Whether we test at T5.6 is less obvious to me, and time is money. I can see it being worthwhile for the T2.9 lenses to at least check the ‘two stops from wide-open is the sweet spot’ theory we all assume is correct. Honestly, I see no reason to test at smaller apertures than that. T8 may be a bit better for some lenses, but diffraction softening is going to be kicking in there for higher resolution cameras. I’m open to having my mind changed, but I need better arguments than ‘just cause.’

We also, as usually happens when we do more testing, added more questions. Why are some lenses not improving in some areas? What else can you tell me about a given lens that is useful and not shown by MTF curves? I’ve talked at length about how much more information we get when we do MTF field curvatures (they’re officially called MTF vs. Field vs. Focus). That’s too time-consuming a test to do at multiple rotations on multiple copies, but if we go with our ‘in-depth testing of a typical copy’ plan, it becomes reasonable and gives us a lot more information.

That’s what I’ll show you in the next post.

 

Roger Cicala, Aaron Closz, with invaluable input and programming from Brandon Dube and Max Bruggerman

Lensrentals.com

November, 2017

Author: Roger Cicala

I’m Roger and I am the founder of Lensrentals.com. Hailed as one of the optic nerds here, I enjoy shooting collimated light through 30X microscope objectives in my spare time. When I do take real pictures I like using something different: a Medium format, or Pentax K1, or a Sony RX1R.

Posted in Geek Articles
  • The data isn’t available yet, except for these lenses. We’ve always tested 10 copies wide open.
    The rule of thumb for your general question, though is in the center 50% of the field, a zoom can hold it’s own with a prime. Outside of that, not really.

  • l_d_allan

    Thanks. Very interesting.

    From Feb 10, 2017 blog article “Things You Didn’t Want to Know About Zoom Lenses” >>
    “Can they [zooms] still be very good? Absolutely. Can they be as good as the best primes? Nope.” …
    … “A great zoom is not as good as a good prime at comparable apertures, but it’s plenty good, especially in the center of the image.”

    Would it be feasible to show some selected side-by-side charts of premium primes vs premium fast f/2.8 zooms at the same focal length to further illustrate the points from the Feb 10, 2017 blog article?

    Such as using existing LR data points at f/4 for Sigma 85mm vs 70-200mm f/2.8 zooms from Canon / Nikon / Sony?

    RC >> I’d think we should have added T2.9 comparisons, too.
    Agree.
    And if the data is already available, what about comparisons at f/2.8? That’s an issue I grapple with … how much IQ am I trading off for the flexibility of a zoom? Such as using my Sony FE 28 f/2 or Sony FE 55 f/1.8 compared to my Canon 24-70mm f/2.8 ii adapted to Sony a7Rii?

  • Brandon Dube

    That’s… not what that means. If it was so popular (like, say, 70-200s), Lensrentals would own hundreds to thousands of lenses of each model and getting ten for testing would be completely painless.

  • Brandon Dube

    f/11 is… doable on the MTF bench, f/22 can’t be done because of limited light.

    Unless you have a spare 5+ kilowatt fiber coupled light source you want to donate 🙂

  • Mike Earussi

    Thanks Roger, that T4 info really helps in evaluating which lens to use. Most good lenses peak out between f2.8 to f5.6, so f8 would be unnecessary especially given the new high resolution sensors (and lenses) being diffraction limited by f8. And one average sample is sufficient to determine the overall characteristics of them all.

    There are obviously many other characteristics of a lens that are important, but how easy (or scientific) it would be for you to test them is unknown (to me at least). That’s why it’s important to use more than one lens test site to evaluate a given lens. For instance on the Sigma 85 f1.4 A another reviewer on YouTube demonstrates its problem with longitudinal CA, something I’m not sure your test could show:
    start at 7:30 https://www.youtube.com/watch?v=H1Rp0JHKPfg&t=221s
    also at 7:30 https://www.youtube.com/watch?v=DLllInVFsSM&t=953s

  • David Bateman

    Ha, I knew it. M43rds so popular you never have 10 in stock for testing. Of course you may only have 10, or 6 lenses. But that was not the point.
    All the best!

  • hywelphillips

    Hi Roger as always thank you for the monstrous amount of work that goes into presenting lens results as simple, clear curves.

    For me at least it would be useful and interesting to see a “complete run” for a small selection of lenses through the whole of their aperture range. Not as a part of regular testing, but just to get a feel for “typical” lens behaviour, if there is such a thing. Wide open, f/2.8, f/4, f/5.6, f/8, f/11, f/16, f/22, on a handful of lenses at common focal lengths, on an average sample lens.

    Because one thing I took from today’s graphs is that the behaviour of these lenses is more similar than different in terms of the change in behaviour as one stops down. So doing a “complete” profile of a couple of lenses through the whole aperture range would be useful in getting a feeling for how lenses behave generally.

    Folk wisdom for full frame lenses is “Everything is soft wide open, especially off axis. Stop down by two stops and you’ve probably hit the sweet spot. So that’s mega-blur at f/1.4, shallow but decent for general use at f/2.8, people are good at f/4, landscapes are good at f/8, and if you go much further diffraction will start to wreck everything.”

    Does that actually work? How much worse are f/11, f/16 and f/22 on a couple of “typical” lenses? It would be great to know! Not as a part of regular testing or lens characterisation or “is the Sigma better than the Canon”, but to get a feel for the general behaviour.

  • What Brandon meant was ‘the rental demand is smaller, and we rarely have enough copies in-house to do a 10 test set of m4/3 or Pentax’. And with Pentax we don’t have a mount for Pentax lenses on the bench; mounts cost around $1,000 each.

    More important both m4/3 and Pentax have some electromagnetic focus lenses. That means we don’t just need a mount, we have to wire the mount to a camera, which increases the cost significantly.

    We have more lenses to test than we do time and as we do more consulting work time decreases further.

  • David Bateman

    That makes sense. On the mtf curves.
    Thanks!

  • Brandon Dube

    When you stop down, you do a lot to get rid of spherical aberration and coma. Astigmatism is reduced, but it takes quite a bit of stopping down for the depth of field to cover it up. Lateral color is unchanged.

    When you stop down, the “general haze” from spherical and coma goes away, and the sagittal orientation is unaffected by lateral color. This allows S to improve more than T, and if the lens is astigmatic a focus shift from spherical aberration could make the astigmatism quite striking.

  • Brandon Dube

    The bad MTF at ~14mm is just field curvature / astigmatism. It’s covered up by spherical aberration and coma at wide apertures, and when you stop down and the focus shifts, that region can go very out of focus.

    The focus shift for a lot of lenses is ~0.1mm. If you pull out your thin lens equations for a hypothetical object at a distance of, say, 5m, see where the object would be if you move the image by 0.1mm.

    It’s not related to sample variation.

    There is not enough market share in M4/3 to motivate testing. Same as Pentax or APS-C.

  • David Bateman

    Thank you for the hard work. The zeiss and sigma oddities make me wonder if you see that if you should just grab an other of those lenses and see if you see the same at that f stop. Could be a sample variation thing in that lenses and not represented of the brand.

    Or instead of all this hard work which will open up more questions and tests, you could just provide data for 43rds lenses. But maybe they are so popular that there are never 10 around for testing?

  • Yugo Nakai

    This is so exciting! I’m waiting with bated breath for the followup posts so I can learn more.

  • “Roger Cicala hates Nikon, uses Cine as excuse to exclude from test”

    That’s what the headlines will read… :-

  • It’s surprising to me that astigmatism (the difference between the sagittal and tangential lines) actually gets worse in many cases when stopping down. I would have expected that most of the aberrations would reduce when stopping down. Can you comment on this?

  • Photoretouchpro

    Maybe you could take one lens that is middle of the pack and do a stop down test at full stops for that lens when you do the 10 copies. This may give some idea of how the lens performs stopped down. Thank you for your tests!

  • Lee

    Andre is right, it gets confusing. As far as I’ve been able to figure out Zeiss made both a 50 T2.1 CP.2 Macro based on the 50/2MP and a “regular” 50 T2.1 CP.2 which was the 50/1.4 with the limited aperture.

  • That’s correct and all these are full-frame coverage.

  • Tim, setting up a lens at a given focal length takes about 5 minutes, the actual run of 4 rotation MTFs that we usually do between 10 minutes (wide angle) and 4 minutes (telephoto). Zooms, of course, have to be done at 3 focal lengths. After the actual tests results have to be checked, cleaned up occasionally (there may be a bad measurement from vibration, etc.), if variance is high we may need to do 15 instead of 10 copies, and then run through software.

    Doing a set of ten 35mm prime lenses is about 4 hours work. Doing a 24-70mm zoom is a day and a half, maybe 2 days.

    You can see why I can’t do ten copies at 6 different aperture settings, but taking one ‘median’ copy and spending a half day doing further testing on just that copy is something I can consider.

  • Lee

    Hopefully the CP.3 will also use the new 18 because the old one had to be the weakest ZF.2/CP.2. If they were smart they would’ve waited and used the new 35/1.4 too but based on the timings of the releases I doubt they did.

  • Lee

    You’re right, in this article all the tested lenses are rehoused FF35 photo lenses

  • Tim Cooper

    This makes me wonder, how long does the actual test process take, vs. setup time?

  • Someone

    Super 35 has max image hight about 15 mm. Some of the cinema lenses may be optimized for quality in this area, not the full photographic 24×36 mm. However, all the presented lenses are adapted photographic lenses if i am not mistaken.

  • Arthur Meursault

    Great testing and worthwhile for real world use. I have the Otus lenses and despite being good performers wide open, I more frequently use them at f 4.0. Wide open test results are interesting but in the real world, I think more photos are a couple of stops up.

  • That is my understanding. I think the CP.3 will have the Milvus 85 and 50mm designs. For the other focal lengths Milvus and ZE/ZF are the same.

  • Thanks Roger! Based on the results, those 85s are probably the Classic rather than the Milvus design?

  • Athanasius Kirchner

    This is great, Roger! Thanks. it goes to show that stopping down isn’t the panacea it’s made out to be.

  • The Zeiss 85s are, if I understand correctly, both based on the Zeiss 85mm f/1.4 lens. The 50mm T1.5 and T2.1 are based on the Zeiss 50mm f/1.4 and f/2 lenses.

  • Roger & team, thank for publishing these tests! Some of the results are indeed surprising.

    For the Zeiss lenses, which photography lenses do they map to? Zeiss has so many versions of their lenses now, and they don’t have an equivalent 85mm/1.8 or whatever the T2.1 lens maps to.

  • Tianhang

    Thanks for finally taking such test.
    For portrait lenses, wide open and one stop down MTF test is useful for most people. And it would be great to test at F8 for landscape lenses.

Follow on Feedly