Geek Articles

Testing Lenses: Stopped-Down MTF Curves

For quite a while now we’ve been publishing our MTF curves. They’ve always been the same thing; at least ten copies of each lens, in 4 rotations, and measured at the widest aperture. The reasons for that are (to us) very straightforward. Our purpose in doing the tests is to establish reasonable expectations, learn what type of optical adjustments are possible, and figure out how much copy-to-copy variation to expect. We often publish the results so you can see them, too.

There are other things that photographers and videographers would like to see us do; field curvature, stopped-down MTFs, high-frequency testing, etc. We’re in kind of a catch-22. The thing we’ve considered most important to show; variation between copies; requires we test at least ten copies of each lens before publishing results. Testing 10 copies mean we just don’t have time to do those other tests.

We’re rewriting our proprietary software and exploring some interesting ways to present data. While we’re doing that, I thought we’d revisit our ’10-copy’ rule and consider presenting some more data for you all to look over. One of those things is stopped down MTF. We don’t have time to test 10 copies at every aperture stop. But we can test 10 copies wide-open, pick out one average copy, and test that. It’s not full 10-copy data, but at least that copy will be representative of the norm.

Today, I’m going to present data from Canon, Sigma, Rokinon, and Zeiss Cinema primes. Why? Because those are the same as the photo primes optically, so what happens with the Cine lenses is representative of what will happen with photo lenses. We get a two-for-one special. (The one exception I’m aware of is that the Canon CN-E 35mm T1.5 lens is optically the same as the Canon 35mm f/1.4L original version, not the Mk II version.)

I’m not going to try every aperture for these lenses, but I will compare them wide-open and at T4. In the future, we’ll probably add T5.6 and perhaps T8 data (I’m open to suggestions) for most lenses. But that will take time, and this is what I have right now. And this is meant as an introduction to some of the other things we’re testing that we haven’t shown yet.

T4 MTF Curves

I’ll just go down the list, by focal length. For each lens, we’ll show an average copy wide open on the left and stopped down to T4 on the right. Remember the starting apertures are different. We would expect the jump from T1.5 to T4 to be more impressive than that from T2.1 to T4.

85mm

Yes, everything is sharper at T4. But the 85mm graphs, I think, may surprise some of you. They did me, at least a bit.

The Canon and Xeen are basically soft-focus lenses wide open, and jump to quite good and certainly adequate, respectively, at T4. The Sigma is already sharp wide open and just breathtakingly sharp at T4. The two Zeiss are nearly identical at T4 (as they should be, they’re very similar lenses). From a resolution standpoint, though, at T4 neither is nearly as sharp as the Sigma, nor quite as sharp as the Canon. (That last part surprised me a bit.)

Canon CN-E 85mm T1.3

Olaf Optical Testing, 2017

Sigma Cine 85mm FF T1.5 

Olaf Optical Testing, 2017

 

Rokinon Xeen 85mm T1.5

Olaf Optical Testing, 2017

Zeiss CP.2 85mm Super Speed T1.5

Olaf Optical Testing, 2017

Zeiss CP.2 85mm T2.1

Olaf Optical Testing, 2017

50mm

At 50mm things are slightly different. The Sigma 50mm is again the sharpest of the wide-aperture lenses wide open, and the sharpest stopped down. The Canon and Zeiss lenses all sharpen up a lot in the center (especially the Zeiss) but still stay somewhat soft in the outer half of the image. The Xeen was the most significant surprise here, going from the lowest MTF wide open to being quite good at T4. In the outer half of the image, it’s the second best lens at T4.

Canon CN-E 50mm T1.3

Olaf Optical Testing, 2017

Sigma Cine 50mm FF T1.5

Olaf Optical Testing, 2017

Rokinon Xeen 50mm T1.5

Olaf Optical Testing, 2017

Zeiss CP.2 50mm Super Speed T1.5

Olaf Optical Testing, 2017

Zeiss CP.2 50mm T2.1

Olaf Optical Testing, 2017

Looking at these 50mm MTFs my first question is whether the Zeiss lenses really get worse off axis when the aperture is stopped down. That makes no sense. So what is happening? For now, I’ll throw out a couple of possibilities, because the next post in this series will get into this more, and also show you some useful new tools. Things to consider are focus shift and field curvature.

The takeaway messages are in two parts. First, standard MTF curves don’t show everything, and occasionally can be a bit misleading. Second, if we delve into them a little more in-depth (which we will in the next few articles) they actually give a lot more information than the standard curves show. Much of that is useful information.

35mm

There’s a bit more uniformity at 35mm with all of the lenses sharpening up nicely at T4. The Zeiss lenses don’t get quite as sharp in the center but have smoother performance away from the center. The Xeen has a lot of astigmatism-like separation of sagittal and tangential lines away from center even stopped down. That can actually be astigmatism, or it could be lateral color. Which is it? That’s another thing we can look at with more detailed testing in the next post.

Canon CN-E 35mm T1.5

Olaf Optical Testing, 2017

Sigma Cine 35mm FF T1.5

Olaf Optical Testing, 2017

Rokinon Xeen 35mm T1.5

Olaf Optical Testing, 2017

Zeiss CP.2 35mm T2.1

Olaf Optical Testing, 2017

 

Wider Lenses

There’s nothing spectacular going on here. Some of the patterns are the same. The Xeen, like the other Xeens, is barely usable wide open but is not only quite good at T4; it’s probably the best off-axis. The old-designed Zeiss 21mm looks terrific at T2.9 and even better at T4. The Canon serves as an excellent demonstration that testing a single lens sometimes results in jerkier graphs than our usual averages of multiple copies. The Sigma raises a similar question to what the Zeiss 50mm lenses did. Is that weakness off-axis at T4 field curvature or is the lens just not sharpening up because its aberrations are not very aperture sensitive?

Canon CN-E 24mm T1.5

Olaf Optical Testing, 2017

 

Sigma Cine 24mm FF T1.5

Olaf Optical Testing, 2017

 

Rokinon Xeen 24mm T1.5

Olaf Optical Testing, 2017

 

Zeiss CP.2 28mm T2.1

Olaf Optical Testing, 2017

 

Zeiss CP.2 21mm T2.9

Olaf Optical Testing, 2017

 

So What Did We Accomplish Today?

Well, I think it’s evident that the idea of testing multiple copies wide open and then doing more extensive tests on a ‘typical’ copy is worthwhile. As we all knew, stopping down improves all of the lenses, but it doesn’t do so equally. I’d think we should have added T2.9 comparisons, too. The Xeens, in particular, have changed so dramatically at T4 that I want to know how they look at T2.9.

Whether we test at T5.6 is less obvious to me, and time is money. I can see it being worthwhile for the T2.9 lenses to at least check the ‘two stops from wide-open is the sweet spot’ theory we all assume is correct. Honestly, I see no reason to test at smaller apertures than that. T8 may be a bit better for some lenses, but diffraction softening is going to be kicking in there for higher resolution cameras. I’m open to having my mind changed, but I need better arguments than ‘just cause.’

We also, as usually happens when we do more testing, added more questions. Why are some lenses not improving in some areas? What else can you tell me about a given lens that is useful and not shown by MTF curves? I’ve talked at length about how much more information we get when we do MTF field curvatures (they’re officially called MTF vs. Field vs. Focus). That’s too time-consuming a test to do at multiple rotations on multiple copies, but if we go with our ‘in-depth testing of a typical copy’ plan, it becomes reasonable and gives us a lot more information.

That’s what I’ll show you in the next post.

 

Roger Cicala, Aaron Closz, with invaluable input and programming from Brandon Dube and Max Bruggerman

Lensrentals.com

November, 2017

Author: Roger Cicala

I’m Roger and I am the founder of Lensrentals.com. Hailed as one of the optic nerds here, I enjoy shooting collimated light through 30X microscope objectives in my spare time. When I do take real pictures I like using something different: a Medium format, or Pentax K1, or a Sony RX1R.

Posted in Geek Articles
  • Brandon Dube

    I do not think we disagree.

    Some aberrations are drastically more offensive to the human eye than others; in state of the art designs from yesteryear, it is much less likely that the aberrations present in the design are there by any intention of the designer; that these older lenses are often preferred I think has far more to do with nostalgia than any artistic merit.

    Objectively, super 8mm footage is pretty crap. Yet there is a large renewed interest in shooting it, because the format’s flaws are evocative of people’s childhoods and have a happy association. This has nothing to do with any “objective” measure of beauty.

  • Jeff Allen

    Lenses are part of a “look” so we need to supply both very sharp, optically perfect as we can make them lenses as well as lenses with intentional characteristics that some may view as flaws.
    Increasingly we see people use diffusion filters, uncoated lenses, and lenses from the seventies, eighties etc. that were the best in their day but are not viewed as such today.
    The key is choice storytelling is not a science lab its visual art.

  • Brandon Dube

    The people I know at Leica have never said the Summilux-C were particularly difficult to manufacture.

    I do not think it is sharpness, but a lack of veiling glare combined with nostalgic bias that lends this “clinical” assessment to more modern lenses.

    I’ve talked to plenty of cinematographers and DPs about their lenses — most are surprisingly knowledgeable about optics, and know what flaws are in the lenses they have chosen and these choices were made intentionally.

  • Jeff Allen

    Thats because they were designed by Iain Neil optically. They are however difficult to manufacture and maintain that optical performance mechanically.

    Sometimes the purists locked-up with test benches forget someone has to make the thing, someone has to keep it working that way and the end users push the product in ways they never imagined.

    Right now in cinematography we have too much sharpness or resolution making images look clinical and unnatural hence why 30-40 year old lenses are in vogue.

    Try telling top cinematographers / directors they are using flawed lenses you may get an unpleasant reply.

  • Lee

    Except that darn Zeiss 21 that just doesn’t seem to age.

  • Carleton, perhaps the most accurate, concise description of the 80s I’ve read.

  • Carleton Foxx

    I lived through the 80s and lenses weren’t the only thing that had trouble focusing—we were all pretty hazy back then. Life was great as long as you had enough quarters to keep the merry-go-round spinning…
    Thanks for the detailed answer.

  • Brandon Dube

    Without making a very lengthy comment- like all things in life, this is because of aberrations. Some, like lateral color, are very offensive to our eyes and should be kept at an utmost minimum, even if the MTF goes down as a result. The 80s was (so I’m told) filled with soft focus, hazy looking lenses — this is the signature of third order or primary spherical aberration. It basically positions the in-focus point for low frequencies (contrast) in front of those for high frequencies. The result is variable ‘haze’ through-focus, which is not very smooth or natural. Fifth order spherical will reduce the MTF (‘make the image less clinical’) and restore the “natural” behavior of moving in or out of focus. Mixing it and third in the right balance is a bit of an art form.

    I find the visual signature of the Summilux-C lenses very distinctive and naturalistic. I realize how juxtaposed that may sound given the above, but they have few flaws, and the ones they are are artfully chosen by design, they are true masterpieces. $40,000, available in exceptionally limited quantity masterpieces.

  • Carleton Foxx

    “‘Smoothness’ of image.” Eureka!!! Current lenses are lacking something, but I could never quite put it into words. Thank you for coming up with the perfect way to describe the missing quality.
    What, from your observations, makes one lens smoother than another? And could you single out a couple as worthy of special mention?

  • Well, looks like that will be happening eventually. But it will take some time. There are many lenses and only so much time.

  • Sean T

    Neat stuff, thank you guys. No other comments than I appreciate all your work. And it’d be great for you guys to run f/2.8 zoom lenses for ALL THE FANBOY FUEL.

  • Brandon Dube

    Something like the Summilux-C or Sumicron-C lenses, or Cooke, Angeieux, etc, will trash this category of lens in many ways (breathing, “smoothness” of image, vignetting, etc). The raw resolution (MTF) could be better or worse; it depends which lens.

  • boeck hannes

    nothing is as pleasing and relaxing as the numbers and graphs you produce. i am forever grateful!

    but after reading this post i naturally want to know how these photo lens design derived cine lenses compare optically to original cine lenses designed for the s35 format. any chance?

  • There is some truth to that. Newer designs, using more powerful computer design programs, newer methods of lens making, increased use of aspherics and better coatings (allowing more elements) all have made lenses better than they were 1 or 2 decades ago.

  • Tim Cooper

    Thanks Roger. I don’t know about anyone else, but I would enjoy a time-lapse video of your process if that’s something that interests anyone around there.

  • Ok, so a better performer is a better performer in many fields, not only in the sharpness department. Good to know! Thank you again!

  • Andrew Hughes

    One short question: How much does age matter? At first glance it would seem that newer designs do better, particularly stopped down. Any truth in this?

  • Marcello, those are mostly things I don’t test for, except distortion. Distortion as a % number doesn’t change much, but the pattern of distortion can when a lens is tilted or decentered. Of course all the aberrations can change, so there may be more coma, lateral color, etc. on one side than the other, but those affect MTF so you can infer it from our 4-quadrant MTF graphs for individual copies.

  • 10 copies is really a minimum for getting a range; sometimes we have to do more, especially with zooms. There will be a few more posts of things we’re considering doing this week and next, then hopefully we’ll settle on a new routine.

  • about copy variation: we read a lot about MTF test but i was asking myself what other factor change or stay the same in copy variation. Does a copy of a lens handle lens flare better than the other? Are vignetting and distortion the same from copy to copy? I really would like to see it discuss in a future post! Thank you for sharing your work, Roger!

  • User Colin

    Thanks Roger for doing these stopped-down tests. I think f/4 is probably a good compromise that demonstrates what stopping down achieves and can be compared across many lenses.

    Could your “10 copies” figure be calculated more dynamically. Like you test 4 or 5 lenses and decide if the variation is high. If not then stop; otherwise do more. And if you stopped at 7, say, would that let you test 3 copies at f/4? Testing a few copies at f/4 (at least in these early days) would also help confirm if picking an “average lens wide open” results in a representative “average lens stopped down”. If it generally does, then the approach of picking one would be fair.

    Some Sony cameras focus with the lens stopped down. I know some lenses have focus shift, though I’d have thought also that the modern intelligent cameras with focus-by-wire could compensate for that as you stop down. I wonder if your tests will shed light on this approach of focusing stopped-down vs wide-open.

  • grubernd

    from my in-real-life testing of single or at most three copies of a lens: the photographically (is that a word?) most interesting change in terms of aperture happens in the first f-stop from fully open. after that it get’s quite boring and academic.
    so i’d skip testing at fixed f- or T-stops and rather test 2/3 and 2 stops stopped down, because that is much more interesting.
    just take a single good average lens and run the full test for every 1/3 stop, down to 3 stops (yes, that’s 10 measurements) to see what i mean.

  • Samuel H

    Your ‘in-depth testing of a typical copy’ plan is a really, really, really good idea that almost nobody else could tackle. Thanks a lot for publishing all this extra info.

  • Ummon

    I concur – if limiting oneself to three apertures,
    wide open,
    one stop down and
    ´best´ aperture, if there is time to find out which this is ( if not f/8)
    would be the most interesting to me

  • Like most generalizations, it works some of the time. Two stops down is often plenty in the center to get very near max. There’s a lot of variation in the outer 1/3 though and the rule breaks down almost completely. We did something like that a long time ago, using Imatest and 50mm prime lenses and it was interesting to see how different they were.

  • I think you’ll find the next article interesting, particularly in regards to that Sigma 85 🙂

Follow on Feedly