Lenses and Optics

Variation Measurements for Wide-Angle Lenses

Published July 18, 2015

When we started this series, we introduced our methods using 24mm lenses, then followed up with looks at the 50mm and 35mm groups. Today we’re going to go back to the wide-angle lenses; the ones we expect to have the most variation of all. We probably should call this post the Zeiss Invitational, since they have by far the most wide-angle options. We also left a couple of the 24mm ‘ish’ lenses out of our opening post, so this post will include some of those, too.

Most of the lenses we’ll discuss today are of fairly recent development; released after 2007. The price range isn’t too large, ranging from $350 to $2200. One thing that should be noted: ultra-wide angle lenses get very complex. A lot of people like to make the assumption that more expensive means some increased level of quality control and therefore less copy-to-copy variation. Truth is more expensive often means more complex and sometimes, even with better quality control, it means more variation.

We also are including some tilt-shifts (shot untitled and unshifted, of course) in the results today. Remember that tilt-shift lenses have other complexities that make them a bit more likely to have some sample variation, so they have a bit of disadvantage here.

MTF Curves of Wide Angle Lenses

Ten copies of each lens were tested on our Trioptics Imagemaster Optical Bench using the standard protocol, which we described in the introductory blog post. The Nikon 20mm f/1.8G is an exception – we don’t have 10 copies available so we settled for 5.  All lenses are tested at their widest apertures, so take that into consideration when comparing MTFs; stopping down a lens would improve its MTF, so consider that when you compare and f/1.8 lens to an f/4 lens or whatever. These are presented roughly in order of widest to longest. I’ve included the Canon 24mm f/1.4 Mk II at the end to even out the number of graphs. We’d already presented it in our original article but I thought including it here would give some quick and easy comparison since it was the best of the 24mm f/1.4 lenses.


Roger Cicala and Brandon Dube, Lensrentals, 2015
Roger Cicala and Brandon Dube, Lensrentals, 2015
Roger Cicala and Brandon Dube, Lensrentals, 2015
Roger Cicala and Brandon Dube, Lensrentals, 2015
Roger Cicala and Brandon Dube, Lensrentals, 2015
Roger Cicala and Brandon Dube, Lensrentals, 2015


There aren’t a ton of surprises seen in the MTF curves of the wide-angle primes. As a general rule, the wider the lens, the more the MTF falls off away from center. Among the widest lenses, the Zeiss really do an amazing job. The Zeiss 15mm is clearly better off axis than the Canon 14mm f/2.8 L, and the Zeiss 18mm (from a pure MTF standpoint) is better than the 15mm.  The Rokinon 14mm doesn’t resolve in the center nearly as well as the others, but does keep that sharpness across the entire field of view. Off axis it is better than most of the other ultra-wide lenses.

All of the other lenses are pretty much excellent, with the exception, perhaps of the Canon 28mm f/1.8 lens. This isn’t surprising since this is a much older design and an inexpensive lens. The Zeiss 21mm f/2.8 looks just as good as one would expect (maybe better given it’s classic design). The Canon 24mm f.2.8 IS surprised me a bit, it’s actually a little better than the 28mm f/2.8 IS and I would have thought it might be a bit weaker.

Copy-to-Copy Variation

The simplest way to look at variation is with our Consistency number (for a complete discussion of how we arrive at the Consistency number, see this post). In summary, a higher consistency number means there is less copy-to-copy variation; the lens you buy is more likely to closely resemble the MTF average we presented above. In general, a score over 7 is excellent, a score from 6-7 good, 5-6 okay, 4-5 is a going to have significant copy-to-copy variation, and under 4 is a total crapshoot.

Our expectation was that the ultra-wide lenses, and to a lesser degree the tilt-shift lenses, would have more sample variation than the longer focal lengths. Here are the variation graphs in the same order as the MTF charts above.

Roger Cicala and Brandon Dube, Lensrentals, 2015

Roger Cicala and Brandon Dube, Lensrentals.com, 2015

Roger Cicala and Brandon Dube, Lensrentals.com, 2015

Roger Cicala and Brandon Dube, Lensrentals.com, 2015

Roger Cicala and Brandon Dube, Lensrentals.com, 2015


The first comment I have shouldn’t surprise many people. My experience with 14mm lenses meant I already knew they had a lot of copy-to-copy variation, and our tests certainly documented that. I was surprised that the Zeiss 15mm and 18mm lenses, despite being nearly as wide, had significantly higher consistency numbers (meaning less copy-to-copy variation).

If you’ve worked with tilt-shifts very much, you probably aren’t surprised to see there’s a bit of copy-to-copy variation with these lenses. There is a fair amount of mechanics involved in tilting and shifting that are going to add to variation. Not surprisingly, the variation curves show they are quite consistent in the center, but tend to have a bit of tilt as you go off axis.

The other lenses all did pretty well, and honestly much better than I expected. The one very pleasant surprise, with a Consistency score of 9.3, was the Canon 28mm f/2.8 IS. This lens isn’t particularly expensive but is amazingly consistent, much like the inexpensive Canon 50mm f/1.8 STM lens. Why it does so well (and better than the similar Canon 24mm f/2.8 IS lens) I can’t say. We’ve never taken a look inside it, but we’ll be correcting that shortly.

Another Big Picture Look

We’ve published results for 35 prime lenses so far, with another 15 or so on the way. I can’t say the results have surprised me greatly, but then I repair and optically adjust lenses all day. Most of the surprises I’ve gotten from this data, though, have been positive surprises. There are some amazingly consistent prime lenses with little copy-to-copy variation and they are often relatively inexpensive and excellent lenses. To keep you caught up, here are the consistency numbers for all the lenses we’ve tested so far in a sortable form — just click on the top row to sort by brand, aperture, or focal length.

Manufacturer Focal Length (mm) Aperture Consistency
Canon Mk II241.46.3
Canon STM501.89.3
Zeiss Makro5027.3
Zeiss Otus551.46.5
Sigma Art501.47.5
Canon Mk II142.84.0
Canon TS-E II243.55.3
Canon TS-E1744.9
Canon IS242.85.9
Canon IS282.89.3
Canon IS3527.7
Sigma Art351.45.1

Please don’t make a lot of generalizations yet. This is a big, ongoing project, and this is still less than half the data. I think speculations are interesting, and I do think the information is useful. We’ll be publishing the data for longer focal length lenses soon. Then we’ll follow up by looking at some of the wide-aperture lenses stopped down to see how much copy-to-copy variation decreases as the aperture stops down.

And remember, the consistency number is just a massive compression of other data — you’ll get a lot more information looking at the variation graphs than the consistency number. Philadelphia and Phoenix both have population numbers of about 1.5 million, but that doesn’t make them the same.

We’ll also continue to update this table of Consistency scores so there’s a single, long-term reference for you to go to. And hopefully, within a couple of weeks we’ll complete arrangements for there to be a comparison tool so you can look at MTF and Consistency graphs side-by-side when you’re trying to make lens decisions.

Roger Cicala and Brandon Dube


July, 2015

Author: Roger Cicala

I’m Roger and I am the founder of Lensrentals.com. Hailed as one of the optic nerds here, I enjoy shooting collimated light through 30X microscope objectives in my spare time. When I do take real pictures I like using something different: a Medium format, or Pentax K1, or a Sony RX1R.

Posted in Lenses and Optics
  • Sid

    You mention the Nikon 20/1.8 but I don’t see any MTF or variance numbers. Am I missing something? Thanks for your invaluable resource 🙂

  • Roger Cicala

    Lynn, the Samyangs are mostly non-adjustable. What you get is what you get. Even if they were adjustable, the cost of adjusting them would probably add at least 50% to their price.

    We do actually provide MTF measurements for some of the better lenses sold through LensAuthority, but they are all tested during their lifetime and right before sale. LA is actually one of those ‘wholly owned subsidiary’ kind of things so they only ask for MTF printouts on certain lenses. Many others sell so quickly when listed they just don’t think it’s needed.

  • I’m curious if LR is able to adjust Samyangs so that their “Adjusted Consistency” is good to very-good. That would suggest that buying a used Samyang from LR … certified to be a ‘good copy’ … might be a good to very-good idea 🙂

    Related? Is LR able to supply an Imatest and/or OLAF measurement for a specific for-sale used lens? That would be somewhat equivalent to my Dell U2410 IPS a1998 monitor being delivered with measurements for that monitor, including De2k.

  • Roger Cicala

    Adam, we will do some of the better zooms, but zooms have a LOT more variability than primes and take way more time to test, so we probably won’t do the whole set.

  • Adam

    Interesting, love this data. Are you guys planning to do zooms as well in the future or are you just focusing on primes? Obviously zooms are more work since they zoom, but they are also what most people tend to use.

  • Roger Cicala

    Brian, tilt-shifts in general have lower consistency. RokiBowYang has lower consistency. The RokiBowYang tilt shift is exactly what you’d expect – low consistency.

  • Brian

    Now I’m very curious about the consistency of the “RokiBowYang” 24mm tilt-shift lens. When you dissected one two years ago (http://wordpress.lensrentals.com/2013/05/first-look-rokinon-24mm-f3-5-tilt-shift-part-2), you found that the lens consisted of two assemblies with no means of adjusting tilting and centering. I suspect that would lower the consistency score quite a lot, unless Samyang’s manufacturing tolerances are quite good.

  • Roger Cicala

    Hi Alan,

    We’ve done some Leica, but with most of them we don’t have 10 copies to test. We might get a series of 5 copies, although I think it will have to wait until things slow down in the fall.

    Lens tuning I think would be something limited to some of the more expensive lenses. It’s time consuming and requires a lot of expensive equipment. I think most people wouldn’t be interested in it at the price we’d have to charge. There are a few lenses, though, where the adjustment is pretty straightforward most of the time, like Canon 50mm f/1.2, etc. and we might offer it for just that select group. But once we get into things taking 4 hours on an MTF bench, it gets really really pricey.

  • Alan


    Thank you for a very informative series of posts. Considering the time and the associated cost to do these tests, I really appreciate what you are doing here, especially for sharing these.

    I am wondering if you are going to test M-mount lenses next, especially those Leica lenses. Despite the high price tag, I feel like Leica M might have a surprisingly high sample variation, mostly due to the complexity of the design.

    Also, have you considered offering a lens tuning service? With this info right here, together with your extensive experience, I believe many would rather send their lenses to Lensrental for tuning than the lens maker themselves. Just a thought :D.

  • Roger Cicala

    Seth, yes, we hope to have something like that available next week partnering with one of our friends that already has such a tool online.

  • Seth

    Excellent work, as always!

    Are there any thoughts on whipping up a webapp with this data, that lets you overlay two selected lenses on the same chart or at least next to each other? I realize that might be outside the scope of what you guys are doing, since it would require web design and possibly different hosting.

    I keep wanting to compare curves on lenses like the zeiss 25mm and canon’s 28mm, but not having the charts aligned makes it awfully hard to compare the curves visually.

  • A

    Thank you (both) so much for your hard work on this; it’s really interesting!

    Something I would find interesting is to find out which factories produced which lenses. For example, are the low variance Canon lenses all being built on a new/different production line to the other lenses? Or is it a design/process change? I guess at it’s simplest the question is – what countries are the lenses being made in?

  • Roger Cicala

    grubrend, it is indeed missing. We planned on doing it, but weren’t able to test 10 copies (we don’t carry a lot of that lens and many of the copies were out on rental).

  • is the summer heat confusing my senses or are the data and graphs for the Nikkor 20/1.8G missing?

  • MayaTlab

    Wow, the difference between the 24mm IS USM and 28mm IS USM surprises me. I’ll be following with interest any investigation into the matter !

Follow on Feedly