Lenses and Optics

Variation Measurements for Telephoto Lenses

Published July 21, 2015

We’re nearing the end of the Varation series for prime lenses. If you are joining in late, you may want to go back to the original article for an introduction into the methods used. Today will look at the short telephoto group, lenses ranging in focal length from 85mm to 150mm. We’ve also included a summary table of all the lenses we’ve tested to date at the end of the article.

 

Today’s lenses are somewhat of older design, with several models released in the 1990s. The prices range from $400 to $4200. My own assumption (and we all know how well my assumptions usually turn out) was that these longer lenses would have less copy-to-copy variation than the wider focal lengths did.

MTF Curves of Telephoto Lenses

Ten copies of each lens were tested on our Trioptics Imagemaster Optical Bench using the standard protocol, which we described in the introductory blog post.  All lenses are tested at their widest apertures, so take that into consideration when comparing MTFs; stopping down a lens would improve its MTF, so consider that when you compare an f/1.8 lens to an f/2.8 lens or whatever. These are presented roughly in order of widest to longest.

Roger Cicala and Brandon Dube, Lensrentals.com, 2015

Roger Cicala and Brandon Dube, Lensrentals.com, 2015

Roger Cicala and Brandon Dube, Lensrentals.com, 2015
Roger Cicala and Brandon Dube, Lensrentals.com, 2015

 

Before you all go crazy over the Canon 85mm curves, remember that f/1.2 is significantly wider aperture than f/1.4. Also remember the Canon 85mm f/1.8 and 100mm f/2 lenses are ancient (1991-92) designs. The Nikon 85mm lenses are much more recent, released in 2010 and 2012 respectively and it shows in their resolution. They are excellent from center to edge. The Zeiss 85mm f/1.4 is particularly good in the center, where it has significantly higher resolution than even the Nikons, but not as good off axis. The 85mm f/1.4 Otus, of course, is just amazing optically, with easily the best resolution in the center and excellent resolution to the edges, although it does have significant astigmatism in the outer 1/3. The Rokinon 85mm f/1.4 shows again that Rokinon makes some lenses that are nearly as good as the brand-name lenses at very low prices.

I separated the MTF curves for the Macro and 135mm lenses, since they are really quite different beasts than the 85mm lenses.

Roger Cicala and Brandon Dube, Lensrentals.com, 2015
Roger Cicala and Brandon Dube, Lensrentals.com, 2015
Roger Cicala and Brandon Dube, Lensrentals.com, 2015

Looking at the 100 range Macro lenses, it’s apparent that the Nikon 105  f/2.8 VR Micro and Canon 100 f/2.8 Macro are excellent, but don’t resolve quite as well as the other two Macro lenses. This isn’t too surprising; they are both older designs. The amazing thing to me is just how excellent the Zeiss 100mm f/2 Makro Planar is; it compares very well with the new Canon 100mm f/2.8 IS L, even though the Zeiss is being tested at f/2 and the Canon at f/2.8. That is really amazing.

I would point out, though, that this is testing at infinity focusing distance. As we know, Macro lens performance can be quite different at macro shooting distances.

The Canon 135 f/2 L is a superb lens, one of my favorites. But there’s not much question if you don’t mind lugging around the weight, paying the price, and manually focusing, the Zeiss 135 f/2 gives you higher resolution except at the edge of the image.

Copy-to-Copy Variation

The simplest way to look at variation is with our Consistency number (for a complete discussion of how we arrive at the Consistency number, see this post). In summary, a higher consistency number means there is less copy-to-copy variation; the lens you buy is more likely to closely resemble the MTF average we presented above. In general, a score over 7 is excellent, a score from 6-7 good, 5-6 okay, 4-5 is a going to have significant copy-to-copy variation, and under 4 is a total crapshoot.

Our expectation was that the telephoto lenses would have less sample variation than the wide-angle lenses. Here are the variation graphs in the same order as the MTF charts above.

Roger Cicala and Brandon Dube, Lensrentals.com, 2015
Roger Cicala and Brandon Dube, Lensrentals.com, 2015

Roger Cicala and Brandon Dube, Lensrentals.com, 2015

Roger Cicala and Brandon Dube, Lensrentals.com, 2015

 

The most consistent of the 85mm lenses are the two Nikons along with the Zeiss Otus. The Canon 85 f/1.2 is, of course, an f/1.2. Its optical assembly is basically a single unit; no adjustments are possible as far as we know. The other two Canon lenses are much, much older design and probably designed more towards film tolerances, which would be less critical than that needed for digital sensors. Still, my theory that longer focal lengths would have less copy-to-copy variation takes a pretty big hit here.

Roger Cicala and Brandon Dube, Lensrentals.com, 2015
Roger Cicala and Brandon Dube, Lensrentals.com, 2015
Roger Cicala and Brandon Dube, Lensrentals.com, 2015

 

The longer focal lengths do seem to have less copy-to-copy variation. The older designs of the Canon 100mm f/2.8 Macro and Nikon 105 VR Macro aren’t quite as good as the others. The newer Canon 100mm f/2.8 IS L Macro and the Zeiss 100mm f/2 Makro are just excellent, with some of the highest Consistency numbers of any lenses. The 135mm lenses are also both very good, although the older Canon design isn’t as good as the new, and much more expensive, Zeiss 135mm Apo-Sonnar.

Updated Table

I’ve updated the sortable table of consistency scores by brand, focal length, and aperture and reproduced it below. If you want to look at the variables differently, you can click on any of the headings and sort things by that category.

Manufacturer Focal Length (mm) Aperture Consistency
Rokinon241.44.0
Nikon241.44.6
Sigma241.44.9
Canon Mk II241.46.3
Canon501.26.0
Canon501.45.5
Canon STM501.89.3
Nikon581.46.7
Nikon501.44.6
Nikon501.86.3
Zeiss501.46.1
Zeiss Makro5027.3
Zeiss Otus551.46.5
Sigma Art501.47.5
Rokinon501.44.0
Canon Mk II142.84.0
Rokinon142.84.0
Zeiss152.86.5
Zeiss183.56.1
Canon TS-E II243.55.3
Canon TS-E1744.9
Zeiss212.85.2
Zeiss2526.2
Canon IS242.85.9
Canon281.86.0
Canon IS282.89.3
Canon351.46.6
Canon IS3527.7
Nikon351.45.9
Nikon351.84.5
Zeiss351.45.7
Zeiss3526.4
Sigma Art351.45.1
Rokinon351.43.6
Canon851.25.9
Canon851.84.5
Rokinon851.44.2
Nikon851.47.4
Nikon851.88.1
Canon10023.6
Zeiss 851.45.3
Zeiss Otus851.47.8
Canon Macro1002.86.4
Canon Macro IS L1002.89.5
Nikon Micro VR1052.86.1
Zeiss Makro10029.3
Canon13527.0
Zeiss13528.8

Variation Sumary

I had planned another post showing good predictors of copy-to-copy variation with a bit more in-depth analysis. The truth is, the summary doesn’t require another post. There are some trends, but nothing that serves as a nice, general, overall predictor of which lenses have little copy-to-copy variation. There are a few things worth noting though.

I hate to get into brands because it makes the fanboys go insane, but there are a couple of trends that are pretty obvious when you sort the table above. Zeiss lenses overall do better than the others; no Zeiss lens falls below “5” on our scale, and the vast majority are over “6”. Canon’s newer lenses are also superb, with very high consistency scores, and several of these are of the reasonably priced variety. Rokinon lenses are at the lower end of the consistency scores, which is entirely understandable. These are superb optics made at an amazingly low price. Achieving that low price means not having adjustable optics and taking some cost saving measures. They’re still awesome bargains and I shoot several of them myself. Copy-to-copy variation is pretty understandable at that price.

Second, it seems we probably should put an ‘aperture adjustment’ constant in our formula, or at least emphasize more that wider aperture is going to have more variation. The trend isn’t very strong, but there is some correlation that f/1.4 lenses have more variation than f/2 or f/2.8 lenses. Before we decide why that might be, though, we need to look at variation of a group of f/1.4 lenses as we stop them down. I don’t know how much of the variation is from shooting at a wider aperture itself and how much is the more complex design required to get a lens to that wide aperture. I suspect it’s a design thing, though, because some wide-aperture lenses have very little variation.

My theory that longer focal lengths have less variation than wide-angle lenses wasn’t nearly as important as I thought.  There’s a little trend that way (the trend line in the graph below is Excel’s default linear trend), but it’s not all that impressive.

 

The most favorite internet theory, that more expensive lenses have some magic ‘quality control’ that eliminates variation, is totally untrue. The trend line for that is just about flat. If you’re surprised don’t be. Money goes into multiple elements, more expensive glass, and complex designs. The lenses get made in the same factory, generally, and tested with the same equipment as that company’s other lenses. Actually, the QA probably has to be a bit better just to keep the variation the same. More complex lenses, designed to tighter tolerances, will normally have more variation, not less. Tighter QC is probably needed just to keep variation the same.

 

Overall, year of design didn’t have a lot of impact either. But we did see that new Canon lenses seem to be doing much better. When you sort the table above, it’s clear that there are several new Canon designs at the top of the consistency scores. We’ve also been noting for several years the increased modularity in Canon’s lens designs when we do lens teardowns. I suspect the two trends are related.

 

But at this time, there’s not much that can be done to predict how consistent the copy-to-copy variation in a given lens will be. We’ll just have to keep testing it.

 

Roger Cicala and Brandon Dube

Lensrentals.com

July 2015

ADDENDUM

Our MTF and Consistency graphs are now available in a comparison tool at The Digital Picture. You can use their lens comparison tool to compare any two lenses that we’ve done, and will update their site as we do more. I think this is amazingly useful compared to searching through our blog posts to find what you are looking for. We appreciate Bryan and the staff at TDP for doing all the heavy lifting to make this possible.

Author: Roger Cicala

I’m Roger and I am the founder of Lensrentals.com. Hailed as one of the optic nerds here, I enjoy shooting collimated light through 30X microscope objectives in my spare time. When I do take real pictures I like using something different: a Medium format, or Pentax K1, or a Sony RX1R.

Posted in Lenses and Optics
  • JohnL

    Quite amusingly (considering your table leader) I just traded in my Canon EF100 Macro for the L version and I’m on my third one (one a day). The first one wasn’t good on the right and had focusing issues at close range. The second one was just simply blurry down the left side. (Well, I guess that is some sort of consistency.) I have a third one but am losing the will to live, so having micro-focus adjusted it I’ll go with it if it isn’t that bad (the sides seem okay, not convinced about the focus consistency and it’s the noisiest of the bunch). I used FoCal to adjust it and it claims it has a lot of astigmatism, although it said the second one was great. Why do I buy lenses?

    Oh and while I’m at it should a (fairly basic) lens quality rating (A-C, A-E?) be added to the variation score (X/Y say), so you can see if you care (some variation on a super-star lens being better than almost none on a so-so one)? At a minimum it would fill up the comments section 🙂

  • Jim Thomson

    “I hate to get into brands because it makes the fanboys go insane”

    As a certified tinfoil hat wearing, witch hunting, Canon fanboy, with confirmation bias, who throws out red herrings I think you got that just right.

  • Igor

    The Otus is excellent in the center of the frame, competitive in the middle and poor at the edge (both resolution and astigmatism). Nothing like “from edge to edge”.

  • Roger Cicala

    Mike, if they aren’t good we get them repaired and optically adjusted.

  • Derek

    Another in this great series, many thanks.

    I note you are testing at infinity and admit the Macro lenses may perform differently at marco… as many of these medium telephoto are used in portraiture, does anything change at around 5 meters focussing distance? does their performance peak at some focus distance other than infinity?.. perhaps a test of just one lens to prove the point… when you’ve got nothing better to do 🙂

  • Mike Ronesia

    Two questions. When do you plan to add M4/3 lenses? And the big question, how do order one of the good ones when we rent a lens?

  • Brandon

    Lee,

    There’s about 12 different sets of stop down data I’ve taken so far. The goal is eventually to provide the MTF at all apertures, but it is absolutely out of the question to do variance at all apertures.

    Within 2 weeks the first stop down data sets will go up. Nikon mount will never get it because we have no accurate form of aperture control, but EF mount, sony E, and micro four thirds are all possible.

    Regards,
    Brandon

  • Lee Saxon

    I hope as this testing database grows, especially now that we’ve got it plugged into the Digital Picture backend which allows for aperture changes, that we see more apertures than just “wide open.”

    I know that for time reasons there’s no way you could do every aperture on every lens. You’ve got to let renters have these lenses sometime! But I would add one (in some cases two) more apertures:

    1.) One “stopped down” aperture a few clicks from “wide open.” I use 5.6 personally, but that’s subjective.

    2.) “Matching” apertures. 85’s are a great example of why. You’ve got one 85/1.2 in a set of 85/1.4’s, and you point out that it makes a difference. That makes me want to see the 85/1.2 AT 1.4 – in fact, I’d rather see that than “wide open” (if time prevented you from doing both).

  • Roger Cicala

    Zooms are coming, Bill. Give us a couple of weeks.

  • Good work Roger & Brandon!

    Would it be too much if I asked to see some zoom lenses added in the future ?

    Id’ like to see the Nikon 14-24, 24-70, 70-200 and their counterparts from Canon, Tamron, Sigma tested along.

    Thank you very much for the contributions and services to the photographic community guys!

  • Brandon

    Mike,

    I have pitched the idea of turning the data into some sort of academic paper to my research director at the university I attend. She is graduating about 9 PhDs or masters students in the coming months and is too busy to look into a new endeavor, but it is something we are looking into.

    Best,
    Brandon

  • Michael Stratil

    Roger,

    This is a magnificent study, and I would like to recommend that you try to publish it in an appropriate journal. Your method probably does not have a large enough sample to provide probability estimates of central tendency and dispersion, but maybe a journal would take it as an exploratory starting point for subsequent research.

    Mike Stratil

  • Roger Cicala

    Andreas, I was too, but no ideas why. Maybe when we take some apart we’ll get a hint.

  • Roger Cicala

    Brett, all the lenses we tested were less than a year old, so I can’t really comment about older copies, although what you say seems logical.

  • Andreas D.

    I am surprised that the Canon 24/2.8 IS and 28/2.8 IS have very different consistency scores (5.9 vs. 9.3). Any ideas on that?

  • Brett F

    For the lenses with older designs, were the tested lenses from various manufacturing years or from a single manufacturing year? My assumption would be that, say, two EF 85 f/1.8s that were both manufactured in 2015 would be more likely to be similar in performance than two lenses built 10 or 20 years apart. Any thoughts on this?

  • Roger Cicala

    John,

    We don’t know how Zeiss tests exactly, and there are a number of things that could be involved. My best guess is that when Zeiss measures they refocus at each point while we measure each point at best center focus. In our measurements field curvature will lower MTF since the off-axis points are not perfectly focused. (I personally think this is more real-world, but there are certainly people who disagree with me.)

    Other possibilities: We use Photopic eye light for our measurements, but single wavelength light or complete white light are also options. Again, I don’t know what the Zeiss MTF machine does, but the Trioptics machine increases lighting as the lens vignettes. If the Zeiss MTF machine doesn’t, then the results would be different (not increasing lighting would give a higher MTF but might be more variable). Also our MTF number is an average of 4 rotations for each lens and then that average for 10 lenses. It may well be that the Zeiss MTF curves are one measurement per lens rather than 4, etc. Finally, I think the Zeiss K benches measure 5 points compared to our 20, but I only have been told that, i don’t know for certain.

    In other words, I don’t know for certain wy, but I’m not surprised that their’s are different.

  • Roger Cicala

    John,

    We don’t know how Zeiss tests exactly, and there are a number of things that could be involved. My best guess is that when Zeiss measures they refocus at each point while we measure each point at best center focus. In our measurements field curvature will lower MTF since the off-axis points are not perfectly focused. (I personally think this is more real-world, but there are certainly people who disagree with me.)

    Other possibilities: We use Photopic eye light for our measurements, but single wavelength light or complete white light are also options. Again, I don’t know what the Zeiss MTF machine does, but the Trioptics machine increases lighting as the lens vignettes. If the Zeiss MTF machine doesn’t, then the results would be different (not increasing lighting would give a higher MTF but might be more variable). Also our MTF number is an average of 4 rotations for each lens and then that average for 10 lenses. It may well be that the Zeiss MTF curves are one measurement per lens rather than 4, etc.

    In other words, I don’t know for certain wy, but I’m not surprised that their’s are different.

  • John

    Dear Roger

    There is one thing about the Zeiss 135mm APO that I do not Understand. Can you please explain the reason why your MTF chart is so different from the one provided by Zeiss itself? As far as I know, contrary to Canon and Nikon, Zeiss’ MTF charts come from testing actual lenses and they are not just theoretical, and Your statistics also show very little variance between samples so yours cannot be far from what Zeiss has used for testing. Here is the link to the MTF chart provided by Zeiss.
    http://www.zeiss.com/content/dam/Photography/new/pdf/en/downloadcenter/datasheets_slr/aposonnart2135.pdf

  • Wilson Laidlaw

    I suspect it is just as well you did not have Leica lenses in sufficient quantity to test variation. I expect it would have been pretty bad. Out of my four most recent Leica M lens purchases, two had to go straight back to the factory for adjustment and one was so bad it had to be replaced. Each one came with a signed QC certificate, where I can only assume that someone looked at it and said “yes it’s a lens.” It only took me a few seconds comparing the rangefinder focus point with the focus peaking confirmation to realise that these lenses were unusable in their as delivered state. Some Leica lenses are notorious for their sample variation, e.g. the 35mm ASPH Summilux, mostly for the degree of aperture shift. It makes you wonder about the benefits of hand assembly versus a mechanised production line, which the likes of Canon use.

  • Kim, Norway

    This test confirms that there IS a point in seeking out the gems, ’cause they ARE out there! And don’t write off a lensmodel; you may have one of the duds.

    Even if the manufacturers didn’t adjust their lenses at all, pure chance would produce a perfect sample now and then!

    I own an old Nikkor DX 18-70mm that beats or equals all 17-55mm/2.8, 35/1.8, 50/1.4 and 50/1.8 I have compared it against on resolution. Should not be possible, but the resolution in that lens is just way better on all focal-lenghts than every other sample of that lens I have come across.

    Moral: Hang on to the ‘perfect’ samples! You may never see one as good again.

  • Peter

    Roger, thanks for your explanation.

    Nevertheless, I would love to know if Leica’s sample variation is any different than the other big names. And I am sure you would too! Perhaps you might run with a smaller sample of say 50/ 1.4, or 50/ 2 APO and bootstrap the error around Leica values to estimate your score value vs. the others.

    I mean if any third party has the chops to do this comparison, it is your team.

  • Roger Cicala

    Peter, what’s up basically is we don’t have 10 copies of most Leica lenses. There’s just not enough demand to stock more.

  • Peter

    Very interesting study of an issue we all know about. You have pulled back the curtain in a very helpful manner, thanks!
    But I am curious about a very obvious omission from your brand selection… there is a German producer that prides itself, usually with good cause, on its decades of excellent design, production and quality (assurance and control). You rent their lenses, some of which are notorious for whomping sample variation. Sadly, I have rented a couple of those examples of poor 28mm and 24mm from LensRentals, but then found other copies that were good to excellent.
    My question is, Roger, what’s up with tests of your Leica M inventory?

  • Yet another informative article Roger. Thanks!

  • Matteo

    Oh Noooooo, I wish I could see even how the nikkor 135mm F2 DC behave compared to the other two 135mm

  • Roger Cicala

    Carol, the older electronic lenses, like the Canon, stay where you put them. So we can focus to infinity on a camera, then take it off and test it. Many of the micro 4/3 are that way too. The newer Sony ‘electromagnetic’ focusing, when you take it off the camera, just falls back to close focus. So to test them on the bench, we have to keep them electrically powered during the test.

  • Carol Teater

    Thanks, Roger, of course I didn’t even think of the electronic control versus mechanical control of focus. That is one of the downsides of most mirrorless lenses (though it does make me wonder how you tested the 85/1.2…).

    Thanks for all you do! Love me some LensRentals! 🙂

  • Brandon

    Andre,

    Please email me at brandondube@gmail.com and I will discuss vignetting and how it impacts MTF with you – I fear you may have things a bit backwards and the topic is too long for a comments section.

    Florian,

    I have run quite a number of Leica lenses, but they are very expensive, don’t rent as often, and are made in much lower quantity. These factors together contribute to LR carrying far fewer of them. Most models we don’t have even 5 copies of to test. The Loxia lenses are for sony E mount and could be tested, but they are not something I have aimed for yet.

    Regards,
    Brandon

  • Mike M

    Lens Rentals, you could sell new and used lenses with your “certified” optical test numbers. You could add an additional fixed dollar amount onto the price for the testing cost. You could also add a variable cost onto exceptional performing copies. I would pay more (a little more, if my wife lets me) to be sure I’m getting a good copy.

Follow on Feedly