Variation Measurements for Telephoto Lenses
We’re nearing the end of the Varation series for prime lenses. If you are joining in late, you may want to go back to the original article for an introduction into the methods used. Today will look at the short telephoto group, lenses ranging in focal length from 85mm to 150mm. We’ve also included a summary table of all the lenses we’ve tested to date at the end of the article.
Today’s lenses are somewhat of older design, with several models released in the 1990s. The prices range from $400 to $4200. My own assumption (and we all know how well my assumptions usually turn out) was that these longer lenses would have less copy-to-copy variation than the wider focal lengths did.
MTF Curves of Telephoto Lenses
Ten copies of each lens were tested on our Trioptics Imagemaster Optical Bench using the standard protocol, which we described in the introductory blog post. All lenses are tested at their widest apertures, so take that into consideration when comparing MTFs; stopping down a lens would improve its MTF, so consider that when you compare an f/1.8 lens to an f/2.8 lens or whatever. These are presented roughly in order of widest to longest.
Before you all go crazy over the Canon 85mm curves, remember that f/1.2 is significantly wider aperture than f/1.4. Also remember the Canon 85mm f/1.8 and 100mm f/2 lenses are ancient (1991-92) designs. The Nikon 85mm lenses are much more recent, released in 2010 and 2012 respectively and it shows in their resolution. They are excellent from center to edge. The Zeiss 85mm f/1.4 is particularly good in the center, where it has significantly higher resolution than even the Nikons, but not as good off axis. The 85mm f/1.4 Otus, of course, is just amazing optically, with easily the best resolution in the center and excellent resolution to the edges, although it does have significant astigmatism in the outer 1/3. The Rokinon 85mm f/1.4 shows again that Rokinon makes some lenses that are nearly as good as the brand-name lenses at very low prices.
I separated the MTF curves for the Macro and 135mm lenses, since they are really quite different beasts than the 85mm lenses.
Looking at the 100 range Macro lenses, it’s apparent that the Nikon 105 f/2.8 VR Micro and Canon 100 f/2.8 Macro are excellent, but don’t resolve quite as well as the other two Macro lenses. This isn’t too surprising; they are both older designs. The amazing thing to me is just how excellent the Zeiss 100mm f/2 Makro Planar is; it compares very well with the new Canon 100mm f/2.8 IS L, even though the Zeiss is being tested at f/2 and the Canon at f/2.8. That is really amazing.
I would point out, though, that this is testing at infinity focusing distance. As we know, Macro lens performance can be quite different at macro shooting distances.
The Canon 135 f/2 L is a superb lens, one of my favorites. But there’s not much question if you don’t mind lugging around the weight, paying the price, and manually focusing, the Zeiss 135 f/2 gives you higher resolution except at the edge of the image.
Copy-to-Copy Variation
The simplest way to look at variation is with our Consistency number (for a complete discussion of how we arrive at the Consistency number, see this post). In summary, a higher consistency number means there is less copy-to-copy variation; the lens you buy is more likely to closely resemble the MTF average we presented above. In general, a score over 7 is excellent, a score from 6-7 good, 5-6 okay, 4-5 is a going to have significant copy-to-copy variation, and under 4 is a total crapshoot.
Our expectation was that the telephoto lenses would have less sample variation than the wide-angle lenses. Here are the variation graphs in the same order as the MTF charts above.
The most consistent of the 85mm lenses are the two Nikons along with the Zeiss Otus. The Canon 85 f/1.2 is, of course, an f/1.2. Its optical assembly is basically a single unit; no adjustments are possible as far as we know. The other two Canon lenses are much, much older design and probably designed more towards film tolerances, which would be less critical than that needed for digital sensors. Still, my theory that longer focal lengths would have less copy-to-copy variation takes a pretty big hit here.
The longer focal lengths do seem to have less copy-to-copy variation. The older designs of the Canon 100mm f/2.8 Macro and Nikon 105 VR Macro aren’t quite as good as the others. The newer Canon 100mm f/2.8 IS L Macro and the Zeiss 100mm f/2 Makro are just excellent, with some of the highest Consistency numbers of any lenses. The 135mm lenses are also both very good, although the older Canon design isn’t as good as the new, and much more expensive, Zeiss 135mm Apo-Sonnar.
Updated Table
I’ve updated the sortable table of consistency scores by brand, focal length, and aperture and reproduced it below. If you want to look at the variables differently, you can click on any of the headings and sort things by that category.
| Manufacturer | Focal Length (mm) | Aperture | Consistency |
|---|---|---|---|
| Rokinon | 24 | 1.4 | 4.0 |
| Nikon | 24 | 1.4 | 4.6 |
| Sigma | 24 | 1.4 | 4.9 |
| Canon Mk II | 24 | 1.4 | 6.3 |
| Canon | 50 | 1.2 | 6.0 |
| Canon | 50 | 1.4 | 5.5 |
| Canon STM | 50 | 1.8 | 9.3 |
| Nikon | 58 | 1.4 | 6.7 |
| Nikon | 50 | 1.4 | 4.6 |
| Nikon | 50 | 1.8 | 6.3 |
| Zeiss | 50 | 1.4 | 6.1 |
| Zeiss Makro | 50 | 2 | 7.3 |
| Zeiss Otus | 55 | 1.4 | 6.5 |
| Sigma Art | 50 | 1.4 | 7.5 |
| Rokinon | 50 | 1.4 | 4.0 |
| Canon Mk II | 14 | 2.8 | 4.0 |
| Rokinon | 14 | 2.8 | 4.0 |
| Zeiss | 15 | 2.8 | 6.5 |
| Zeiss | 18 | 3.5 | 6.1 |
| Canon TS-E II | 24 | 3.5 | 5.3 |
| Canon TS-E | 17 | 4 | 4.9 |
| Zeiss | 21 | 2.8 | 5.2 |
| Zeiss | 25 | 2 | 6.2 |
| Canon IS | 24 | 2.8 | 5.9 |
| Canon | 28 | 1.8 | 6.0 |
| Canon IS | 28 | 2.8 | 9.3 |
| Canon | 35 | 1.4 | 6.6 |
| Canon IS | 35 | 2 | 7.7 |
| Nikon | 35 | 1.4 | 5.9 |
| Nikon | 35 | 1.8 | 4.5 |
| Zeiss | 35 | 1.4 | 5.7 |
| Zeiss | 35 | 2 | 6.4 |
| Sigma Art | 35 | 1.4 | 5.1 |
| Rokinon | 35 | 1.4 | 3.6 |
| Canon | 85 | 1.2 | 5.9 |
| Canon | 85 | 1.8 | 4.5 |
| Rokinon | 85 | 1.4 | 4.2 |
| Nikon | 85 | 1.4 | 7.4 |
| Nikon | 85 | 1.8 | 8.1 |
| Canon | 100 | 2 | 3.6 |
| Zeiss | 85 | 1.4 | 5.3 |
| Zeiss Otus | 85 | 1.4 | 7.8 |
| Canon Macro | 100 | 2.8 | 6.4 |
| Canon Macro IS L | 100 | 2.8 | 9.5 |
| Nikon Micro VR | 105 | 2.8 | 6.1 |
| Zeiss Makro | 100 | 2 | 9.3 |
| Canon | 135 | 2 | 7.0 |
| Zeiss | 135 | 2 | 8.8 |
Variation Sumary
I had planned another post showing good predictors of copy-to-copy variation with a bit more in-depth analysis. The truth is, the summary doesn’t require another post. There are some trends, but nothing that serves as a nice, general, overall predictor of which lenses have little copy-to-copy variation. There are a few things worth noting though.
I hate to get into brands because it makes the fanboys go insane, but there are a couple of trends that are pretty obvious when you sort the table above. Zeiss lenses overall do better than the others; no Zeiss lens falls below “5” on our scale, and the vast majority are over “6”. Canon’s newer lenses are also superb, with very high consistency scores, and several of these are of the reasonably priced variety. Rokinon lenses are at the lower end of the consistency scores, which is entirely understandable. These are superb optics made at an amazingly low price. Achieving that low price means not having adjustable optics and taking some cost saving measures. They’re still awesome bargains and I shoot several of them myself. Copy-to-copy variation is pretty understandable at that price.
Second, it seems we probably should put an ‘aperture adjustment’ constant in our formula, or at least emphasize more that wider aperture is going to have more variation. The trend isn’t very strong, but there is some correlation that f/1.4 lenses have more variation than f/2 or f/2.8 lenses. Before we decide why that might be, though, we need to look at variation of a group of f/1.4 lenses as we stop them down. I don’t know how much of the variation is from shooting at a wider aperture itself and how much is the more complex design required to get a lens to that wide aperture. I suspect it’s a design thing, though, because some wide-aperture lenses have very little variation.
My theory that longer focal lengths have less variation than wide-angle lenses wasn’t nearly as important as I thought. There’s a little trend that way (the trend line in the graph below is Excel’s default linear trend), but it’s not all that impressive.
The most favorite internet theory, that more expensive lenses have some magic ‘quality control’ that eliminates variation, is totally untrue. The trend line for that is just about flat. If you’re surprised don’t be. Money goes into multiple elements, more expensive glass, and complex designs. The lenses get made in the same factory, generally, and tested with the same equipment as that company’s other lenses. Actually, the QA probably has to be a bit better just to keep the variation the same. More complex lenses, designed to tighter tolerances, will normally have more variation, not less. Tighter QC is probably needed just to keep variation the same.
Overall, year of design didn’t have a lot of impact either. But we did see that new Canon lenses seem to be doing much better. When you sort the table above, it’s clear that there are several new Canon designs at the top of the consistency scores. We’ve also been noting for several years the increased modularity in Canon’s lens designs when we do lens teardowns. I suspect the two trends are related.
But at this time, there’s not much that can be done to predict how consistent the copy-to-copy variation in a given lens will be. We’ll just have to keep testing it.
Roger Cicala and Brandon Dube
Lensrentals.com
July 2015
ADDENDUM
Our MTF and Consistency graphs are now available in a comparison tool at The Digital Picture. You can use their lens comparison tool to compare any two lenses that we’ve done, and will update their site as we do more. I think this is amazingly useful compared to searching through our blog posts to find what you are looking for. We appreciate Bryan and the staff at TDP for doing all the heavy lifting to make this possible.
64 Comments
Larry ·
When looking at the variability of a lens design, is it the case where if the lens has higher than average scores in the center it will have lower than average scores at the edges, or are the better lens samples equally better across the whole image? is there a pattern?
I guess another way of looking at this is what causes the most variability? Is it decentered elements or something else?
Brandon ·
Larry,
Any variance is caused by a decenter or tilt in each lens assembly. While no copy is every 100% the nominal design, if a lens scores over 6-6.5 or so, pretty much all copies are very close.
A decenter will cause coma in the middle of the image that will grow towards the edges. How it balances with the other aberrations in the lens is a question for each individual model.
Regards,
Brandon
CarVac ·
No Sigma 85/1.4?
Do you not have enough copies?
Brandon ·
CarVac,
Like the Nikon 20mm f/1.8 from the last batch, LensRentals carries very little stock of that model. While there are enough, that model is more of a long term project.
Regards,
Brandon
Anton Berlin ·
Lens Variance by Release Year
If you pull out the Canon data do you see a flatish linear trend line ?
The Zeiss 85mm’s make for a good apples to apples comparison – “made in the same factory by the same people” but why is the Otus consistency score 47.2% above the ZE?
James Thomson ·
“only one Zeiss falls below “6” on our scale”
Zeiss 21 2.8 5.2
Zeiss 85 1.4 5.3
Zeiss 35 1.4 5.7
Needs some proof reading 🙂
Link-up with TDP is great. Really makes the data much more useable. Looking forward to your future posts.
Could you include the year of release in the next summary table? It would be interesting to see how the various manufactures are changing with time.
Brandon ·
Anton,
The average MTF of the Otus is much higher (0.6774 vs .5306). The average standard deviation of the two is very similar, which is reflected in the plots which show absolute variance. The score itself is relative, favoring higher resolution lenses.
Regards,
Brandon
Anton Berlin ·
Brandon,
Could you please post average MTFs for all of the lenses and the formula for calculating?
What about omitting the Canon data from the lens year data ?
Andre Y ·
Great work guys!
What do you think about making your consistency score more independent of lens resolution? The subtext of the current consistency score is: what lens is likely to give me higher resolution. I guess that’s useful, but when I look at the consistency graphs (and not just the number), I am trying to figure out what lens will have a higher chance of resembling the sample that was measured or reviewed.
That’s a useful thing to know, because if you like what you read in a review (and this includes the subjective comments), then it’s good to know how likely you are to have a similar experience as the reviewer.
Separate consistency scores for center and corners might an interesting thing to compute as well. Perhaps break it up along the DX/APS-C boundaries.
Brandon ·
Anton,
The formula for the consistency score is in the introductory blog post to this. The average formula simply works like this:
IH = -20 -18 -16 -14 -12 -10 -8 -6 -4 -2 0
MTF= xxx xxx xxx xxx xxx xxx xxx xxx xxx xxx xxx
MeanMTF = Mean(MTF) i.e average center-to-corner
This is done in each of the sagittal and tangential planes separately.
The average MTF itself has no direct value and requires that I re-calculate each model individually, so I will not post it. Lots of lenses to test and software to write on my end…
Best,
Brandon
Brandon ·
Andre,
The consistency score contains a slight modification in favor of lower res lenses. Essentially the higher res lenses have their score weighted down about 5-10% by a formula. The “Absolute variance” can be obtained from the graphs, which plot a fixed +/- 1.5 standard deviations.
It makes more sense to us to use a relative score metric, since a difference of 20% at an MTF of 0.7 is entirely invisible, while a difference of 20% at an MTF of 0.3 is extremely obvious.
All the best,
Brandon
Randy ·
I would argue that the lenses with the greatest consistency are effectively a better value since a lot of consumers don’t have the time/patience/ability/access to test multiple lenses.
Also, my observation over the years has been that if you have a lens that’s exceptional (even if it’s not supposed to be) hang on to it, because the improved, updated replacement may not be as good.
Andre Y ·
Thanks for the explanation Brandon! One more question: will you guys offer the raw test data, so that we can compute our own metrics?
Brandon ·
Andre,
A better question for Roger, but all but certainly not. OLAF has 3 key assets – access to more lenses than anyone else, some rare test equipment, and a 100,000+ measurement database. Giving away the database does not make business sense.
Best,
Brandon
Andre Y ·
Brandon, understood. I’ll try to work from the graphs.
Carol Teater ·
Would be nice to see some mirrorless systems evaluated, especially m4/3, hint, hint…at the least, Sony FE, since they also are 35mm lenses, and compete with the ones covered in this series.
Roger Cicala ·
Carol, Sony’s mirrorless lenses are electromechanical, meaning they can’t be focused unless attached to a camera. Until we can make an electronic mount for them in the optical bench we can’t test them. We’re working on that actually, but it’s a lot of engineering.
Andre Y ·
Another question or perhaps speculation: how much of the Zeiss 2/135 and 85 Otus’s edge drop-off might be due to their strong vignetting? I’d guess that vignetting would reduce contrast, especially since these lenses are measured wide open where vignetting is at its worst. And perhaps another clue is that it’s their sagittal MTF that drops off rather than the tangential numbers, and vignetting tends to be a sagittal issue (ie. it changes intensity based on radial distance).
ARPL ·
I own, for instance, the EF 85mm f 1.2 version II.
From my understanding’ , I guess the Nikon’s graph looks by far the best.
Somewhat right with my understanding ?
However, would, for instance f.4 or f 5.6, not be moore/better telling ?
(when regarding this point.)
I mean, the widest apperture shurly is most likly to be least represenative of what it’s “average apperture” will be. No ?
Martin_MM ·
Roger, thank you for putting so much effort into this article and for publishing it openly. I have to say again that amongst all the lens-related websites, it is your blog that I come to read regularly and then go back more enlightened, amused and eager for more next time :-). Thanks a lot again!
nhz ·
First of all thanks for your efforts, very interesting topic 🙂
I’m wondering about a factor that could be skewing the variation measurements and didn’t really see it mentioned in the first article or this one, although maybe indirectly. Sorry if I overlooked something.
Do you take measurements from newly purchased lenses only, or do you just take lenses from your rental inventory? Some lenses may have been used/abused for years (even though you probably service them for more obvious problems). And possibly some older lenses have had some minor changes in mechanics and optics over the years that could influence variation in optical performance.
Is there any ‘reference’ for how much the performance changes when a lens is used frequently (I guess primes change less than zooms but who knows …).
Most of the very high performers in this group (and many of those in earlier comparisons) are very recent lenses. Part of the explanation will be improved optics and manufacturing tolerances, but maybe it is also because they are all ‘just from the assembly line’?
Roger Cicala ·
nhz,
We screen all new lenses on intake, and reject about 2 to 2.5% of them. The remainder go into stock and are retested after every rental. If they change optically, they are either readjusted by us or sent to the manufacturer if we can’t do it. Or both. The lenses tested in this series are not new in box, but all are recent purchases, less than 6 months old, have been repetitively tested. The data is, obviously, skewed because we’ve constantly screened out bad copies or changed copies.
In addition, we’ve previously done pretty large series comparing new in box lenses with lenses at various durations of their rental life and not seen any significant changes except in a couple of zoom lenses, none of which are included in this test series.
Roger
SSL ·
Thank you guys for confirming what my Master taught me 40 years ago:
For the same focal length, a brighter lens of one stop costing roughly double.
The money goes to higher material cost but more likely due to the tighter tolerance.
Your tests added one more element of modern design and manufacturing.
Great work!
winc06 ·
Have you ever tested the same lens multiple times to check the consistency of your methods? And if you have done that do you have routine checks to make sure that everything is operating up to standard before a test like this?
Roger Cicala ·
winc06, of course, on both points, constantly. One of the reasons for moving to an optical bench was consistency – target analysis like Imatest varies 3-4% with same sample, while the optical bench is around 1%.
JGro ·
I totally love this entire project of yours, kudos!
I’m also happy to see that with the EF100L 2.8 IS one of my favourite and most used lenses is doing so well.
Do you have any explanation for the quite significant difference between the Canon 85 1.8 and the 100 2.0? I always thought they were rather similar designs (and I am under the impression that the 100 2.0 is in general more highly regarded, in particular with less purple fringing).
Jim Thomson ·
Which Zeiss lens is less than “5” ?
Roger Cicala ·
Jim, sorry, miscorrected that. Fixed now.
Andre Y ·
Hmm, I answered my own question: it seems vignetting improves MTF readings: http://www.edmundoptics.com/technical-resources-center/imaging/sensor-relative-illumination-roll-off-and-vignetting/
Also, I confused the sagittal and tangential lines here, being used to solid lines being sagittal. Oh well, back to the drawing board.
Florian ·
Thank you for making these measurements available to the public.
I hope that you will be able to get the Sony Lenses to work soon.
Is there a good reason for not including Leica M or Zeiss Loxia Lenses in the test? A direct comparison between Mirrorless- and DSLR- Lenses would be very interesting.
Mike M ·
Lens Rentals, you could sell new and used lenses with your “certified” optical test numbers. You could add an additional fixed dollar amount onto the price for the testing cost. You could also add a variable cost onto exceptional performing copies. I would pay more (a little more, if my wife lets me) to be sure I’m getting a good copy.
Brandon ·
Andre,
Please email me at brandondube@gmail.com and I will discuss vignetting and how it impacts MTF with you – I fear you may have things a bit backwards and the topic is too long for a comments section.
Florian,
I have run quite a number of Leica lenses, but they are very expensive, don’t rent as often, and are made in much lower quantity. These factors together contribute to LR carrying far fewer of them. Most models we don’t have even 5 copies of to test. The Loxia lenses are for sony E mount and could be tested, but they are not something I have aimed for yet.
Regards,
Brandon
Carol Teater ·
Thanks, Roger, of course I didn’t even think of the electronic control versus mechanical control of focus. That is one of the downsides of most mirrorless lenses (though it does make me wonder how you tested the 85/1.2…).
Thanks for all you do! Love me some LensRentals! 🙂
Roger Cicala ·
Carol, the older electronic lenses, like the Canon, stay where you put them. So we can focus to infinity on a camera, then take it off and test it. Many of the micro 4/3 are that way too. The newer Sony ‘electromagnetic’ focusing, when you take it off the camera, just falls back to close focus. So to test them on the bench, we have to keep them electrically powered during the test.
Matteo ·
Oh Noooooo, I wish I could see even how the nikkor 135mm F2 DC behave compared to the other two 135mm
Andrew Krucko ·
Yet another informative article Roger. Thanks!
Peter ·
Very interesting study of an issue we all know about. You have pulled back the curtain in a very helpful manner, thanks!
But I am curious about a very obvious omission from your brand selection… there is a German producer that prides itself, usually with good cause, on its decades of excellent design, production and quality (assurance and control). You rent their lenses, some of which are notorious for whomping sample variation. Sadly, I have rented a couple of those examples of poor 28mm and 24mm from LensRentals, but then found other copies that were good to excellent.
My question is, Roger, what’s up with tests of your Leica M inventory?
Roger Cicala ·
Peter, what’s up basically is we don’t have 10 copies of most Leica lenses. There’s just not enough demand to stock more.
Peter ·
Roger, thanks for your explanation.
Nevertheless, I would love to know if Leica’s sample variation is any different than the other big names. And I am sure you would too! Perhaps you might run with a smaller sample of say 50/ 1.4, or 50/ 2 APO and bootstrap the error around Leica values to estimate your score value vs. the others.
I mean if any third party has the chops to do this comparison, it is your team.
Kim, Norway ·
This test confirms that there IS a point in seeking out the gems, ’cause they ARE out there! And don’t write off a lensmodel; you may have one of the duds.
Even if the manufacturers didn’t adjust their lenses at all, pure chance would produce a perfect sample now and then!
I own an old Nikkor DX 18-70mm that beats or equals all 17-55mm/2.8, 35/1.8, 50/1.4 and 50/1.8 I have compared it against on resolution. Should not be possible, but the resolution in that lens is just way better on all focal-lenghts than every other sample of that lens I have come across.
Moral: Hang on to the ‘perfect’ samples! You may never see one as good again.
Wilson Laidlaw ·
I suspect it is just as well you did not have Leica lenses in sufficient quantity to test variation. I expect it would have been pretty bad. Out of my four most recent Leica M lens purchases, two had to go straight back to the factory for adjustment and one was so bad it had to be replaced. Each one came with a signed QC certificate, where I can only assume that someone looked at it and said “yes it’s a lens.” It only took me a few seconds comparing the rangefinder focus point with the focus peaking confirmation to realise that these lenses were unusable in their as delivered state. Some Leica lenses are notorious for their sample variation, e.g. the 35mm ASPH Summilux, mostly for the degree of aperture shift. It makes you wonder about the benefits of hand assembly versus a mechanised production line, which the likes of Canon use.
John ·
Dear Roger
There is one thing about the Zeiss 135mm APO that I do not Understand. Can you please explain the reason why your MTF chart is so different from the one provided by Zeiss itself? As far as I know, contrary to Canon and Nikon, Zeiss’ MTF charts come from testing actual lenses and they are not just theoretical, and Your statistics also show very little variance between samples so yours cannot be far from what Zeiss has used for testing. Here is the link to the MTF chart provided by Zeiss.
http://www.zeiss.com/content/dam/Photography/new/pdf/en/downloadcenter/datasheets_slr/aposonnart2135.pdf
Roger Cicala ·
John,
We don’t know how Zeiss tests exactly, and there are a number of things that could be involved. My best guess is that when Zeiss measures they refocus at each point while we measure each point at best center focus. In our measurements field curvature will lower MTF since the off-axis points are not perfectly focused. (I personally think this is more real-world, but there are certainly people who disagree with me.)
Other possibilities: We use Photopic eye light for our measurements, but single wavelength light or complete white light are also options. Again, I don’t know what the Zeiss MTF machine does, but the Trioptics machine increases lighting as the lens vignettes. If the Zeiss MTF machine doesn’t, then the results would be different (not increasing lighting would give a higher MTF but might be more variable). Also our MTF number is an average of 4 rotations for each lens and then that average for 10 lenses. It may well be that the Zeiss MTF curves are one measurement per lens rather than 4, etc.
In other words, I don’t know for certain wy, but I’m not surprised that their’s are different.
Roger Cicala ·
John,
We don’t know how Zeiss tests exactly, and there are a number of things that could be involved. My best guess is that when Zeiss measures they refocus at each point while we measure each point at best center focus. In our measurements field curvature will lower MTF since the off-axis points are not perfectly focused. (I personally think this is more real-world, but there are certainly people who disagree with me.)
Other possibilities: We use Photopic eye light for our measurements, but single wavelength light or complete white light are also options. Again, I don’t know what the Zeiss MTF machine does, but the Trioptics machine increases lighting as the lens vignettes. If the Zeiss MTF machine doesn’t, then the results would be different (not increasing lighting would give a higher MTF but might be more variable). Also our MTF number is an average of 4 rotations for each lens and then that average for 10 lenses. It may well be that the Zeiss MTF curves are one measurement per lens rather than 4, etc. Finally, I think the Zeiss K benches measure 5 points compared to our 20, but I only have been told that, i don’t know for certain.
In other words, I don’t know for certain wy, but I’m not surprised that their’s are different.
Brett F ·
For the lenses with older designs, were the tested lenses from various manufacturing years or from a single manufacturing year? My assumption would be that, say, two EF 85 f/1.8s that were both manufactured in 2015 would be more likely to be similar in performance than two lenses built 10 or 20 years apart. Any thoughts on this?
Roger Cicala ·
Brett, all the lenses we tested were less than a year old, so I can’t really comment about older copies, although what you say seems logical.
Andreas D. ·
I am surprised that the Canon 24/2.8 IS and 28/2.8 IS have very different consistency scores (5.9 vs. 9.3). Any ideas on that?
Roger Cicala ·
Andreas, I was too, but no ideas why. Maybe when we take some apart we’ll get a hint.
Michael Stratil ·
Roger,
This is a magnificent study, and I would like to recommend that you try to publish it in an appropriate journal. Your method probably does not have a large enough sample to provide probability estimates of central tendency and dispersion, but maybe a journal would take it as an exploratory starting point for subsequent research.
Mike Stratil
Brandon ·
Mike,
I have pitched the idea of turning the data into some sort of academic paper to my research director at the university I attend. She is graduating about 9 PhDs or masters students in the coming months and is too busy to look into a new endeavor, but it is something we are looking into.
Best,
Brandon
Bill Peppas ·
Good work Roger & Brandon!
Would it be too much if I asked to see some zoom lenses added in the future ?
Id’ like to see the Nikon 14-24, 24-70, 70-200 and their counterparts from Canon, Tamron, Sigma tested along.
Thank you very much for the contributions and services to the photographic community guys!
Roger Cicala ·
Zooms are coming, Bill. Give us a couple of weeks.
Lee Saxon ·
I hope as this testing database grows, especially now that we’ve got it plugged into the Digital Picture backend which allows for aperture changes, that we see more apertures than just “wide open.”
I know that for time reasons there’s no way you could do every aperture on every lens. You’ve got to let renters have these lenses sometime! But I would add one (in some cases two) more apertures:
1.) One “stopped down” aperture a few clicks from “wide open.” I use 5.6 personally, but that’s subjective.
2.) “Matching” apertures. 85’s are a great example of why. You’ve got one 85/1.2 in a set of 85/1.4’s, and you point out that it makes a difference. That makes me want to see the 85/1.2 AT 1.4 – in fact, I’d rather see that than “wide open” (if time prevented you from doing both).
Brandon ·
Lee,
There’s about 12 different sets of stop down data I’ve taken so far. The goal is eventually to provide the MTF at all apertures, but it is absolutely out of the question to do variance at all apertures.
Within 2 weeks the first stop down data sets will go up. Nikon mount will never get it because we have no accurate form of aperture control, but EF mount, sony E, and micro four thirds are all possible.
Regards,
Brandon
Mike Ronesia ·
Two questions. When do you plan to add M4/3 lenses? And the big question, how do order one of the good ones when we rent a lens?
Roger Cicala ·
Mike, if they aren’t good we get them repaired and optically adjusted.
Derek ·
Another in this great series, many thanks.
I note you are testing at infinity and admit the Macro lenses may perform differently at marco… as many of these medium telephoto are used in portraiture, does anything change at around 5 meters focussing distance? does their performance peak at some focus distance other than infinity?.. perhaps a test of just one lens to prove the point… when you’ve got nothing better to do 🙂
Igor ·
The Otus is excellent in the center of the frame, competitive in the middle and poor at the edge (both resolution and astigmatism). Nothing like “from edge to edge”.
Jim Thomson ·
“I hate to get into brands because it makes the fanboys go insane”
As a certified tinfoil hat wearing, witch hunting, Canon fanboy, with confirmation bias, who throws out red herrings I think you got that just right.
JohnL ·
Quite amusingly (considering your table leader) I just traded in my Canon EF100 Macro for the L version and I’m on my third one (one a day). The first one wasn’t good on the right and had focusing issues at close range. The second one was just simply blurry down the left side. (Well, I guess that is some sort of consistency.) I have a third one but am losing the will to live, so having micro-focus adjusted it I’ll go with it if it isn’t that bad (the sides seem okay, not convinced about the focus consistency and it’s the noisiest of the bunch). I used FoCal to adjust it and it claims it has a lot of astigmatism, although it said the second one was great. Why do I buy lenses?
Oh and while I’m at it should a (fairly basic) lens quality rating (A-C, A-E?) be added to the variation score (X/Y say), so you can see if you care (some variation on a super-star lens being better than almost none on a so-so one)? At a minimum it would fill up the comments section 🙂
John ·
Dear Roger
I wanted to share my findings with you.I rented the Zeiss 135mm APO and I put it into some serious tests and I reached the conclusion that something should be wrong with your MTF chart.Your chart shows discernible separation between Sagittal and Meridionial lines which should lead to seeing lots of lateral chromatic aberration in the image.I could see some traces of bokeh fringing whereas the lens is advertised as APO but I didn’t see any lateral chromatic aberration in the photos I took with this lens.
Brandon ·
John,
A difference in the MTF of the tangential plane can be caused by lateral color, astigmatism, coma, eliptical coma, vignetting (limitation of measurement, not very relevant to the 135/2), or tilt.
“Apochromatic” correction means three wavelengths are brought to equal focus. This does not mean there is no “color distortion” which is what lateral color is. The two are distinct aberrations.
The level of quasi-astigmatism is “large,” but the MTF of the worse plane is still high enough such that a discrepancy would be extremely difficult to detect on a camera the lens is going to sit in front of, up to a 5DsR. Keep in mind that if the MTF is > 0.5 at 30lp/mm, the lens is “outresolving” the sensor though that is a dirty word and I hate to use it.
Regards,
Brandon
Turniphead ·
It seems logical (intuitively) that faster lenses should have more variation – the DOF is narrower, so any variation is more likely to be visible. If you get really bored at some point, try shooting testing a few manual f1.4 lenses at f2.8 and see what happens to the variation numbers. I’d be amazed if they didn’t improve to the levels of say an f2.8 lens of a similar focal length 😉
I’m still surprised by the amount of variation the Canon 14mm mk2 has though. Given the price (and Canon’s other recent lenses) I’d expect the variation to be significantly better than the Rokinon, but it isn’t…
dslr pundit ·
For faster lenses the depth of field is usually better (with a very few exceptions). Also when you go full frame, the bokeh quality improves greatly.
http://www.dslrpundit.com