Supertelephoto MTF Curves
Brandon has accomplished much during his summer with us, including several things I never thought we’d be able to do. One of those things was measuring the MTF of supertelephoto lenses on our optical bench. A vertical bench just isn’t designed to handle the mass of those big lenses and technically isn’t supposed to be able to test anything over 200mm focal length. Brandon found some workarounds that allowed us to get accurate MTF readings on a number of supertelephoto lenses.
Some of the workarounds are simple. For example, there’s a lot more vibration with a big, long lens. So after the machine moves to each measurement point we had to program in a long delay before doing the measurements. There is also a bench limitation that lightly affects the results. Our largest collimator provides up to a 100mm diameter beam of light for tests, so the 300mm f/2.8 lenses are at about f/3 and the 400mm lenses are at about f/4. There’s nothing we can do about that, Trioptics doesn’t even make a bigger collimator and if they did it would cost $100,000 by itself.
We weren’t able to do our full set of 10 copies of each lens there just weren’t enough in stock. There are at least 5 copies (Nikon 400 f/2.8), though, and most have 7 to 9 copies tested. But because there are less than 10 we won’t be putting out consistency numbers today.
Before we present these results to you, let me get one thing out of the way: we aren’t going to be testing many more of these. Brandon’s going back to school soon. Measuring these is too time-consuming, meticulous, and repetitious for me to want to do it. To put it in perspective, I could test 5 or 6 standard prime lenses in the time it takes to do one super telephoto run. So what I share today is likely to be all that’s going to be shared when it comes to measuring MTF curves on super telephoto lenses.
One other thing I should mention, you’ll notice some of these real MTF curves are fairly different from the manufacturer’s released MTF curves. (This is true for other lenses, too, but readily apparent for these lenses because the manufacturer’s MTF curves are so nearly perfect.) That’s the difference between a computer generated ideal MTF curve of the design, and an actual measurement of the manufactured product. The reality of the manufactured lens accounts for some of the difference, but also the reality of actual measurement (versus computer simulation) accounts for some of it too. In this case particularly, I want to emphasize that we’re using our MTF bench outside of its intended purpose, so I can’t be certain, especially at 400mm, that we aren’t introducing some other factors that might be affecting the measured MTF curves.
200mm Lenses
There is a lot of ways to get to 200mm. Most commonly, it’s a 70-200mm zoom shot at 200mm (we’ll release those MTF curves when we start releasing zoom lenses). Canon also has a relatively inexpensive 200mm f/2.8 prime lens. Both Canon and Nikon have very expensive 200mm f/2.0 primes. As always, when you look at these graphs, remember different apertures are comparing apples and oranges. A lens tested at f/2 would have higher MTF if tested at f/2.8, etc.
Just to give you a bit of orientation, I’m going to start with the MTF curves of two excellent lenses: the Canon 70-200 f/2.8 IS II at 200mm and the Canon 200mm f/2.8 L lens. These are both really good, although slightly different. The older 200mm f/2.8 L prime isn’t quite as good in the center at higher frequencies, but clearly does better off axis, which is what we would expect from a prime lens compared to a zoom.
Those were very good lenses, but the f/2.0 prime lenses are just spectacularly good.
Even at f/2.0, their MTF is clearly better than either of the f/2.8 lenses. Any fanboys who want to claim one is better than the other are being pretty silly. They’re different, sure, with the Canon 200mm f/2.0 IS L is a tiny bit better at higher frequencies, but having more off-axis astigmatism than the Nikon 200mm f/2.0 VR II. But both are totally amazing.
One other lens that I get asked about a lot is the Sigma 120-300mm f/2.8 HSM A1 zoom. I put its curve beside the Canon 70-200 f/2.8 IS II lens just for comparison purposes. I’ll be the first to admit I’m surprised at just how good the Sigma 120-300 f/2.8 Sport lens is. I put the Canon 70-200 f/2.8 IS II graph beside it to make the comparison easier and show just how excellent the Sigma is.
The overall the 200mm summary is pretty easy: they’re all damn good. And yes, I’m aware that everyone wants to know more about the 70-200 f/2.8 zooms. That post will be out in a week or two; I just put this one example up for a comparison point today.
Longer Focal Lengths
I started off the 200mm section using some examples that a lot of people were likely to be familiar with, and I’ll do the same thing here. Below is the MTF curve for the Canon 300mm f/4 IS lens. It’s an older design, but one that’s pretty popular. I’ll put it beside the Canon 200mm f/2.8 we used above to give you a comparison point. The 300 f/4 IS is a pretty good lens, but the absolute resolution isn’t quite as good as the 200mm f/2.8. While we aren’t presenting consistency numbers with these lenses, we did do 10 copies of the 300 f/4 IS and it did show significant variation (Consistency 4.2 for the 300 f/4 IS, compared to 8.9 for the 200 f/2.8).
The 300mm f/2.8 lenses are pretty expensive, and the MTF curves show some of the reason why; they are amazingly good. Like the 200mm primes, the Canon 300 f/2.8 IS II and Nikon 300 f/2.8 VR II they are slightly different, although the differences are minimal. The Canon, again, has a bit better center and sagittal resolution, the Nikon has less astigmatism away from center giving it better tangential resolution, but these are nit-picking, pixel-peeping differences. They’re both incredible.
The Sigma 120-300mm f/2.8 Sport at 300mm surprised me again. I knew it was very good, but in the center it’s every bit as good as the two name-brand prime lenses. It drops off away from center, obviously, but even there it’s still very good. That’s amazing performance for a zoom lens. Obviously, there is a lot more that goes into choosing a 300mm lens than just MTF curves, but the Sigma puts in an amazing performance.
400mm f/2.8
The 400m f/2.8 prime lenses are generally considered to be the absolute best lenses made. The MTF curves, which are at f/4, in my mind, suggest that they’re maybe a teeny tiny bit better than the 300 f/2.8s, but I’d be hard pressed to say I think you’ll see the difference in a picture. I do need to mention here that the aperture of the collimator in the MTF bench means these lenses aren’t being actually tested at f/2.8; the effective aperture is close to f/4.
So What Did We Learn Today?
Not much, really. I learned the Sigma 120-300 f/2.8 HSM OS is really good; better than I thought it was. A lot of you already knew that. You may have learned that manufacturer’s MTF charts are sometimes better than any actual lens is (no copy of the supertelephoto lenses we tested approached the published MTF, period). Other than that, this post just documents what we already know. Super telephoto lenses set the high end of the MTF bar. They are as good as photo lenses get.
Roger Cicala and Brandon Dube
Lensrentals.com
July 2015
36 Comments
Brian ·
I have to disagree with you, Roger, and say that the 300mm f/2.8s measure a little better than the 400mm f/2.8s. I wonder if the (effective) f/4 aperture is producing a little loss of resolution due to diffraction? Hard to draw a conclusion since you are so far out of the test equipment’s designed capabilities.
Brandon ·
Brian,
Here is a plot of one of the superteles vs spatial frequency with the diffraction limit: http://i.imgur.com/ecHmdKr.png
I believe the superteles are approximately diffraction limited as designed, but for whatever reason this is not the case as built.
The 200mm limit imposed by the bench is largely to do with magnification; the target projected at infinity for the lens has some angular size, which is magnified by the lens. On a 14mm lens 2mm on the sensor covers over 8 degrees, on a supertele it is a mere fraction of a degree. Eventually the target’s width becomes an issue for MTF calculation, but the bench doesn’t seem to be erroneous until past about 900mm or so.
I believe the biggest issue is the mass of the optics, the front couple of elements would pose quite the challenge to hold absolutely still even in transit. I believe that either they become decentered or tilted over time, the IS unit does not remain perfectly aligned over time, or the drop in filters develop alignment error over time. The astigmatism in most of these plots isn’t real per-se, but is an artifact of small, highly magnified tilts inside the lenses.
Regards,
Brandon
Anton Berlin ·
Brandon, will more diffraction limit charts (like the one referenced above) be made available ?
Brandon ·
Anton,
I don’t know, we’ve never talked about it since MTF vs spatial frequency is not a chart commonly used on the consumer side of the photo/video industry. It is certainly possible.
Jim Thomson ·
Any chance of doing the 400 mm f5.6L? It’s the only super telephoto prime that mere mortals can afford. It also has very few lens elements and I am curious about how that may affect consistency.
The consistency score for the 200mm F2.8 is remarkable for such an old lens.
Tony ·
Here’s a random thought. The 200, 300, and 400 primes have an interesting correlation. The ones that use fluorite elements have a bit more central resolution but more edge astigmatism. The new Nikkor 400 E is so different than it’s older siblings, yet it’s similar to the Canons. Maybe this is a ripple effect of using either fluorite or ED glass for the control of chromatic aberration; there will be different things to deal with “downstream” in the design.
Brandon ·
Tony,
Fluorite (CaF2) is functionally very similar to any “super ED” glass. They all have low refractive index (nD~1.35-1.45) and very low dispersion (Vd>80) and an anomalous partial dispersion. They are also all very fragile. The edge astigmatism in all of these is due to so-called build aberrations, not design faults. In the case of these lenses it is mostly caused by small tilts inside visible in the 4 quadrant reports for the various copies.
Here’s CaF2’s refractive index plot: http://refractiveindex.info/?shelf=main&book=CaF2&page=Li
Compare it to a super ED glass: http://refractiveindex.info/?shelf=glass&book=FK51A&page=SCHOTT
And compared to a “normal” ED glass: http://refractiveindex.info/?shelf=glass&book=BAK1&page=SCHOTT
And a highly dispersive glass: http://refractiveindex.info/?shelf=glass&book=SF11&page=SCHOTT
Note the axis changes.
-Brandon
Ala Fersht ·
Why does the Nikon 400mm have different values for sagittarial and tangential at the centre for 40 and 50 lp/mm? I thought they were always the same at the centre.
Brandon ·
Ala,
In aberration theory, Astigmatism is forbidden on axis by rotational symmetry. If you break symmetry with a decenter or a tilt, you can get any aberration on axis that shouldn’t be there =)
Alan Fersht ·
Brandon
For the MTFs of all of the shorter focal length lenses (all the 200mm or less, and basically true at 300mm), the sagittarial and tangential are the same at the centre. But, for the Nikkor 400mm that you have really serious deviations between sag and tan at the centre, and smaller ones for the Canon 400mm. It would seem very unlikely that the average of the Nikkors would have tilt or decentering (random tilts etc should cancel out in an average). Is it more likely that you have a systematic error because you are out of range of the equipment?
Brandon ·
Alan,
I do not believe it is a systematic error. These lenses required a close watch be kept on the measurements as they proceeded due to a high chance of the IS units settling in a different position from the vibration of spinning the sample. More so than other lenses.
Here’s one sample of the 400 Nikkor: https://www.dropbox.com/s/8nz3kr2ezhzttuc/MTF_Rotations.png?dl=0 and another: https://www.dropbox.com/s/ur1o3lttasb99uh/MTF_Rotations.png?dl=0
With sample 250241 it is clear from the MTF that there is a decenter and a tilt balancing each other. In the “0” orientation they are adding constructively, producing a large loss of resolution in the direction of the coma flare. In the “90” orientation, they add destructively and mostly cancel out. 244986 clearly has more coma (decenter) and less astigmatism (tilt).
The sample size for these is very small. We are at n=5. If we had the resources to measure to n=25 or n=50, they would mostly cancel out. In wider angle lenses with some field curvature the tilt makes one side worse and one better, so it cancels nicely in the average. The superteles have very flat fields, so tilt hurts both sides and brings the average down. The field of view is very small, so build aberrations do not have far to grow and it produces a more uniform loss of resolution.
“out of range” is arbitrary with this type of equipment. There is a compensation function in the MTF calculation based on the magnification, which is derived from the EFL. In the config of the ImageMaster this function is fit from a 10mm EFL to a 1000mm EFL. The 200mm limit suggested in the software is due to the spread between the two crosses used to measure on-axis EFL. A longer EFL will push them too close together for accurate computation. A second consideration is aperture; past 200mm you cannot measure large apertures, so they simply do not recommend measurement. Centering also becomes very difficult as the adjustment range limit is approached and exceeded and the reference “0” image height must be changed.
Regards,
Brandon
obican ·
#HireBrandon
Wally ·
CaF2 crystals are birefringent, if the lens has not been cut perfectly or ground perfectly some of this birefringence might show up in the actual lenses, while it would be excluded from the computer models as the lenses are normally orientated to eliminate this effect. Could this be effecting the astigmatism.
Brandon ·
Wally,
I am no expert in materials, so I may be wrong. However, the size of individual crystal cells of CaF2 is much smaller than the wavelengths that make up the visible band, so birefringence should not be an issue in visible light applications.
-Brandon
Tobias ·
Hi,
I know Brandon is going back to school, any chance of getting the 400 5.6 tested? It’s a lot more affordable and I’d like to see the real life trade-off =)
Thanks and keep up the great work!
Bill Slattery Jr ·
Wondering if you’re also amazed at the number for repairs you’re experiencing with the 120-300mm Sport as compared to the old model 120-300mm Sigma?
Roger Cicala ·
Bill, the repair rate for Sigma is better than it was a few years ago, although 120-300 Sports still have a fairly high rate. Given the mass of that lens I suspect it’s going to always have a tendency to suffer when it gets bumped around.
Wally ·
CaF2 lenses are single crystal, and the birefringence is very visible if the crystal is held in the correct orientation. Lenses are normally cut so that the lens is not birefringent on the optical axis, but if it is misaligned it can have a little bit of birefringence. The amount is very small at small angles. I am not saying this is the issue with the Canon lenses, I am a materials engineer but other than normal cameras have little experience with optics, so just bringing up the possibility.
Doug McEwen ·
Thank you for publishing such valuable data. Perhaps because of the extreme difficulty of testing supertelephotos on the optical bench, a few of these results don’t seem to correlate well with the ISO 12233 tests at TDP. For example, the EF 300mm f/2.8L IS II USM shows mediocre tangential MTF but I don’t see any astigmatism at all on the ISO 12233 results even with the 5DsR which looks flawless to my eyes. The same with the EF 200mm f/2L IS USM. But curiously, the EF 200mm f/2.8L II USM shows the same tan and sag MTF, which correlates well with the ISO 12233 results that don’t have astigmatism, so it seems unlikely that there is some kind of error in the optical bench tangential results at long focal lengths. I am not an optical expert, so perhaps just I just don’t understand something here.
Again, thanks for sharing such valuable data with the photographic community.
Brandon ·
Doug,
The absolute limit of detectability for the 5DsR is an MTF of 0.5 at 50lp/mm. Realistically anything as bad as 0.3 will look completely fine. The worst corner of the worst copy of the 300mm f/2.8L IS II is still better than even the 5DsR can detect.
Regards,
Brandon
Alan Fersht ·
As a comment has been made about the 300mm f/2.8 II data, I must add that Lensfreaks have published data from the Hasselblad factory equipment on the Canon 300mm f/2.8 II. Their measured MTFs for 20 lp/mm are nearly flat (~0.92-0.88 from centre to edge) as are those at 30 lp/mm (~0.82-0.75) for both sag and tan http://www.lensfreaks.com/lens-reviews/canon/canon-ef-300mm-f28-l-is-ii-usm-review-a-nice-little-beast/. They have measured only one lens, but the discrepancy is huge.
Have you done repeat measurements for the same lens and looked at the variances for those?
Roger Cicala ·
Alan, we measured 12 copies of the 300 f/2.8 IS II and the measurement variation was less than 2%. My guess is that the data you reference is measured at best focus for each point across the image plane. Or, if their bench was an Optikos bench, there is a setting that removes tilt. That’a valid way to do things, and pertinent if you want to see what the MTF would be at that distance if you chose, for example, and AF point at that position. However, if the lens is slightly tilted, as all of the 300 f/2.8 IS IIs we measured were, you would get an ‘idealized’ MTF.
Our measurements are all made at the best focus position for the center of the lens. This means a slight tilt puts the other points a bit out of focus and therefore MTF is lower. My own feeling is this is important in the real world since you can’t focus your lens differently at 20 points in one photograph. More practically speaking, since my purpose in testing is to find lens defects, however minor, seeing tilt is important to me. If we measured at best focus point for each position we’d have curves much closer to their curves. I’m just not interested in doing that.
Roger
Lasse Beyer ·
Roger,
when you put the Zooms on the bench, please do me a favor and test the Sigma 50-150mm f/2.8 APO EX DC OS HSM! Not sure how much you have in stock, but it’ll surely do 😉 .
The Sigma Art 18-35mm f/1.8 is on top of the list for sure, but just in case… please test it, too.
Regards,
Lasse
Andre Y ·
Brandon, how does one generally compute the limits of MTF resolution for a sensor? I noticed that you mentioned the 5DSR’s limit is an MTF of 0.5 at 50 lp/mm.
Brandon ·
Andre,
Keeping this a bit short, but the absolute limit of detectability is when two airy disks overlap more than 50%, for diffraction limited systems it is known as the rayleigh criterion. This correlates very closely to an MTF of 50%. The 5DsR has a pixel density of about 200px/mm, but you lose some due to the bayer filter and the non perfect MTF of the sensor itself.
Please email me if you would like more detail.
All the best,
Brandon
Andre Y ·
Thanks Brandon! That’s a good answer for me. I was just wondering if there’s some kind of formula for doing it, especially for estimating the loss from the Bayer CFA, but it sounds like perhaps this is empirically determined?
Brandon ·
The loss from the bayer array is approximately 50%. With sharpening one can achieve about a “100%” utilization of the camera’s resolution (e.g 4000lp/ph on an 8000px camera). However, for a non-monochromatic scene using a standard converter you can expect 50% losses. The loss from the sensor itself increases towards the edges due to angle of acceptance issues (1). An AAless camera will lose about 15-20% peak resolution in the center, as a baseline. To empirically determine it, it would be best to use a telecentric lens that is diffraction limited at a large f/number that is designed for the proper coverglass thickness. You will find these conditions can rarely be met and it must be extrapolated from many nonparametric data points found as resolution data from many high performing lenses.
(1) http://blog.teledynedalsa.com/2012/05/the-angle-on-optical-acceptance/
All the best,
Brandon
AJ ·
Would have been nice to see the older Nikon 300 F4 pitted against the new 300mm PF F4 VR.
Also the Nikon 70-200 F2.8 vs the F4 but understand that time is limited, especially when it comes to the more lengthy procedures for zooms.
Andre Y ·
Thanks as always Brandon!
Lee Saxon ·
Very interesting about the resolution limits of the sensors. I’m surprised to hear that they’re so low. If the sensor can’t resolve above 0.5 I don’t understand why we can see a difference between a lens measuring 0.6 and a lens measuring 0.8? I also wonder how much the bayer filter / debayering affects MTF (visually, we can see looking at Epic Monochrome vs regular Epic that the difference is significant)
Christian Nilsson ·
I was made aware of this thread and just want to comment on the procedure of the MTF measurements we use for making lens reviews at our website, http://www.lensfreaks.com.
We measure MTF about the same way as you guys, that is with best focus adjusted for the center of the lens throughout the whole measurement. We do not adjust focus for every point to obtain a best possible MTF value. That indeed would be a scenario not very well correlated to the actual photographic use of the lens.
We also have MTF at higher frequencies than 20 cycles/mm and they show about the same sagital and tangential behaviour. That is a rather flat curve all the way across the image field. I have uttermost respect for your work and knowledge but perhaps Alan F is correct. You may have some trouble with your equipment.
Best regards
Christian Nilsson
author at lensfreaks.com (and swedish site objektivtest.se)
Brandon ·
Christian,
The tilt in these lenses is real – the astigmatism is seen both on the bench’s target as well as on OLAF, our pinhole collimator. A significant factor at play is the fact that these are rental lenses and get shipped around far more than a lens fresh from the factory has been. The optics themselves are quite massive and it seems likely that something has shifted a bit in all that bouncing around.
An MTF bench performs a pretty simple function internally. It images a known object, adjusts the captured image based on some calibrations (for magnification, the microscope, the collimator, and background subtraction notably) and performs a FFT of the result.
For this to be in error one must deform the spot or break a calibrations. The calibrations remain correct for these focal lengths, all that is left is the spot. Short of clipping the field of view of the microscope / Region of Interest of the calculation, any error is down to the sample lens. All of these lenses were profiled for distortion and closely monitored during testing, with extra care to ensure measurement accuracy.
Best regards,
Brandon
Ilkka Nissilä ·
If the astigmatism (which to some extent is also present in the simulated curves for many lenses, though not so much for the fast supertele primes) is caused by tilted elements during shipping, how do the brand new lens samples perform compared to ones that have been rented many times? If there are large differences between a lens when new and after that same sample has been used in a while, the considerable financial investment involved in getting these lenses would not seem to be matched by expected long term performance. Expensive lenses like these should last with virtually unchanged performance for many years of use. Any comment? Also it seems that some lenses do not show much astigmatism e.g. the 300/2.8 Nikkor. Does this mean it is constructed more robustly (in terms of mechanical and optical tolerances and long term durability) or just not yet beat up as much as some of the others? Does the ultra light weight contstruction of some newer superteles lead to greater sensitivity to impacts? This is an important question as the lighter weight versions in many cases are substantially more expensive than their predecessors.
Brandon ·
Ilkka,
Lots of questions here that can’t be separated perfectly.
I do not know how new samples compare to old. The range of ages for e.g the Canon 300/2.8 (12 copies) is 1 month / 3 rentals to 1.5yr / 41 rentals. The oldest copy and the youngest one are not more different to each other than other copies in the sample in terms of MTF.
For a time I thought it was the drop in filters, since mechanically that is a very imprecise mechanism, or it was the mount being tilted under stress. But removing the filter altogether did not make a significant change to the astigmatism of a worse-off copy, and across the sample the tilts are completely random so it is unlikely to be the mount(which is itself supported on a 1/4″ thick steel plate).
I will say that even the worst corner of the worst copy is still far and away better than what even the highest resolution cameras can see, e.g the 5DsR or the 7D mk II or the D810.
Roger is better capable of answering how robust they are in comparison to each other, but they are all quite well made and I have no significant gripes with any of them, personally. The 300mm f/2.8 Nikkor does extremely well – it is actually the best lens I have tested for consistency. In absolute terms is variance is similar to the 135mm f/2 Apo-Sonnar, with significantly higher resolution which is quite a feat.
I do not know if the lighter weight models are more susceptible to damage than the heavier ones. The actual lens element mounting methods should be quite similar, but the barrels may be more deformable; or they may not be.
I think most of the tilt in these may come from the IS units, but it is impossible to tell.
Best regards,
Brandon
Helder ·
Roger,
Could you test some different super zoom primes? Like the sigma 300 2.8 or other budget primes.
I’m really curious.
Roger Cicala ·
Helder, I won’t say never, but probably never. It’s way, way too time consuming and difficult.