Geek Articles

Developing a Rapid MTF Test for Photo and Video Lenses

Published June 29, 2018

. . . Or What I Did for the Last Year Instead of Having a Life.

Nearly a decade ago, I started testing lenses. A few years later, after I complained about a manufacturer’s claiming an awful lens was in spec, an engineer patted me on the head and told me “you really don’t have the background to understand metrology.” Those of you who know me realize I didn’t take this well.

Following the principles of ‘Anything Worth Doing is Worth Doing to Excess’ I changed my name to Inigo Montoya and went on a quest. It’s taken countless hours and dollars, but I get immense satisfaction knowing that for the last two years that same manufacturer has been paying me to test their lenses. Not that I hold a grudge or anything.

For the last year, it’s gone to a new level. Some of you have probably noticed a lack of blog posts from me lately, so I thought I’d tell you why.

WARNING! THIS IS A GEEK LEVEL III ARTICLE.  It’s about testing, not image making. If you aren’t a lens geek, there’s no reason to read further, but worth reading if you want to learn something.

Developing a Rapid MTF Test for Photo and Video Lenses

A Brief History of My Life

Over the last decade, we have steadily increased the level and complexity of our optical testing. We started with computer analysis of chart images but found the process was slow and too variable for our purposes (results for the same lens could vary by 5% with slight changes in alignment or lighting).

An MTF (Modulation Transfer Function) bench gave us much more accurate and reproducible results. Pretty soon, though, we found that a standard MTF (measuring from the center to one side) missed a lot; lenses are not really rotationally symmetric (the left side may not be the same as the right side, etc.). Some lenses look fine in one area but are not fine in others. In other words, testing 1/4 of the lens was not a good test.

Olaf Optical Testing, 2016

 

So, we developed a 4-rotation MTF tests; testing lenses from one side to the other at four different angles. This lets us look at all four quadrants of the lens, and we stopped missing lenses that were soft on one side or in one corner. We tried 8 and 12 rotations, too. They took a lot longer and didn’t detect anything that four rotations missed.

Olaf Optical Testing, 2016

 

Now we had very accurate testing, but looking at four rotation MTF graphs like the set above can’t really be described as intuitive. So some of our guys developed Full Field Display plots that showed us the information accurately, but much more intuitively.

Olaf Optical Testing, 2017

 

Now we had testing that did a great job of showing us when a lens wasn’t quite right, but it didn’t always show us WHY a lens wasn’t quite right. For example, we couldn’t tell if one side of the lens was soft because of optical decentering, or because of field tilt.

More importantly, we’ve found that about 2% of lenses that looked fine on chart or projector testing are not really fine. Why does that matter? Because when a customer takes critical images of 3-D scenes, especially on today’s high-resolution cameras, the customer will see defects that 2-D testing missed. And that customer is pissed.

That led us to start doing MTF vs. Field vs. Focus testing (AKA Field Curvature). Doing field curvatures showed us a lot more because it looked at how the lens resolved in 3 dimensions rather than the two dimensions of a test chart or lens projector.

Test charts and projectors could tell us a lens was soft on one side or the other. Testing the field curvature let us determine if the softness was because the field was tilted.

Olaf Optical Testing, 2016

 

Or because the lens was decentered.

Olaf Optical Testing, 2016

 

It also let us determine if a lens normally had field curvature.  In the example below, the left side shows little field curvature, but the lens gets soft away from the center. In the right example, the lens stays sharp away from the center, but the field curvature makes it sharp at a different focusing distance than ‘best focus at center.’ On a standard MTF test, which is done at best center focus, they look equally soft away from the center. If you take images of a 3-dimensional scene with them, they look entirely different.

Olaf Optical Testing, 2015

 

Somewhat to our surprise, we also found that some ‘bad’ copies of a lens had different field curvature than was normal for that lens. Not tilted or decentered, but an entirely different shape than most lenses of that type. So checking for normal field curvature is yet another test that gives us useful information.

Field curvatures were a major addition to our testing, but man are they slow. It takes 15 minutes or so to do a 4-rotation MTF graph. It can take 30 minutes to do just one slice of field curvature. Four rotations of field curvature requires more time and patience than we have.

So, Here We Were

After years of refining our testing techniques, we could accurately determine how well a lens resolved across the entire field of view. We could determine if the field was curved or tilted.  And we could tell if the lens suffered from some form of optical maladjustment.

It was far more accurate than any type of chart or projector testing. It was, as best we could determine, the most detailed testing of lenses available anywhere.  It was written up in peer-review journals and presented at optical society meetings. (1, 2)  Several manufacturers and end-users like Caltech/NASA Jet Propulsion Laboratory used us to test their lenses in this way.

But, thoroughly testing a single lens like this takes an hour or more. Even doing just the 4-rotation MTF test took at least 15 minutes. That’s an eternity when you need to test hundreds of lenses a day. Many hundreds of lenses.

So, we tested lenses on the bench periodically but relied on test charts and lens test projectors for everyday testing. There was just no way to test hundreds of lenses on the optical bench every day.

But as camera resolution increased, standard testing had trouble keeping up. We knew this was coming and have been working for several years to figure out some way to get high-resolution testing that could be done rapidly. We didn’t quite make it in time, but we made it nearly in time (see addendum).

So What Did We Do?

We start with the simple test that is the basis for MTF vs. Field vs. Focus; a test called the Through-focus MTF. Let’s look at another MTFvFvF graph. The horizontal axis is the lens field from one side to the other, and I’ve drawn a black line at the center of the lens’ field. The vertical axis is focusing position and “0”, in the middle of the vertical axis, is the proper focusing distance for the lens we’re testing.

Olaf Optical Testing, 2018

 

Through focus MTF starts out of focus, measures the MTF, moves forward a few microns, tests it again and repeats this starting at ‘back focused’ and going to ‘front focused,’ making MTF measurements along that black line in the center position. The ‘through focus MTF’ at the center would look like this:

Olaf Optical Testing, 2018

 

You can see the highest MTF is right where it should be, at 4.2mm (that’s the bench’s position for Canon mount lenses). The MTF there is 0.75 for both sagittal (solid line) and tangential (dotted line) rays. Away from that point, the MTF gets lower and lower because the lens is more and more out of focus.

So, Through Focus MTF tells us what the highest MTF at that point is, and at exactly what focusing distance it occurs.

If we repeat this away from the center, the graph looks different and actually gives us more information. The graph below is from the same lens at 14mm away from the center (about ¾ of the way to the edge of the image). You should notice the highest MTF is now about 0.55; like most lenses, the MTF is lower this far from the center.

 

You might also have noticed that the highest MTF (black arrows) is not at 4.2mm now. The sagittal best focus is at 4.23, the tangential at 4.18. This is astigmatism; the sagittal and tangential lines don’t focus at the same point. If there was field curvature, both peaks would be on one side or the other of the best center focus (4.2mm).

How do we decide if that’s OK or not? We’ve tested a couple of dozen copies of this lens and determined the range of what’s acceptable. For every lens, we know how high the MTF should be at the center and near the edge, how much astigmatism the lens is supposed to have, and how much field curvature (if any) there should be.

The above graph showed the lens at just one side of center. To get a more complete picture, we rotate the lens on the bench, testing it in each quadrant. Now we have done Through Focus MTF at 5 locations, the center and near each corner.

 

If we plot all of this on one graph, it looks a bit more complicated at first glance.

Olaf Optical Testing, 2018

So, what all do we see here? First, I’ve pointed to the center MTF peak with the short black arrow. To the left of that, the red arrow points to the tangential MTF peaks (there are 4 of them, but the yellow one doesn’t show up very well). To the right, the green arrow points to the sagittal peaks. I’ve measured the focusing distance between the sagittal and tangential peaks, which gives us a numeric value for astigmatism. I didn’t show it on this graph, but we calculate out the difference of each off-axis peak from center focus and from each other as well.

Each type of lens is somewhat different, but as mentioned above, we know what to expect for each one. For each lens, we program a set of pass-fail criteria that looks something like this:

Lens X

  • Center MTF > 0.68 (Sharpness)
  • Off-axis sagittal MTF > 0.55 at all 4 locations (Sharpness)
  • Off-axis tangential MTF > 0.5 at all 4 locations (Sharpness)
  • Sagittal-Tangential peak difference < 0.2  (Astigmatism)
  • Largest Sagittal-Sagittal peak difference < 0.5 (Tilt)
  • Largest Tangential-Tangential peak difference < 0.5 (Tilt)

In the center and each corner, we’ve tested the lens for adequate sharpness, amount of astigmatism, and amount of tilt. The above lens not only passes, but it’s also a very good copy. It is theoretically possible that a lens could be OK in all four corners and the center, but have a problem in the mid portion, but we’ve never seen it, and we’ve tested many thousand lenses.

As you probably suspect, instead of plotting out a graph we actually send the numbers to a computer program that compares them to the known acceptable level for that lens and gives it a pass-fail.

Not all lenses look quite as pretty as the one above. Below is a different lens, with different pass-fail criteria. This lens is a little weird, but the same principles apply. The center MTF is pointed to by the green arrow. The center MTF isn’t much higher than the off-axis MTF, which is normal for this lens (a telephoto prime).

You can see all of the off-axis points are to the right side of the center peak, showing this lens has field curvature. The solid and tangential lines are separated a bit, showing there is also a small amount of astigmatism.

Olaf Optical Testing, 2018

 

But the most critical thing in this graph is that some of the dotted (tangential) lines are not focusing at the same place as the others. I put red arrows on the two that are furthest apart and measured the distance between them. (Again, the yellow peak doesn’t show very well, but the computer reads the numbers, not the graph.)

This shows a field tilt. This particular tilt only affects the tangential lines, not the sagittal, which is actually somewhat common. This is a pretty standard amount of tilt for this lens, and it passes easily. My point here is that we are checking the lens in 3 dimensions and getting numbers that show field tilt as well as absolute sharpness.

To give you a better idea of how it works, I’ve put thumbnails of 4 copies of a wide-aperture lens in the image below. The top two lenses are perfect. There’s the normal amount of field curvature (all of the off-axis graphs peak at a different place than the center but they’re together) for this lens.

The bottom left graph passes, but just barely. You can see the center sharpness isn’t quite as good, and there is some tilt in the tangential field, but it’s minor. The lens on the lower right fails because the tilt is very severe.

Olaf Optical Testing, 2018

I’m not trying to show you how to read these graphs, but rather just showing that when a lens isn’t right, it’s pretty apparent using this technique.

So How Good Is It?

We had three goals for this project: to test quickly, to test accurately, and to test the majority of lenses.

The quickness is good enough, although we hope to improve it more. It’s fast enough that we can use an optical bench for daily testing, which we could never do before. Of course, you have to buy an optical bench and have a lot of lens data to do that, but the improved accuracy makes it worth the cost in time and money.

The accuracy is superb. We tested hundreds of lenses using this technique during the debugging phase. Any lens that didn’t clearly pass got the complete one-hour bench test. In the above example, the lower two lenses both got the complete test, which confirmed that the lower left passed and the lower right did not.

It’s more accurate than our very experienced techs testing with the best chart setups made. The techs are about 98% accurate. The Rapid MTF test is about 99.5% accurate. But the important difference is the Rapid MTF test misses 0.5% by saying the lens fails when it actually passes. We’d much rather take 0.5% of lenses out of stock for further testing than to put lenses back in stock that shouldn’t be there.

We can’t test all lenses – yet. This was a trial run to see if the technique really worked (many people I talked to didn’t think it would). Setting it up was time-consuming; we have to write an automation script for each lens. If the lens comes in several mounts, each mount requires a separate automation script. (For those of you who’ve wondered why there haven’t been many blog posts lately, we’ve written and debugged over 200 scripts, tested multiple copies for each script, and then done the full long set of MTF tests on any lens that was questionable. It’s taken 2 of us working full-time for nearly three months to do that.)

Right now we’re using this technique to test all prime lenses; both photo and video. We’ll continue with just prime lenses for several months before expanding into other lenses. More software has to be written, more techs have to be trained to operate the optical bench, and workflows streamlined before we buy more benches and start testing zooms. We’re hoping some tweaks and improvements will speed things up a bit, too, before we order another bench and expand to other lenses.

We expect, though, that within a year this will be our standard testing for almost all lenses. I say almost all because there are some hurdles to overcome before we can do lenses of 400mm focal length and up, and some lenses with very small apertures (12mm f/4 zooms, for example).

Roger Cicala and Aaron Closz

Lensrentals.com

June, 2018

 

Addendum: I say didn’t quite make it in time because our experience is chart and projector testing are adequate for lenses that will be mounted to cameras up to the equivalent of about 30 megapixels full-frame resolution. When you put a lens on a 50-megapixel full-frame camera the charts and projectors become the rate-limiting factor; the chart tests can look fine, but a photo on the camera may show a problem the testing missed. I’ve written about this before

Author: Roger Cicala

I’m Roger and I am the founder of Lensrentals.com. Hailed as one of the optic nerds here, I enjoy shooting collimated light through 30X microscope objectives in my spare time. When I do take real pictures I like using something different: a Medium format, or Pentax K1, or a Sony RX1R.

Posted in Geek Articles
  • Michael Steinbach

    Anything to keep Zack away from the keyboard! ?. More interesting to me would be which lens companies have you testing for them…. can you share?

  • Michael Parchomovsky

    Great job! Thanks for sharing your technique. Love the MTF 1D, 2D (field) and 3D (through-focus) plots 🙂
    What kind of contrast target are you using in the process of the four different orientation MTF analysis?
    P.S. your review “The (sort of) Great 400mm Lens Shootout!” was very useful to me. Thank o lot!

  • Well, Brandon answered the actual question better than I could. I’ve often wondered myself whether my approach is best logically (my thinking is a large population of unknowns is best evaluated in the life sciences methods) or simply that I’m a hammer and everything I see looks like a nail 🙂

  • It’s two things. The angle the collimator moves too for off-axis testing is longer the wider the focal length (takes longer to tilt 40 degrees than 10 degrees) is the lesser. The major is the smaller the absolute aperture (roughly focal length / f stop) the slower the through focus test takes. We’re working with the bench manufacturer to look at options for more powerful light or better microscope cameras to speed that up.

  • Decentering shows up too. Look at the second set of field vs focus graphs. The same lens, on the right is decentered, on the left after decentering corrected.

  • Lorenz Ammon

    Great to have you back on your blog, Roger! Thanks for sharing your work. What about decentering? Would decentering mostly register as tilt in your new test? Is decentering a less common problem than tilt that you don’t mention it in your article?

  • David Bateman

    Thank you Brandon and Roger.
    Sorry, I did not mean to derail the conversation here.
    Thinking about this the other way. With lens tolerance, glass composition, coating thickness and various elemental impurities, the best maximum value may be totally meaningless and might just be 42 across the board. What I now just got for your Canon test, funny eh.
    The reason why I come here in the first place is the actual in use values for multiple lenses. So your way probably is better than an ideal unattainable comparitive value.

  • I always assumed that my physics profs purposely obfuscated the material in order to get people to quit the major…

  • I yelled at my computer “OH YAY A ROGER ARTICLE!!!” when I saw this today. Good to have you back!

    What limits of the speed of the bench tests? Do you have to spend a ton of time waiting for vibrations to damp between measurements or something? I’m guessing it isn’t something that can be solved by a) cranking up the light through the lens, or b) throwing CPUs/GPUs at the image analysis pipeline.

  • Brandon Dube

    By “abuse” I mean, “clone someone’s product with impunity because they can get the patent thrown out.” That hasn’t happened, outside of perhaps China, where IP law, at least for foreigners, doesn’t really exist anyway.

  • David Bateman

    Well we are close to the abuse phase. Companies have started law suits with Chinese manufacturers based on outer appearance and some even on lens design. So maybe its starting.
    Your data may just put a nail in the cofin an design cases and would keep the arguments down to only outside appearance of a lens.
    I am thinking of the recent Zeiss vs Samyang law suit. Solely based on appearance.

  • Brandon Dube

    The entire optics industry works that way, and collectively has, I don’t know, a million lens design patents? If it was a legal issue, someone would have abused it by now.

    The patent is not for “85mm f/1.4 lens.” It is usually for “large aperture medium field of view image stabilized optical device,” or somesuch. Whether their product matches that is not very important to the patent (and its validity).

  • David Bateman

    Interesting and sad at the same time.
    I guess gone are the days of having the lens design in the manual.
    Thank you, I will have to think what this means for the patent now as if the product doesn’t match they may not have a litigation case and may even be voiding their own patents. As in making it not hold up in court.

  • Brandon Dube

    Most of them do not have identified patents, and of those that do the overwhelming majority have intentional errors in the patent such that they do not image well, or even raytrace when entered into optical design software.

    I have only entered a single DSLR lens patent into Code V and had it perform at all like what the manufacture produces. I assume for that one, they forgot to screw up some values.

  • David Bateman

    Not sure if your serious or joking?
    Most are published and available in the online patent data bases. Since you are not building your own lenses, copying other manufacturers patents, I don’t see a problem.
    What you would be doing is holding a manufacturer up to an expected standard based on their very own patented expectations. So I see no grounds for a law suit, but I am not a lawyer.

  • Brandon Dube

    If we had the optical designs for all of these lenses, each of the manufactures would have sued us into the ground by now.

  • David Bateman

    Excellent to read your posts. I have missed your science!
    Reading this its clear to me you’re from the medical field. Which I did know. As in, you tested a whole bunch of lenses or people. Got an expected average, and then use that to compare everything else. This may work, but with lenses can’t we do better?
    Can you look at the lens design, compute a maximum theoretical best expected performance, and then rate all following lenses to that best output based on the lens design? This should not only save time but give you a better metric.
    I just don’t know if it can be done.

  • Y.A.

    Man, this is big. So is this going to factor into the business model? For example can I pay more for a better copy of a lens (or less for a so-so copy that still passes) like KEH? Or are the differences not even noticeable in actual use? Would be cool if nothing else to have a range of MTF data available for each lens. Man between the data management and script writing this all sounds like a ton of fun. Right up my alley. Keep up the good work and thanks!

  • Athanasius Kirchner

    I missed your posts, but since it was all in the pursuit of a lens testing method to end all lens testing methods, it’s all good 😀
    Hopefully some manufacturers will start using the same methodology, while paying you guys a princely sum for the wonderful IP?

  • You have no idea how much that made my day! Thank you.

  • John, actually, the new machine might get your wish. We’ve been testing M4/3 primes. The testing side can’t justify the cost of a Fuji mount for the bench (couple of thousand dollars) but Lensrentals testing everything is going to make that happen. Pentax too, probably.

  • hjwulff

    Great to have you and your articles back. Can I say ‘I’ve missed you’?

    In any case, it’s great to have logical, well reasoned information on the internet and in a blog. I’m slightly knowledgeable in the field of optics, but your efforts to present practical but somewhat technical information in an easily understandable but not dumbed down fashion is exemplary. My physics profs should have been as good. Thanks.

  • John Dillworth

    Science!!! I’m excited to see the results. Any chance of some love for Fuji and M 4/3 primes? I know these are not your big rentals but one can dream.

  • Andre, that was totally clear and totally correct. So 0 degrees takes a side-to-side cut, 90 degrees is top to bottom, and 45 and 135 degrees go (roughly) corner-to-corner.

  • OK, I get that. Let’s say we start at focus position 0 on the MTFvFvF graph (the yellow/purple/orange contour map), and as we walk across it from -18 to +18mm, which part of the lens’s field is being measured? I’m assuming it’s the center horizontal line of the lens. So when you rotate the lens by say 45 degrees, are you now scanning from one corner of the lens to the opposite corner? (As seen from the perspective of the camera mount.) Hope this is clear: I wish I could wave my hands in front of you as I’m trying to explain what I mean! 🙂

  • Andre, I’m not certain I understand which graphs you mean. But if I understand correctly, the curve for center lines basically goes from backfocus, through best focus and on to front focus. In the MTFvFieldvFocus graph if you follow along the black line, the colors correlate with the height of the Through Focus graph: yellow areas are highest MTF dark purples the lowest.

  • Very cool stuff Roger and Aaron! One thing that’s not clear to me is which part of the lens’s field do the 5 sets of curves represent? For example, for the set of curves for the center, it looks like you’re sampling along a line parallel to the longer length of a full-frame field and that line goes through the center of the frame? Are the corners measured diagonally across the frame since you rotate the lens?

  • I like that idea. I actually reread one of those historical posts I wrote years ago and was like ‘wow, this is pretty good!’

  • Clayton Taylor

    Awww…..I was thinking that you were either on vacation on a faraway island or creating a new company to market “No Minivan Lenses” t-shirts. Instead, you are creating an entire new industry. Well done! I await the forthcoming reports.

    On a side note, I was telling a new photo acquaintance about your blog, especially your older posts about photographic history, lens designs and designers, etc. When I went to show her, those old posts are waaay back in the archives. Suggestion – during those weeks when you do not have any other modern posts scheduled, could you re-post some of Roger’s Classics? Thanks!

Follow on Feedly