Geek Articles

Developing a Rapid MTF Test for Photo and Video Lenses

Published June 29, 2018

. . . Or What I Did for the Last Year Instead of Having a Life.

Nearly a decade ago, I started testing lenses. A few years later, after I complained about a manufacturer’s claiming an awful lens was in spec, an engineer patted me on the head and told me “you really don’t have the background to understand metrology.” Those of you who know me realize I didn’t take this well.

Following the principles of ‘Anything Worth Doing is Worth Doing to Excess’ I changed my name to Inigo Montoya and went on a quest. It’s taken countless hours and dollars, but I get immense satisfaction knowing that for the last two years that same manufacturer has been paying me to test their lenses. Not that I hold a grudge or anything.

For the last year, it’s gone to a new level. Some of you have probably noticed a lack of blog posts from me lately, so I thought I’d tell you why.

WARNING! THIS IS A GEEK LEVEL III ARTICLE.  It’s about testing, not image making. If you aren’t a lens geek, there’s no reason to read further, but worth reading if you want to learn something.

Developing a Rapid MTF Test for Photo and Video Lenses

A Brief History of My Life

Over the last decade, we have steadily increased the level and complexity of our optical testing. We started with computer analysis of chart images but found the process was slow and too variable for our purposes (results for the same lens could vary by 5% with slight changes in alignment or lighting).

An MTF (Modulation Transfer Function) bench gave us much more accurate and reproducible results. Pretty soon, though, we found that a standard MTF (measuring from the center to one side) missed a lot; lenses are not really rotationally symmetric (the left side may not be the same as the right side, etc.). Some lenses look fine in one area but are not fine in others. In other words, testing 1/4 of the lens was not a good test.

Olaf Optical Testing, 2016


So, we developed a 4-rotation MTF tests; testing lenses from one side to the other at four different angles. This lets us look at all four quadrants of the lens, and we stopped missing lenses that were soft on one side or in one corner. We tried 8 and 12 rotations, too. They took a lot longer and didn’t detect anything that four rotations missed.

Olaf Optical Testing, 2016


Now we had very accurate testing, but looking at four rotation MTF graphs like the set above can’t really be described as intuitive. So some of our guys developed Full Field Display plots that showed us the information accurately, but much more intuitively.

Olaf Optical Testing, 2017


Now we had testing that did a great job of showing us when a lens wasn’t quite right, but it didn’t always show us WHY a lens wasn’t quite right. For example, we couldn’t tell if one side of the lens was soft because of optical decentering, or because of field tilt.

More importantly, we’ve found that about 2% of lenses that looked fine on chart or projector testing are not really fine. Why does that matter? Because when a customer takes critical images of 3-D scenes, especially on today’s high-resolution cameras, the customer will see defects that 2-D testing missed. And that customer is pissed.

That led us to start doing MTF vs. Field vs. Focus testing (AKA Field Curvature). Doing field curvatures showed us a lot more because it looked at how the lens resolved in 3 dimensions rather than the two dimensions of a test chart or lens projector.

Test charts and projectors could tell us a lens was soft on one side or the other. Testing the field curvature let us determine if the softness was because the field was tilted.

Olaf Optical Testing, 2016


Or because the lens was decentered.

Olaf Optical Testing, 2016


It also let us determine if a lens normally had field curvature.  In the example below, the left side shows little field curvature, but the lens gets soft away from the center. In the right example, the lens stays sharp away from the center, but the field curvature makes it sharp at a different focusing distance than ‘best focus at center.’ On a standard MTF test, which is done at best center focus, they look equally soft away from the center. If you take images of a 3-dimensional scene with them, they look entirely different.

Olaf Optical Testing, 2015


Somewhat to our surprise, we also found that some ‘bad’ copies of a lens had different field curvature than was normal for that lens. Not tilted or decentered, but an entirely different shape than most lenses of that type. So checking for normal field curvature is yet another test that gives us useful information.

Field curvatures were a major addition to our testing, but man are they slow. It takes 15 minutes or so to do a 4-rotation MTF graph. It can take 30 minutes to do just one slice of field curvature. Four rotations of field curvature requires more time and patience than we have.

So, Here We Were

After years of refining our testing techniques, we could accurately determine how well a lens resolved across the entire field of view. We could determine if the field was curved or tilted.  And we could tell if the lens suffered from some form of optical maladjustment.

It was far more accurate than any type of chart or projector testing. It was, as best we could determine, the most detailed testing of lenses available anywhere.  It was written up in peer-review journals and presented at optical society meetings. (1, 2)  Several manufacturers and end-users like Caltech/NASA Jet Propulsion Laboratory used us to test their lenses in this way.

But, thoroughly testing a single lens like this takes an hour or more. Even doing just the 4-rotation MTF test took at least 15 minutes. That’s an eternity when you need to test hundreds of lenses a day. Many hundreds of lenses.

So, we tested lenses on the bench periodically but relied on test charts and lens test projectors for everyday testing. There was just no way to test hundreds of lenses on the optical bench every day.

But as camera resolution increased, standard testing had trouble keeping up. We knew this was coming and have been working for several years to figure out some way to get high-resolution testing that could be done rapidly. We didn’t quite make it in time, but we made it nearly in time (see addendum).

So What Did We Do?

We start with the simple test that is the basis for MTF vs. Field vs. Focus; a test called the Through-focus MTF. Let’s look at another MTFvFvF graph. The horizontal axis is the lens field from one side to the other, and I’ve drawn a black line at the center of the lens’ field. The vertical axis is focusing position and “0”, in the middle of the vertical axis, is the proper focusing distance for the lens we’re testing.

Olaf Optical Testing, 2018


Through focus MTF starts out of focus, measures the MTF, moves forward a few microns, tests it again and repeats this starting at ‘back focused’ and going to ‘front focused,’ making MTF measurements along that black line in the center position. The ‘through focus MTF’ at the center would look like this:

Olaf Optical Testing, 2018


You can see the highest MTF is right where it should be, at 4.2mm (that’s the bench’s position for Canon mount lenses). The MTF there is 0.75 for both sagittal (solid line) and tangential (dotted line) rays. Away from that point, the MTF gets lower and lower because the lens is more and more out of focus.

So, Through Focus MTF tells us what the highest MTF at that point is, and at exactly what focusing distance it occurs.

If we repeat this away from the center, the graph looks different and actually gives us more information. The graph below is from the same lens at 14mm away from the center (about ¾ of the way to the edge of the image). You should notice the highest MTF is now about 0.55; like most lenses, the MTF is lower this far from the center.


You might also have noticed that the highest MTF (black arrows) is not at 4.2mm now. The sagittal best focus is at 4.23, the tangential at 4.18. This is astigmatism; the sagittal and tangential lines don’t focus at the same point. If there was field curvature, both peaks would be on one side or the other of the best center focus (4.2mm).

How do we decide if that’s OK or not? We’ve tested a couple of dozen copies of this lens and determined the range of what’s acceptable. For every lens, we know how high the MTF should be at the center and near the edge, how much astigmatism the lens is supposed to have, and how much field curvature (if any) there should be.

The above graph showed the lens at just one side of center. To get a more complete picture, we rotate the lens on the bench, testing it in each quadrant. Now we have done Through Focus MTF at 5 locations, the center and near each corner.


If we plot all of this on one graph, it looks a bit more complicated at first glance.

Olaf Optical Testing, 2018

So, what all do we see here? First, I’ve pointed to the center MTF peak with the short black arrow. To the left of that, the red arrow points to the tangential MTF peaks (there are 4 of them, but the yellow one doesn’t show up very well). To the right, the green arrow points to the sagittal peaks. I’ve measured the focusing distance between the sagittal and tangential peaks, which gives us a numeric value for astigmatism. I didn’t show it on this graph, but we calculate out the difference of each off-axis peak from center focus and from each other as well.

Each type of lens is somewhat different, but as mentioned above, we know what to expect for each one. For each lens, we program a set of pass-fail criteria that looks something like this:

Lens X

  • Center MTF > 0.68 (Sharpness)
  • Off-axis sagittal MTF > 0.55 at all 4 locations (Sharpness)
  • Off-axis tangential MTF > 0.5 at all 4 locations (Sharpness)
  • Sagittal-Tangential peak difference < 0.2  (Astigmatism)
  • Largest Sagittal-Sagittal peak difference < 0.5 (Tilt)
  • Largest Tangential-Tangential peak difference < 0.5 (Tilt)

In the center and each corner, we’ve tested the lens for adequate sharpness, amount of astigmatism, and amount of tilt. The above lens not only passes, but it’s also a very good copy. It is theoretically possible that a lens could be OK in all four corners and the center, but have a problem in the mid portion, but we’ve never seen it, and we’ve tested many thousand lenses.

As you probably suspect, instead of plotting out a graph we actually send the numbers to a computer program that compares them to the known acceptable level for that lens and gives it a pass-fail.

Not all lenses look quite as pretty as the one above. Below is a different lens, with different pass-fail criteria. This lens is a little weird, but the same principles apply. The center MTF is pointed to by the green arrow. The center MTF isn’t much higher than the off-axis MTF, which is normal for this lens (a telephoto prime).

You can see all of the off-axis points are to the right side of the center peak, showing this lens has field curvature. The solid and tangential lines are separated a bit, showing there is also a small amount of astigmatism.

Olaf Optical Testing, 2018


But the most critical thing in this graph is that some of the dotted (tangential) lines are not focusing at the same place as the others. I put red arrows on the two that are furthest apart and measured the distance between them. (Again, the yellow peak doesn’t show very well, but the computer reads the numbers, not the graph.)

This shows a field tilt. This particular tilt only affects the tangential lines, not the sagittal, which is actually somewhat common. This is a pretty standard amount of tilt for this lens, and it passes easily. My point here is that we are checking the lens in 3 dimensions and getting numbers that show field tilt as well as absolute sharpness.

To give you a better idea of how it works, I’ve put thumbnails of 4 copies of a wide-aperture lens in the image below. The top two lenses are perfect. There’s the normal amount of field curvature (all of the off-axis graphs peak at a different place than the center but they’re together) for this lens.

The bottom left graph passes, but just barely. You can see the center sharpness isn’t quite as good, and there is some tilt in the tangential field, but it’s minor. The lens on the lower right fails because the tilt is very severe.

Olaf Optical Testing, 2018

I’m not trying to show you how to read these graphs, but rather just showing that when a lens isn’t right, it’s pretty apparent using this technique.

So How Good Is It?

We had three goals for this project: to test quickly, to test accurately, and to test the majority of lenses.

The quickness is good enough, although we hope to improve it more. It’s fast enough that we can use an optical bench for daily testing, which we could never do before. Of course, you have to buy an optical bench and have a lot of lens data to do that, but the improved accuracy makes it worth the cost in time and money.

The accuracy is superb. We tested hundreds of lenses using this technique during the debugging phase. Any lens that didn’t clearly pass got the complete one-hour bench test. In the above example, the lower two lenses both got the complete test, which confirmed that the lower left passed and the lower right did not.

It’s more accurate than our very experienced techs testing with the best chart setups made. The techs are about 98% accurate. The Rapid MTF test is about 99.5% accurate. But the important difference is the Rapid MTF test misses 0.5% by saying the lens fails when it actually passes. We’d much rather take 0.5% of lenses out of stock for further testing than to put lenses back in stock that shouldn’t be there.

We can’t test all lenses – yet. This was a trial run to see if the technique really worked (many people I talked to didn’t think it would). Setting it up was time-consuming; we have to write an automation script for each lens. If the lens comes in several mounts, each mount requires a separate automation script. (For those of you who’ve wondered why there haven’t been many blog posts lately, we’ve written and debugged over 200 scripts, tested multiple copies for each script, and then done the full long set of MTF tests on any lens that was questionable. It’s taken 2 of us working full-time for nearly three months to do that.)

Right now we’re using this technique to test all prime lenses; both photo and video. We’ll continue with just prime lenses for several months before expanding into other lenses. More software has to be written, more techs have to be trained to operate the optical bench, and workflows streamlined before we buy more benches and start testing zooms. We’re hoping some tweaks and improvements will speed things up a bit, too, before we order another bench and expand to other lenses.

We expect, though, that within a year this will be our standard testing for almost all lenses. I say almost all because there are some hurdles to overcome before we can do lenses of 400mm focal length and up, and some lenses with very small apertures (12mm f/4 zooms, for example).

Roger Cicala and Aaron Closz

June, 2018


Addendum: I say didn’t quite make it in time because our experience is chart and projector testing are adequate for lenses that will be mounted to cameras up to the equivalent of about 30 megapixels full-frame resolution. When you put a lens on a 50-megapixel full-frame camera the charts and projectors become the rate-limiting factor; the chart tests can look fine, but a photo on the camera may show a problem the testing missed. I’ve written about this before

Author: Roger Cicala

I’m Roger and I am the founder of Hailed as one of the optic nerds here, I enjoy shooting collimated light through 30X microscope objectives in my spare time. When I do take real pictures I like using something different: a Medium format, or Pentax K1, or a Sony RX1R.

Posted in Geek Articles
  • Thank you Roger—I must have missed the posts! Which ones? I completely missed them.

    It still might be a great lens to test on the new system. In any regard, great to see your article—I thought perhaps you got swept up in management hell and had no time for the fun stuff! As you can tell from the pile of posts, we all missed your voice. Very few people seem excited about lenses anymore, which is a darn shame—some great stuff has come out.


  • You seem to think the manufacturers will accept high volume returns. Nope, only if it fails to meet their spec, which are way lower than my spec. A retailer has to eat the customer’s returns. The manufacturer does not have to accept the retailer’s returns. If you return a bunch, they’ll basically just cut you off.

  • I’ve written it up in a couple of posts, very nice lens, particularly smooth and even from side-to-side for a 35mm.

  • Hi Roger! I am so happy to hear of your test work. I thought maybe you had sworn off of the little buggers and had left us high and dry for right proper knew knowledge. So now that you are testing faster, do you think you could get to the Zeiss Milvus 1.4/35? I am telling you, I think it exceeds the Canon Lii and has some pretty interesting chops—just what might be revealed in testing. 🙂 Pete Myers

  • Daniel


    Time to monetize your lab results. Sell lenses that are “Cicala tested and approved.” Toss the failures back to the manufacturer.

  • Clayton Taylor

    Ok, so can we see ThrowBack Thursdays on Even one re-post every other week would be fun. My fingers are crossed….

  • grubernd

    you say a regular 4-rot MTF test takes 15 minutes and that is too long.
    so how rapid is the Rapid MTF for one lens?

  • Unfortunately, I’ll never be able to talk about those. 🙁

  • Although the Rokinon quality control is dismal (I’ve owned dozens of their lenses over the years, so I know this well…) and their customer support is worse (they once denied a warranty claim by me on the grounds that a sticker I put on the lens constituted “modifying” it), unfortunately they have the market cornered for astro-landscape photographers. The 20 1.4, 24 1.4, and 50 1.4 lenses by Sigma/Nikon/Canon have coma problems that make them unusable for astrophotography. The Rokinon lenses generally sacrifice raw resolution in the center of the frame, but (when you find a good copy) they have remarkably constant sharpness out to the corners. And that’s what matters for astrophotography. I grumble at my Rokinon 24 1.4 and 50 1.4 all the time, but unfortunately they’re just the best options*.

    I replaced my (third? I’ve lost count) Rokinon 14 2.8 with an Irix 15 f/2.4. I had high hopes for it, but when it dies I’ll probably go back to another Roki 14. I had to buy three Irix 15s before I found one that was optically acceptable. And even now, it has one corner that is noticeably worse than the others. It’s more expensive than the Roki 14, it has more coma, and it’s bigger. The value just isn’t there.

    *The Otus 55 1.4 is better, yes, but it’s so gargantuan and expensive that I can’t justify it, aside from renting it occasionally for an important shoot.

  • l_d_allan

    > Great to have you and your articles back. Can I say ‘I’ve missed you’?


    And I’m looking forward to an upcoming article mentioned some months ago about a series of lenses that were beyond your expectations for IQ goodness.

    Feb 14, 2018) … RC >> So, yes I am bitter and feeling a bit sorry for myself. I mean, two weeks ago I’m doing some double-naught-secret-agent testing on a spectacular wide aperture prime lens with a resolution that made my jaw drop at frequencies I wasn’t sure was possible to test.

  • I was hoping that Roki-Yang’s QC and overall engineering / workmanship would go up with their new SP/XP lines, because of all the “popular” 14’s the 14 2.4 XP/SP is my preference due to its lower coma, but alas, initial reports are not 100% glowing on these seemingly more robust, sturdy lenses.

    Likely an astro shooter’s best hope for ultra-wide will be Nikon’s response to the Sigma 14-24 2.8, since their own 14-24 2.8 is now over 10 years old. Considering how sturdy the existing 14-24 has proven, (optically, that is, damn that rear barrel casing, why couldn’t they have just made that part out of high-grade plastic?) …but on the other hand, based on the trend in recent high-grade ultra-wide optics, I’m not looking forward to the $2-3K price tag and 2-3 lb weight.

  • Mike

    I’m an engineer, but not nearly enough of an optical geek to fully understand your articles or the discussions that follow. But, it makes me happy that people like you and your group exist, with a passion to push the envelope.
    I will keep reading, and keep not fully understanding, but carry on . . . it makes life richer and more interesting.

  • Brandon Dube

    The four rotations are different in global coordinates. They are not different in local coordinates. The 0mm image height is special. A clocking of the image plane to probe a different position in the FoV will adjust the azimuth that is aligned to “tan” or “sag”, and indeed those terms are really meaningless on-axis.

    Off-axis, tan/sag always have the same meaning, and things are always equivalent. Since T and S should be equal on-axis, it’s a good indication of misalignment, but not what kind.

  • Thank you! SLRGear is alive and well and still has those killer graphs.

    And yes, if we sell it, we’ve tested it and it’s within spec. Just to be sure, it’s the spec for THAT lens. So a Canon 85 f/1.8 isn’t held to the standards of an Otus or anything. And because you mention Astro, even our standards may not be high enough. For instance there’s a popular 14mm lens a lot of people use that we’ve never found to have equal corners. Bottom line is you’ll get a good copy of what you get.

    For #4, well, you may be able to send them in for testing, but optical adjusting, I doubt it. We can barely keep up with our own fleet, much less take on outside work.

    I actually printed up one once, but i don’t have any more.

  • Michael Parchomovsky

    Brandon, I read more carefully the article. I was wrong in my 2nd reply above. The four orientations are all different. Each orientation is a crossection of the FOV on the image (best focus) plane. The only identical, and therefore comparable, location is the center of FOV. One can see, that 0 and 90 orientations in FOV center (zero) are achieving the same results with S and T switched (as expected). The same goes for the 45 and 135 orientations. The self consistency of this results, even in high 50lp/mm frequencies, is very very impressive. Thank you Brandon and Roger.

  • umad?!

    let‘s start a kickstarter campaign for Fuji GFX then! 😀

  • Brandon Dube

    The lens is notionally rotationally symmetric. The manufacture data is equivalent to any of those points for a given image height, with the distinction being a model from optical design software vs as-built.

  • Michael Parchomovsky

    Very interesting. While 0,90 and 45,135 rotations are symmetrical in the target domain (cross reticle), they produce differnet results.
    What do you think the MTF orientation provided by the manufacturer?

  • Brandon Dube

    You’d have to play a lot less roulette with other manufactures. Rokinon (and the other stickers on Samyang lenses) are among the worst made.

    Those cannot be aligned to perfection, because the alignment cannot be adjusted. On an adjustable lens, aligning to “perfect” would be in the neighborhood of $5,000.

  • I think it is safe to assume that 1.) LR tests lenses for /a lot/ of companies, and 2.) none of them are good enough at doing it themselves, YET.

    I also gleaned from an older article that it does seem like Nikon has a pretty good track record for being one of the better ones at actually FIXING an optical problem.

  • What he said.

  • In my experience, a de-centered lens does look quite a bit different from a “tilted” lens.

    As a wedding photographer, I heavily abuse my gear; dangling a body and a 70-200 or 24-70 from a Spider Holster for 10-14+ hrs a weekend is pretty much the worst form of lens mount torture you could possibly imagine. So I’ve seen my fair share of what /appears/ to be either lenses or bodies that have a slight, slight “tweak” to the mount, which literally renders the resulting image as some sort of weird tilt-shift effect. In other words, the plane of focus is still intact and basically perfectly sharp, it’s just “tweaked”…

    A truly de-centered lens, however, usually just goes to trash off-center. You can rack focus all you want, but the sagittal / tangential discrepancy is just so big, you’re screwed.

    From what I gather, it seems that this new “rapid-fire” test should absolutely be able to discern this.

  • Well if I’m trying to get a good copy of a $1-2K lens, I have to have at least 3-4 copies worth of $$$ available on my credit card, so either way it’s probably out of my reach for now.

    I’m unlucky enough that I like to shoot things like stars, where field curvature, de-centering, (or “tilt” as Roger points out) …and other things are made rather obvious when shooting the night sky.

    If I were a portrait photographer, I’d have the luxury of not really having to care at all about almost any of this, because I’d almost always be just focusing on one small part of the image, and almost every decent prime and zoom in recent years can do that.

    I guess I should still consider myself lucky, though, that all I have to do is point a camera up at a starry sky, take 5-10 test shots with minute focus-racking, and I’ll know pretty quickly if a lens is acceptable or not.

    It just stinks that I have to play “lens roulette” at all, especially since some of my favorite lenses are Rokinon. But, you get what you pay for, of course.

  • Brandon Dube

    I’ll leave someone who works for LR to reply on the LR stuff, but for 4) we could tune a lens to perfection for you, but you wouldn’t like the price.

  • 1.) It’s great to have you back, Roger. The withdrawal was severe.

    2.) Do you remember the website SLRGear, and those old 3-D graphs they used to make? I used to think those were the coolest tests ever, and I never understood why other review sites only ever published a few numbers or a 2-dimensional line graph. The 3-D graphs made it very obvious when you had a “horribly bad” lens, but indeed, you would hard-pressed to tell if it was merely de-centered, or if it had other issues as well.

    So, it is fantastic news that you are taking things to the next level, and actually finding ways to discern the more subtle issues that can occur with a lens, let alone do it “rapid-fire”. My local mom-and-pop rental shop has a couple lenses that are BAD, and I’ve tried to tell them that the lenses need service, but they usually can’t afford to send them in for repair until /multiple/ customers complain.

    3.) Can it be assumed that a lenses we buy from LR’s used lens store is “within spec” for YOUR standards, as opposed to the nearly-useless factory standards? Or do you sometimes “let lenses go” when they are NOT “within spec” anymore?

    I ask because, as an astro-landscape photographer, I’m always on the hunt for literally /perfect/ lenses. When focusing on stars, any field curvature, de-centering, or other issues are a real show-stopper. I cannot remember how many times I’ve had what I thought was a super-sharp lens, only to find weird blobs of OOF stars at a seemingly random section of the image, only to realize that what is happening is /EXACTLY/ the type of defect that YOU have figured out how to reliably sniff out. (I’m also convinced that my body’s lens mount was tweaked due to a couple hard impacts, which made it truly impossible to shoot stars, period.)

    4.) On that note, will we ever be able to send lenses in to you guys, for the purpose of “achieving perfection”? Or should we just stick with the game of “lens roulette”, buying 3-4 copies and returning all but the least-bad one?

    5.) Can I get a T-shirt with some good/bad OLAF graphs on it? 😀

  • Brandon Dube

    Most of our for-manufacture testing comes with nondisclosure, sorry.

  • Well KEH is grading on appearance and mechanicals, this would be on optics.

  • Exactly. Decentering will generally cause MTF (sharpness) to fail and usually astigmatism to increase. Classic optical decentering would like like slightly lower center MTF, lower MTF in one or two off axis areas, and usually higher MTF in another off-axis area; probably increased astigmatism too.

  • Michael, for all the tests it’s evaluating a collimated 10 micron size cross reticle with a microscope.

  • Lorenz Ammon

    OK, thanks. I was asking because I didn’t read “decentering” in the pass-fail criteria that you listed. But I guess a decentered lens would fail the “Off-axis sagittal/tangential MTF > xy at all 4 locations (Sharpness)” for one or more locations.

Follow on Feedly