Geek Articles

Testing NASA Optical Lenses – How We Measure Sharpness and Accuracy

Published November 7, 2016

We’ve spent about 5 years developing and repurposing various types of optical testing for use testing photo and video lenses. It’s been a long process with lots of learning (learning is when you find out you’re doing it wrong), and improvements along the way. We’re pretty close to finished; our methods are fully developed. Sure there will be some more tweaks, some of the software will be fine-tuned, and some additional tests probably added in the future. But we have been doing commercial testing for the last 18 months.



Those of you who read the Geek articles on the blog have seen some of what we’re doing. I thought, though, that at least some of you would enjoy seeing what the process looks like when we go all-out-lens-testing like we do when we test a set of JPL/NASA lenses being screened for use in satellites and stuff. I’m not allowed to show you pictures of, or even tell exactly what the lenses that we’ve tested are, but I can show you the process they go through.


So What Are We Looking For?

First and foremost, for all you guys who love to scream, “For $1,200 I expect my copy to be perfect,” the primary thing we’re doing is looking for how much sample variation exists, because even in lenses being considered for satellites, variation exists. We want to know which copies are best, of course, but there’s more to it than that. One copy might have a little better center resolution, another a flatter field or more even corners. Depending upon the application (and we don’t always know what the application will be) the engineers designing the lens-camera combination may prefer one over the other.

Second, we’re sometimes looking at specific parameters. This is a lot nicer and more fun than a lens designed for “any possible camera it can mount on.” For example, we might get a specific request to evaluate a lens not just in general, but at, say 90 line pairs / mm because that has been calculated as the maximum possible resolution of the lens-camera unit. Another time it may be to evaluate tilt, or exactly measure the lens best back focus distance within 5 or 10 microns to help calibrate mounting the lens to the camera. It might be an accurate assessment of distortion or vignetting so that image-enhancement software can be optimized for the camera. Sometimes it’s all of that.

Finally, and the most interesting to us, we often will test the same set of lenses several times. In between our tests the lenses may have been baked, frozen, subjected to vacuum, vibrated, and centrifuged. Obviously, if one of those things is going to affect the lens, they’d like to know about it before they mount it to a multi-million dollar satellite and subject it to vibration, temperature extremes, and 4 gravities of acceleration on the way to it’s destination.

Initial Screening

Each lens gets physically examined, of course, and the glass examined under some really bright lights to make sure there’s no films, fingerprints, or other unwanted things on the surfaces. Yes, it could happen.

Next, we shine a 10-micron pinhole through the lens center and look at the dot the lens produces through a microscope. It should look like a nice, round dot of light like this.



If it looks like a smeared or decentered dot, we know right away the lens has problems. A normal looking center point doesn’t guarantee the lens is OK, but a smeary one guarantees it’s not OK.



Complete testing starts with simple measurements. First we accurately measure the actual focal length, because “about 35mm” doesn’t really cut it for these applications. OK, 33.829mm may be a little overkill, but hey, we’re accurate.



Next, we accurately measure the back focus (flange to sensor) distance. This is measured to the nearest 10 microns with tangential and sagittal lines checked separately (they may be slightly different). We may also check back focus distance for different wavelengths of light.



Then the distortion is mapped. The distortion plot can identify some problems (like the one below), may be needed for image optimization software designed into the lens-camera system, but also is used in the optical tests to make the MTF measurements more accurate.



Optical Testing

Standard MTF Testing

We do our standard optical test next. It’s actually far more thorough than the usual MTF test, which just makes one measurement from one side of the lens to the other. Our standard measures the lens 4 times: from side-to-side, top-to-bottom, and two diagonal measurements, so each lens gets MTF measurements that look like the two below.

Olaf Optical Testing, 2016

Olaf Optical Testing, 2016


Olaf Optical Testing, 2016

Olaf Optical Testing, 2016


The printouts give a summary, but the engineers usually want a spreadsheet with all the actual measurement numbers. Because they’re engineers, and engineers like numbers.

On the other hand, it takes some concentration to look at the graphs above and translate what goes where in your mind. To make that more intuitive, we replot the four rotation MTF data as a ‘full-field MTF’ graph like the ones below. They don’t have as much information as the actual MTF measurements but make it easy to compare lenses at a glance. Since we’re often screening 6 or 8 lenses to pick the 3 or 4 that will be used, this can be quite helpful.

For example, here are the full-field MTF graphs for 4 copies from a recent set. This test, at least, shows at a glance that the upper right lens has probably the best MTF of the 4.

Olaf Optical Testing, 2016

Olaf Optical Testing, 2016


High Frequency MTF Testing

The standard MTF testing above was done at low to moderately high (50 line pairs / mm) ranges. Often the engineers designing the lens-camera system are working at the very limits of resolution and want to know exactly how the lens will (or won’t) perform at higher frequencies. This is done in two ways. First we can do an MTF versus frequency at several selected points, usually the center and another point or two off-axis. This is useful because it shows how the lens performs at that single point at all possible frequencies.



Of course what is usually wanted isn’t the graph, but the actual MTF numbers at each frequency, which can be plugged into calculations to determine the highest possible frequency that provides the resolution they need, at least at the specific points we measured.



We can also plot an MTF curve at a given high frequency. This is useful if the engineer wants to know what portion of the field of view would provide the highest resolution the camera is capable of sensing. For example, the graph below is an MTF at 80 lp/mm that basically shows that resolution will only be availalbe within 5mm away from the center of the lens.


4 Rotation Distortion Curves

In some cases it’s critical that distortion is checked not just from one side to the other but in 4 rotations. It’s rarely necessary, but when needed we will do distortion curves for each rotation with the MTF curves. You can see there are very slight differences in the curves below.

Olaf Optical Testing, 2016

Olaf Optical Testing, 2016


We then generate a full-field distortion map for the copy in question. Again, it’s a little more intuitive to look at it this way instead of from the rotation curves done above, but it’s basically the same data.

Olaf Optical Testing, 2016

Olaf Optical Testing, 2016

Field of Focus Curves

We always do at least one MTF versus field of focus curve test on each lens from side-to-side. If one rotation on the MTF curves looked out of sorts we might also do a field curvature for that rotation.



Relative Illumination

Whether for software image enhancement, for calculation of the relative brightness of objects in an image, or for other reasons we sometimes do relative illumination curves. It’s a time consuming test but luckily only has to be done on one copy of each type.


Chromatic Aberration Tests

We don’t have to do this very often, but we can check longitudinal chromatic aberration if needed. Usually, this is just at the engineers request, but we might also do it if we noticed flange to sensor focus distance was different with different colors of light.



We’re even less likely to test for lateral chromatic aberration but occasionally it is requested. It’s a really time consuming test for us, though, because it’s one of the few tests we haven’t automated; it has to be done by hand. (For most of our tests we plug parameters into the machine do a calibration, push ‘start’ and then go have coffee while it runs. This test requires us to make calculations and move components by hand for each of dozens of measurements.)




What the Workflow is Like

This varies a bit depending upon the project. Usually, we receive a set of several copies of a given lens after they have been modified and are ‘flight ready’. The engineers have already designed the lens-camera system and know how they expect the lenses to perform. They are almost always small prime lenses that have had the focus and aperture soldered at a fixed position. They may have the original mount removed and a proprietary mount added, but sometimes the stock mounts are used.

During the first test of each copy, we have three goals. The first is to determine if any of the lenses are bad copies that probably should not be used. And yes, all you forum warriors, even with this kind of lens bad copies can happen. Second, we are confirming that the actual lenses can deliver the optical resolution that the engineers expected (and that the manufacturer claimed) when the original order was placed. Finally, we are usually determining some other, specific criteria the engineers need to know that the manufacturer wasn’t able to provide. (And no, the manufacturers don’t have the capabilities to provide the kind of information we just showed you.)

It usually takes us a couple of hours to set up for testing a particular lens, then 2 to 3 hours for testing each lens. Once that’s done, we email the results off to the project’s optical engineers to see if they have any questions, want any tests repeated, or want any other testing done. After testing is complete, we send the lenses back. The engineers may eliminate one or two samples, or rank them in order of preference, but they don’t ask our opinion about that; they are rocket scientists after all. We just furnish the data.

In most cases, JPL (or the private company, which happens a lot these days) will then stress-test the lenses. They get subjected to vacuum, temperature extremes, vibration, high gravity forces, maybe some other things we don’t know about. Then we retest the lenses optically to make certain the harsh conditions haven’t damaged them or altered the image quality in any way. They may get sent back to us for retesting after each specific stress test; for example, we’ve had one set that was back here every 3 weeks or so for most of the summer. In other cases, they’re only sent back for retesting once, after all of the stress tests are completed.

A complete set of tests like this costs from $300 to $500 per lens for complete initial testing. When we do repeat testing after the lenses have been stressed and abused we usually perform just the 4-rotation MTF test ($150 per lens), since this is plenty to determine if there has been any interval change. That may sound expensive, but it’s actually a fraction of what this kind of testing used to cost.

So don’t give up your day job to become a full-time optical metrologist. There’s really not much money in it when you consider the cost of the equipment and software required. On the other hand, we do have very cool NASA Rocket Scientist T-shirts, so we’ve got that going for us.


Roger Cicala and Aaron Closz

November, 2016


Author: Roger Cicala

I’m Roger and I am the founder of Hailed as one of the optic nerds here, I enjoy shooting collimated light through 30X microscope objectives in my spare time. When I do take real pictures I like using something different: a Medium format, or Pentax K1, or a Sony RX1R.

Posted in Geek Articles
  • Ed Bambrick
  • Brandon Dube

    My copy is actually pretty average.

    The 90/2 apo has pretty much the exact same MTF as the 100MP. On-axis, around 0.55~0.6 at 50lp/mm for the 100MP. About 0.45 off-axis. The 90apocron has about 0.65 on axis, 0.45 off-axis. I took the first 75/2 test in the folder but it is on average very similar to the 90/2. The 135 is also basically the same as the 90. Ditto for the 50, it is even numerically better sometimes.

    The 300/2.8 is better than this by a little, it is even a statistically significant difference. But I could have shown you a copy of the 300/2.8 where the MTF goes as low as 0.3 at 50lp/mm.

    Then the 100MP et al are “obviously better,” but of course this presentation is “biased,” and I “cherry picked the copy of the 300” or “intentionally dropped it.” When, really, all I’m doing is going into the database and taking the first lens by its internal tracking number for each, because it takes the least time. For the 300/2.8, I took the second one because the first one was pretty bad. Has it been dropped? Is it a dud? Well, Nikon let it out the factory door like that. Unless the shipper broke it (possible), the manufacture calls that within spec. There is equal or greater variability in all of the supertelephotos. Remember, I said the 300/2.8, by the old number I made a formula for that doesn’t really perfectly encompass variance, but that people like, the most consistent lens. Ever. Period. Full stop.

    FWIW, if you look at the image in the blog post and read it (a novel concept at times) the MTF at 50lp/mm is around 0.5~0.55 across the entire frame. I know exactly what measurement that is and could just reproduce the chart without its title photoshopped out, but what would be the fun in that.

    I think all of the lenses I showed you the MTF of are basically the same. The 12-40 stylo zoom is a bit worse than I had remembered, but the rest are within the variance, the same. I say that having tested ~1,500 lenses for MTF, personally, and written enough documentation on its measure and computation to fill a book chapter. I’ve also worked at Optikos in the past, the company that owns the original patents on MTF benches. I’m reasonably well equipped to read an MTF chart.

    I think you’re seeing what you want to see.

  • Ed Bambrick

    The Zeiss MP might squeeze in. The rest of them not even close.

  • They might, but there’s also the part about I really want to test their lenses. 🙂

  • Christopher J. May

    I love this blog. The mix of practical knowledge, techno-geekery and outright fun stuff like this make it one of my favorite photographic haunts on the web. Couple this with the amazing service I receive when I rent gear and you’ve earned a die-hard fan who won’t go anyplace else to rent gear from.

    Another great and fascinating article, Roger. Thanks.

  • Ben

    I’m shocked that you are only getting $100/hr for this kind of testing. They would pay four times that amount easily.

  • Brandon Dube

    I’m not sure what you want me to say here. Is the 300 VR II among the highest resolution lenses we’ve ever tested at Olaf? Yes. When we would compute the consistency score, was it the highest in that metric, by far? Yes.

    Is it the highest resolution lens I’ve ever tested? No. I have tested lenses that go onto autonomous vehicles and gather map data which have MTF > 0.8 at 100lp/mm. I’ve tested microscope objectives with MTF > 0.8 at 300lp/mm. Reference optics for interferometers that have MTF > 0.8 at 500lp/mm and f/0.75.

    I’ve also tested a telescope that I designed and built myself, along with other students, that had an MTF < 0.2 at 10lp/mm the first time we built it, because of severe misalignment.

    Here's a gaggle of single lens MTF measurement sets that are comparable to the one in the actual blog post. The 100/2 MP in there is my own personal copy.

    I believe ever 300/2.8 VR II measurement Olaf has is still made by me. It's a great lens. While I am getting old and senile and the almanac of every single consumer lens and its performance in my brain is fuzzier than it once was, all of the lenses I listed have MTF very comparable to the one in the blog post. If you want, I can show you the MTF of a lithographic lens design that has high MTF, even over 1000lp/mm. These systems are built, and they made the circuits in the computer you made your comment with. The current generation could write an 800×800 pixel image onto a red blood cell.

  • Ed Bambrick

    Brandon, you lose credibility when you try to make such a point. Look at the right chart closely. The best of all of the lenses (the new Nikon) in your PSOOYOA list doesn’t even come close to the 300 2.8 VRII. The only others I am aware of are the other Nikon tele’s and even with all of the benefits of being stopped down, most wide angles and normal and wide zooms never approach this level.

  • That is correct, Phillip, but I’m not taking a chance on even visiting the edges of nondisclosure so I’m not mentioning aperture we tested at.

  • LMAO!!! Not real, because no link to buy all Sony lenses.

  • Philip Service

    First, many thanks for the two articles on Fun with Field of Focus. Most informative and enlightening. Also this article. But I’m curious about the complete (I think) absence of information about aperture. For example, theoretical expectations about MTF vs. frequency, will surely depend on diffraction, and thus aperture.

  • obican

    SonyAlphaRumors: Roger Cicala (from says they use E-mount lenses(Amazon link here) as benchmark when testing lenses for NASA!!!

  • obican

    It is because due to some management decision, all the lenses you’ve been testing are Canon EF 50/1.8 II lenses, isn’t it? I wouldn’t want to disclose that information.

  • Ilya, you caught me. Since I wanted to make sure we didn’t show any data we weren’t supposed to disclose, I used posted a data set from an-in house test. Guessing from the focusing distances, that would be an E-mount lens, given the low MTFs we were focusing at a high frequency, and given the variation we were probably either adjusting or calibrating something between runs.

  • Ilya Zakharevich

    Roger, in the top half of the first table you provide, the summary “Best Tan (mm)” for MTF 1 is 3.52?—?and this is where this column has its maximum. However, MTF 2 has maximum at 3.572, but the summary says 3.522. Likewise, for MTF 3 the maximum of the column is at 3.55, but the summary is 3.523. How comes?

    Moreover, what is the reason for so large variation between MTF 1/2/3 ?

  • Brandon Dube

    100 makro-planar, 105/1.4E, 90 apocron, 75apocron, 135 tele-telyt, 50 apocron 300/2.8 VR II, a lot of lenses stopped down, angenieux 12-40 stylo, canon 30-105… the list goes on.

  • Ed Bambrick

    If I had to guess the exceptional charts shown above could have only come from a Nikon 300mm or one of the other Nikon tele’s.

  • Ed Bambrick

    What ??? No comparison to an Otus? Or at the very least one of Brandon’s $85 Canon lenses that scored higher than an Otus?

  • Lee

    Interesting to see those flat MTF charts rather than center-sharpness-optimized. I’m surprised we don’t get more lenses like that for photography, I know few of us do scientific work but I think it’d be good for repro, architecture, and landscape too.

  • Athanasius Kirchner

    I figured it was a pain, but your description is hilarious. Sound like those stories people make up when asking about something very embarrassing. Like, “my brother’s friend has this medical condition where I- err, *he* gets a rash…”

  • Paul Santek

    I can contribute a bit to that from my side of view:
    Sometimes I take pictures for the austrian space forum ( – not an agency, just a bunch of scientists and engineers providing a platform for others and working with space agencies and doing mars research stuff with an space suit simulator) – and it can happen that I am asked to take pictures for internal educational, medical, …. purposes. But you have to make sure that none of the information about that is released into public.
    As long as you are allowed to know what you are doing it’s not that bad 😉
    Working with secret stuff sounds cool at first, sometimes it’s really annoying. You just can’t ask others specific questions or providing helpful information when solving problems with ‘modified’ equipment. You always have to invent descriptions and stories and think a few steps ahead to avoid beeing led into an dead end… (been there as an mechanical engineer….). You even can’t tell a supplier what you exactly want to do if you want to buy machinery 😀

  • Brandon Dube

    Well, in the photo of the corner of the room, you can see the back of my head (white shirt). I’m surprised I was wearing pants, TN is considerably hotter than where I am usually.

  • Athanasius Kirchner

    Sure, sounds simple 🙂 But I’m imagining you do take photos of the lenses in some cases, for documentary, educational o reference purposes, and I’m guessing those are pretty sensitive, yes? Also, there’s a number of your staff working at the lab, and they must all be beholden to secrecy, right?

  • I’m absolutely not allowed to talk about the time-travel testing we do, so I’d better change that back to 2016. 🙂

  • Mat

    So, I guess it’s easy to find out Roger is working for NASA when you see this piece was written in November 2017 and you’re reading this in 2016!

  • Athanasius, they’re pretty simple. I can talk about what we do. I can’t talk about what we do it to. 🙂

  • Athanasius Kirchner

    Thanks Roger and Aaron for the great insight. The NDA’s must be draconian when working with JPL, or is that a thing of the past?

  • Athanasius Kirchner

    My thoughts, exactly. And only a $150 for the MTF data? That’s usually enough to shut people up, since there are no other worthwhile characteristics to a lens :/

  • Carleton Foxx

    I’m drooling….

  • *puts forum warrior helmet on*

    Wait, it only costs $500 or less to prove the supremacy of my lens to my fellow forum warriors?! Sign me up! 🙂

    *takes forum warrior helmet off*

    Awesome write up. I had no idea you guys were doing stuff like this. I hope OLAF will pay for itself eventually!

Follow on Feedly