Testing NASA Optical Lenses – How We Measure Sharpness and Accuracy
We’ve spent about 5 years developing and repurposing various types of optical testing for use testing photo and video lenses. It’s been a long process with lots of learning (learning is when you find out you’re doing it wrong), and improvements along the way. We’re pretty close to finished; our methods are fully developed. Sure there will be some more tweaks, some of the software will be fine-tuned, and some additional tests probably added in the future. But we have been doing commercial testing for the last 18 months.
Those of you who read the Geek articles on the blog have seen some of what we’re doing. I thought, though, that at least some of you would enjoy seeing what the process looks like when we go all-out-lens-testing like we do when we test a set of JPL/NASA lenses being screened for use in satellites and stuff. I’m not allowed to show you pictures of, or even tell exactly what the lenses that we’ve tested are, but I can show you the process they go through.
So What Are We Looking For?
First and foremost, for all you guys who love to scream, “For $1,200 I expect my copy to be perfect,” the primary thing we’re doing is looking for how much sample variation exists, because even in lenses being considered for satellites, variation exists. We want to know which copies are best, of course, but there’s more to it than that. One copy might have a little better center resolution, another a flatter field or more even corners. Depending upon the application (and we don’t always know what the application will be) the engineers designing the lens-camera combination may prefer one over the other.
Second, we’re sometimes looking at specific parameters. This is a lot nicer and more fun than a lens designed for “any possible camera it can mount on.” For example, we might get a specific request to evaluate a lens not just in general, but at, say 90 line pairs / mm because that has been calculated as the maximum possible resolution of the lens-camera unit. Another time it may be to evaluate tilt, or exactly measure the lens best back focus distance within 5 or 10 microns to help calibrate mounting the lens to the camera. It might be an accurate assessment of distortion or vignetting so that image-enhancement software can be optimized for the camera. Sometimes it’s all of that.
Finally, and the most interesting to us, we often will test the same set of lenses several times. In between our tests the lenses may have been baked, frozen, subjected to vacuum, vibrated, and centrifuged. Obviously, if one of those things is going to affect the lens, they’d like to know about it before they mount it to a multi-million dollar satellite and subject it to vibration, temperature extremes, and 4 gravities of acceleration on the way to it’s destination.
Each lens gets physically examined, of course, and the glass examined under some really bright lights to make sure there’s no films, fingerprints, or other unwanted things on the surfaces. Yes, it could happen.
Next, we shine a 10-micron pinhole through the lens center and look at the dot the lens produces through a microscope. It should look like a nice, round dot of light like this.
If it looks like a smeared or decentered dot, we know right away the lens has problems. A normal looking center point doesn’t guarantee the lens is OK, but a smeary one guarantees it’s not OK.
Complete testing starts with simple measurements. First we accurately measure the actual focal length, because “about 35mm” doesn’t really cut it for these applications. OK, 33.829mm may be a little overkill, but hey, we’re accurate.
Next, we accurately measure the back focus (flange to sensor) distance. This is measured to the nearest 10 microns with tangential and sagittal lines checked separately (they may be slightly different). We may also check back focus distance for different wavelengths of light.
Then the distortion is mapped. The distortion plot can identify some problems (like the one below), may be needed for image optimization software designed into the lens-camera system, but also is used in the optical tests to make the MTF measurements more accurate.
Standard MTF Testing
We do our standard optical test next. It’s actually far more thorough than the usual MTF test, which just makes one measurement from one side of the lens to the other. Our standard measures the lens 4 times: from side-to-side, top-to-bottom, and two diagonal measurements, so each lens gets MTF measurements that look like the two below.
The printouts give a summary, but the engineers usually want a spreadsheet with all the actual measurement numbers. Because they’re engineers, and engineers like numbers.
On the other hand, it takes some concentration to look at the graphs above and translate what goes where in your mind. To make that more intuitive, we replot the four rotation MTF data as a ‘full-field MTF’ graph like the ones below. They don’t have as much information as the actual MTF measurements but make it easy to compare lenses at a glance. Since we’re often screening 6 or 8 lenses to pick the 3 or 4 that will be used, this can be quite helpful.
For example, here are the full-field MTF graphs for 4 copies from a recent set. This test, at least, shows at a glance that the upper right lens has probably the best MTF of the 4.
High Frequency MTF Testing
The standard MTF testing above was done at low to moderately high (50 line pairs / mm) ranges. Often the engineers designing the lens-camera system are working at the very limits of resolution and want to know exactly how the lens will (or won’t) perform at higher frequencies. This is done in two ways. First we can do an MTF versus frequency at several selected points, usually the center and another point or two off-axis. This is useful because it shows how the lens performs at that single point at all possible frequencies.
Of course what is usually wanted isn’t the graph, but the actual MTF numbers at each frequency, which can be plugged into calculations to determine the highest possible frequency that provides the resolution they need, at least at the specific points we measured.
We can also plot an MTF curve at a given high frequency. This is useful if the engineer wants to know what portion of the field of view would provide the highest resolution the camera is capable of sensing. For example, the graph below is an MTF at 80 lp/mm that basically shows that resolution will only be availalbe within 5mm away from the center of the lens.
4 Rotation Distortion Curves
In some cases it’s critical that distortion is checked not just from one side to the other but in 4 rotations. It’s rarely necessary, but when needed we will do distortion curves for each rotation with the MTF curves. You can see there are very slight differences in the curves below.
We then generate a full-field distortion map for the copy in question. Again, it’s a little more intuitive to look at it this way instead of from the rotation curves done above, but it’s basically the same data.
Field of Focus Curves
We always do at least one MTF versus field of focus curve test on each lens from side-to-side. If one rotation on the MTF curves looked out of sorts we might also do a field curvature for that rotation.
Whether for software image enhancement, for calculation of the relative brightness of objects in an image, or for other reasons we sometimes do relative illumination curves. It’s a time consuming test but luckily only has to be done on one copy of each type.
Chromatic Aberration Tests
We don’t have to do this very often, but we can check longitudinal chromatic aberration if needed. Usually, this is just at the engineers request, but we might also do it if we noticed flange to sensor focus distance was different with different colors of light.
We’re even less likely to test for lateral chromatic aberration but occasionally it is requested. It’s a really time consuming test for us, though, because it’s one of the few tests we haven’t automated; it has to be done by hand. (For most of our tests we plug parameters into the machine do a calibration, push ‘start’ and then go have coffee while it runs. This test requires us to make calculations and move components by hand for each of dozens of measurements.)
What the Workflow is Like
This varies a bit depending upon the project. Usually, we receive a set of several copies of a given lens after they have been modified and are ‘flight ready’. The engineers have already designed the lens-camera system and know how they expect the lenses to perform. They are almost always small prime lenses that have had the focus and aperture soldered at a fixed position. They may have the original mount removed and a proprietary mount added, but sometimes the stock mounts are used.
During the first test of each copy, we have three goals. The first is to determine if any of the lenses are bad copies that probably should not be used. And yes, all you forum warriors, even with this kind of lens bad copies can happen. Second, we are confirming that the actual lenses can deliver the optical resolution that the engineers expected (and that the manufacturer claimed) when the original order was placed. Finally, we are usually determining some other, specific criteria the engineers need to know that the manufacturer wasn’t able to provide. (And no, the manufacturers don’t have the capabilities to provide the kind of information we just showed you.)
It usually takes us a couple of hours to set up for testing a particular lens, then 2 to 3 hours for testing each lens. Once that’s done, we email the results off to the project’s optical engineers to see if they have any questions, want any tests repeated, or want any other testing done. After testing is complete, we send the lenses back. The engineers may eliminate one or two samples, or rank them in order of preference, but they don’t ask our opinion about that; they are rocket scientists after all. We just furnish the data.
In most cases, JPL (or the private company, which happens a lot these days) will then stress-test the lenses. They get subjected to vacuum, temperature extremes, vibration, high gravity forces, maybe some other things we don’t know about. Then we retest the lenses optically to make certain the harsh conditions haven’t damaged them or altered the image quality in any way. They may get sent back to us for retesting after each specific stress test; for example, we’ve had one set that was back here every 3 weeks or so for most of the summer. In other cases, they’re only sent back for retesting once, after all of the stress tests are completed.
A complete set of tests like this costs from $300 to $500 per lens for complete initial testing. When we do repeat testing after the lenses have been stressed and abused we usually perform just the 4-rotation MTF test ($150 per lens), since this is plenty to determine if there has been any interval change. That may sound expensive, but it’s actually a fraction of what this kind of testing used to cost.
So don’t give up your day job to become a full-time optical metrologist. There’s really not much money in it when you consider the cost of the equipment and software required. On the other hand, we do have very cool NASA Rocket Scientist T-shirts, so we’ve got that going for us.
Roger Cicala and Aaron Closz
Author: Roger Cicala
I’m Roger and I am the founder of Lensrentals.com. Hailed as one of the optic nerds here, I enjoy shooting collimated light through 30X microscope objectives in my spare time. When I do take real pictures I like using something different: a Medium format, or Pentax K1, or a Sony RX1R.