Is Your Camera Really The Best Optical Test?

“But you must not eat from the tree of the knowledge of good and evil, for when you eat from it you will certainly die.” Genesis 2:17.

I have enjoyed, for some years now, the process of learning about lenses and optics. This blog shares a lot of what I’m in the process of learning. A lot of you like reading about what we’re doing and finding out what we’re finding out about what is. A lot of others don’t like it because it upsets their preconceived notions of what should be.

This post is going to be more polarizing than most. I’m going to talk about the very real phenomenon that people are describing on the internet; that they’re seeing more lenses with optical problems. If you aren’t interested in that, or are comfortable just saying they’re all bad photographers, this will be a long, boring article you want to skip.

 

One of the more common responses on various internet forums seems to be “this manufacturer or that needs to up their quality control.” There’s a bit of truth to that. A slightly less common response has been “your manufacturer, but not mine, can’t make their lenses without a lot of variation.” There’s a bit of truth there sometimes, but not very much. The broader truth is we’re observing an industry-wide issue. It’s not happening because lenses are worse. Lenses are actually better than they’ve ever been overall. There are some exceptions, of course.

For those of who don’t want to read further, here’s the summary:

• The optics of new lenses are designed to take advantage of high resolution sensors. They are sharper than they were a decade ago.
• An optically misaligned lens is just as soft in the bad area no matter how good the design. But the difference between the good and bad areas may be greater.
• Opto-mechanical design and sample variation isn’t really much different than it was when the best cameras were 16 megapixels. (Some manufacturers have made improvements, but not most.)
• Cameras resolve far more than they did a few years ago. They can demonstrate lens flaws you might not have seen with your last camera.
• Metrology (optical testing and measurement of lenses) is the same as it was decades ago (There are some exceptions, particularly Sigma). In many cases you may see defects in your pictures that the service center can’t see with their testing.

In a nutshell, you can now take a picture with a high-resolution camera and see defects in the lens that might not have been apparent on your last camera. The manufacturer or service center often still use crude optical tests that don’t show the defect.

Why Higher Resolution Makes a Difference

Yes, I know this is obvious, sort of. But still, it’s worth discussing.

First, let me say that optical physics guys can do all kinds of math and show that, particularly if you involve vague terms like ‘depth of field’ and ‘resolution’ any statement about resolution is either right or wrong. I’m going to stay out of that and try to limit this discussion to words, pictures, and common sense. We’ll start with a simple, common-sense analogy. Things are way more complex than this, but it gives an idea of the overall picture.

Looking at Points

We all understand that a lens resolves a point in the world around you into a point on the sensor. If we look at a 5 micron point projected through a good lens mounted on our Olaf Bench, the point looks like a point, at least in the center of the image.

 

Out towards the edges, where aberrations are warping the light rays around, that point of light looks somewhat different.

 

Now remember, that’s a magnified view of that point source I’m showing you; we’re basically looking at the image the lens makes magnified by a microscope. Let’s pretend that the white box in the image below represents a 6.25 micron pixel from a Canon 5D Mk III (22.3 megapixel full-frame sensor). Despite the fact that the dots look different on the microscope of my optical bench, they may look like perfect dots to your camera .  .  .  . or nearly perfect. (And again, I’m leaving out details about AA filters, Bayer arrays, etc. but the analogy holds if we made this a 4 or 16 pixel array instead of a single pixel.)

 

This little analogy should explain why I often say, “Yes, there’s a difference in the optical bench tests, but I doubt you’ll see it on your camera.” Those dots are clearly different and the MTF taken at those two points will be different. But basically either one was going to fill up 1 pixel on my camera, with maybe just a tiny bit of bleed over from that lateral point on the right. The optical bench might see the difference, but your camera sensor probably wouldn’t.

But now many of us have a  40- or 50-megapixel cameras. Which means our pixel pitch is now 4 microns instead of 6.25. What the lens rendered as a 1 pixel dot now is bigger than a pixel. Still, even that edge pixel looks pretty good, with just a little spill into adjacent pixels (or array of pixels, whatever your pleasure).

 

But that was a really great edge pixel from a $5,000 prime lens. What about one from a more standard looking lens? Well, this edge pixel from this zoom lens is smearier and has a more noticeable chromatic aberration. And you should get the idea that it might be more apparent on a camera with smaller pixel pitch, as represented on the left below.

 

My analogy might be a little clearer if I average each of my pretend pixels from above, since my allegorical camera would only be reporting one value for each pixel. (Yes, I’ve invented color sensing pixels and eliminated the Bayer filter for this allegory, but if you prefer, consider each box to be a 4-pixel array complete with Bayer filter).  But the point is simple: higher resolution means you’re going to get better detail about how smeary that dot is. In this case, either camera would show you this lens has lateral chromatic aberration, but the higher density pixels (on the left) also show the complex coma-type aberrations a bit more too.

 

Depth of Field

Depth of Field is one of those interesting phenomenon that’s both very scientific and very arbitrary. It’s determined by a complex mathematical formula, so it’s seems uber scientific. The key definition in determining depth of field (and some other concepts) is the Circle of Confusion (CoC). The definition of the circle of confusion (in words) is the width where the point blurs enough that it is larger than some maximum allowable diameter.

Depending upon your purpose, ‘maximum allowable diameter’ can have different definitions, and therefore depth of field calculations that come out of the formula can be very different. For an SQF calculation you might want to make it larger than some point size on a final print of a certain size. When we’re working with camera sensors, reasonable definitions might be the width of a pixel, or a 4 pixel square, or a 16 pixel square. Whatever formula you want to use, though, a smaller pixel means a smaller circle of confusion, which means a narrower depth of field.

Now, you may ask, why does that matter? Well, one thing people are seeing more frequently is a tilt in their lens. (In forums people call every lens defect “decentering”, but optical decentering often isn’t involved. Tilt and spacing errors are at least as common as optical decentering.)

Let me show you the effect of depth of field in two field-curvature graphs of the same lens, which has a small tilt. With a larger depth of field (on the right) there’s no way you’d notice it on any type of test chart testing; focus on the center and the left and right sides are going to be equally sharp. With a narrower depth of field (on the left), you might start to notice one edge is softer than the other if you were testing on a chart, brick wall, or other flat surface.

 

The drawing is a bit exaggerated since I basically doubled the depth of field by stopping the lens down (which is why the MTF is higher on the left). With the wider depth of field, though, the left edge MTF is 0.82 and the right 0.77. Those numbers aren’t very different and most of us would agree that was normal. With the narrowed depth of field, however, the MTF on the left edge is about 0.84 but on the right edge it’s 0.57. If I give you those numbers for your lens you’ll understandably scream that it’s awful and decentered. (Of course, it’s actually tilted, but people on the forums always scream ‘decentered’. It’s like playing Lens Bingo.)

This was probably pretty obvious without the illustrations, but the bottom line is if you move from 20-ish megapixels to 40- or 50-ish megapixels, you’re more likely to see flaws in your lens if you look for them. This has happened to some excellent photographers I know. They have noticed a lens here or there that was fine on their 5D Mk III or A7 that has a weak side or corner on their 5Dsr or A7rII.

So Why Doesn’t the Service Center See This?

This section is not going to name names. I’m under nondisclosure agreements with any company we help with testing so I’m ethically and legally unable to share names. You can take it on faith that I’m sharing facts or feel free to think I made it all up. I would point out, though, that for 10 years I’ve only made stuff up on April 1 and this is September.

The summary is almost all of you greatly overestimate the type and amount of optical testing that lenses get, whether it’s at the factory after assembly or in the repair center when it has a problem.

A lot of repair locations literally do resolution testing on an 8 X 11 or 13 X 19 ink jet printed chart. One used pictures of a bookshelf across the office to do optical adjustments on very expensive lenses (I know because they left their memory card in the camera they insisted we send in with the lens). AF 1951 charts are still commonly used (and remember, the ‘1951’ is the year the chart was developed, which means it was designed for film).  Some use large, high-resolution ISO 12233 charts, but not many.

Factory Service Centers and Factory Authorized repair facilities generally use a factory-specified graph, again often printed at ink-jet resolution. It’s shot with a camera hooked up to adjustment software that gives pass-fail readings. This isn’t necessarily bad, although it’s not great. And depending on the factory, that chart may actually not be as good as an AF1951 or ISO12233 chart.

Some service centers use Lens Test Projectors and a center-only collimator to do testing and adjustments. If you ask a manufacturer’s engineer what the gold standard of testing is, with few exceptions that’s what they’ll say: a center-only collimator and a lens test projector. (Trust me on this one, I’ve had several say just that until we showed them why it wasn’t). If you ask them how long has that been the standard, though, they’ll tell you since the 1960s. That means since film.

 

Don’t get me wrong, a lens test projector and centering collimator is really good testing; generally better than test charts. We’ve used them in the past. They’re still the gold standard for Cine lenses everywhere. But Cine lenses are resolving 4K (just under 9 megapixels) or perhaps 6k (about 19 megapixels), not the 40- or 50-megapixels of a high resolution SLR. (BTW — I’ve got a couple of lens test projectors and centering collimators sitting on shelves in the back gathering dust if anybody wants to buy one.)

Things may actually be a little worse than what I’ve described so far. Remember, the ‘spec’ of ‘in spec’ is whatever the manufacturer says it is. A wide-angle lenses has a field of view of over 50 degrees to each side, for example, but I know of two manufacturers that don’t test wider than 30 degrees. They feel that anything significant will show up by that angle. That’s not my experience, but that’s what they say. Another does all of their testing at f/4 because that’s the standard for their automated testing software. And yes, a whole lot of problems that show up at f/1.4 disappear at f/4.

For example, scroll back up and look at the introductory picture, the crop from the center of a test chart. That lens was sent back from a factory service center twice, the last time with a note that said it met manufacturer’s specifications, no they couldn’t share those specifications, and there was nothing else they could do. It’s fixed now, but not by them.

My point here is not to get you all gathered together with pitchforks and torches to go storm the castle. This wasn’t a plot, in my opinion, it’s simply a lot of inertia. Metrology isn’t sexy and doesn’t make money. Until recently complaints about lenses weren’t any more frequent than they were back in 1990, so there was no real motivation for change. And I do want to repeat, the reason I know some of this stuff is because a number of manufacturers and service centers are suddenly and rapidly trying to improve their testing because they realize it now is a problem.

Better Metrology is the Answer, Right?

Well, yes, of course, but there’s another problem with better metrology: it’s too much better. I’m going to pull back the curtain a bit and show you things you don’t want to see. For years now, you’ve been seeing my summary MTFs, the average of 10 lenses, each tested at 4 rotations. So the MTF you see shows either the average or the range of 40 measurements at each point. It’s sometimes more accurate than the computer generated MTFs you see listed with a lot of lenses, but it’s still a summary.

 

We do 10 or more copies and come out with an ‘expected range’ for a given lens. Then we can plot a particular lens (the lines in the graph below) against that range and find out if it’s acceptable. The one below, for example, shows as about average or maybe a little better.

 

That’s good, but remember, even the line above is the average of the 4 rotations and two sides of the tested lens lumped together. What I don’t show  you very often is what a single lens looks like at 4 rotations because you’ll get upset.

Let’s look at the raw measurements for that went into making the lines for the lens above: this is the full MTF at each of 4 rotations; “0 degrees” is side-to-side as mounted on a camera, “90 degrees” is top-to-bottom, and the other two are diagonals.

 

That’s not quite as smooth and beautiful, is it? That’s reality, my friends, for basically ever lens we test (and while I’m not identifying the lens, this is a really good prime lens). I expect you’re now wondering, after looking at the 90 and 135 degree rotations above if one corner of the lens is significantly softer than the rest. Obviously it is on the MTF bench. But will you see it on your camera?

Just to give you an idea below are 4 more copies of that lens, hand selected to be among the best from 20 copies or so. Even with these hand-selected  best copies of prime lenses there is, if you look, always one corner that’s a little softer or more astigmatic than the others. But none are quite as weak as the one above. But most of the 20 looked about like the one above?

 

 

Unlike chart testing or even lens test projectors, optical bench testing shows us MORE resolution than your camera can see. At this degree of examination, no lens we’ve ever tested is perfectly identical in all 4 rotations. That’s many thousands of lenses tested. And none that were perfect. Ever. The fact is, those 4 lenses in the graph above are as close to perfect as we ever see.

This is why, when people suggest I cherry pick them a perfect copy, I tell them I’ve never seen one. With better lenses, mainly primes, we expect ‘so good that it looks perfect on your camera’. For most lenses we are looking for ‘within the expected range we see for this lens after eliminating bad copies’.

Let’s go back to the examples above. That lens on top, you may have noticed, is indeed a bit worse than the 4 ‘best possible’ copies out of 20 I picked out for the second image. The worst copy, the one on top, was an ‘average’ copy of this lens. Was it acceptable? Actually it was. It rented 20 times and no one ever had the slightest complaints about it. After every rental it was tested on a high-resolution, oversized chart and it looked fine.

Even when we made a point of looking for the weakest area on test charts or in photos, it looked fine on a 5D III. If we put it on a 5Dsr, though, we did agree we could see a little weakness on test charts. We thought.

My point here is that when we have a ‘too good’ test, so within that test we have to decide ‘where is the cutoff.’ In other words, we still have to decide ‘what is in spec.’

I’ll pause now for 476 of you to say “well, for that kind of money they should all be perfect.” Because I know you want to. Even those of you who know it’s not possible. And then I’ll say like I always do, “that was just a $1,5oo prime.” Because you can’t get perfect with $15,000 primes. The goal is not perfect. The goal is close enough so you can’t see any different in the picture you take.

So What’s the Bottom Line?

It’s pretty simple, actually. New cameras are showing defects in lenses that manufacturer’s weren’t quite ready to deal with. They (and we) are regrouping to put more adequate testing in place. We had a bit of a head start because we have been doing higher level testing for a while. We’re still, for example, the only place on the planet (as best I know) that can test electromagnetic focus lenses on an optical bench, for example, although at least one manufacturer will be doing it in a few months.

But we still don’t know exactly where the cut-off should be. We test lenses that go into satellites and stuff. Those aren’t perfect either, but in those cases the engineers who designed the cameras know exactly what resolution the camera can resolve and tell us so we can say a given lens is OK or not OK. The manufacturers, if they want to, can do that same thing. And at least in some cases they are starting to. But we’re all going to have to live through a transition period while those changes are made.

And I want to make it clear that we’re struggling with this, too. Months ago I tested a lens for Fred Miranda. I correspond with Fred regularly and I know he is an exceptionally careful photographer and will notice any (and I do mean any) flaw in a lens. I sent him a lens that passed with flying colors. Sure it had a slight abberation in one corner, but it was well within ‘my spec’. Fred sent it back after one day’s shooting for a soft corner — and absolutely identified that same corner I had passed as ‘good enough, you’ll never notice it on a camera’. My ‘in spec’ standard for that lens was developed when a 5D III was high resolution. He had put the lens on a much higher resolution camera and was noticing something that wouldn’t be noticeable on a lesser camera. (To be honest, it probably wouldn’t be noticed by most photographers, either. But Fred has bionic MTF eyes, I think.)

When high-resolution SLRs came out all of us, myself included, were wondering which lenses would let us maximize all that resolution. Few of us; not me, not you, and not the manufacturer, worried that that high resolution would let us see the weaknesses in a lens. Crowd-sourced complaints, legitimate complaints, have gotten their attention (and mine).

Things will get better in a couple of years, because changes are being made, but it’s not better right now. I know you want instantaneous changes, but they aren’t possible. For example, we took the decision a few months ago to improve our standard testing – the optical bench is excellent, but it takes 10 or 15 minutes to test a lens. We can’t run 800 lenses a day through it. But even our custom-made, high-definition test charts are barely adequate for testing the best lenses on the best cameras. We’re having to develop, in conjunction with a metrology company, new equipment that can be as accurate as an MTF bench but fast. That equipment isn’t cheap (think small house or maybe a Bentley) but more importantly you can’t just order one from Amazon. They’re built to order and it will be months before it’s up and running.

And remember, we’re a small company that responds to change quickly. Big companies have 12 layers of management approval and budgets made out a year or two in advance.  The testing department is managed by an optical engineer who is human, and humans are often a bit resistant to change. There’s accountants who are going to want to know why everything is costing so much, especially because this wasn’t in next years budget. The companies are aware of the problem, they’re addressing it, but it’s going to take them some time to make changes.

And, as best I know, everyone is still figuring out the best ways to test lenses to make sure we catch every defect that might be visible on a photograph without all the ‘noise’, if you will, of false-positive results. Remember, we’ve never seen a perfect MTF curve on any lens, but we’ve seen lots of lenses that are photographically perfect on the best cameras. Some experimenting is needed to figure out exactly which things are significant and which aren’t. (If you’ve read this far and still think ‘they should all be perfect’, I don’t know what else to say except reality sucks, doesn’t it?)

For example, we’ve just started doing field-of-focus testing on lenses. It may well be that this is going to correlate better with the defects you can see in a photograph than actual MTF tests do. We’re still deciding what frequency of MTF test is most appropriate for newer cameras. Historically, 20 line pairs / mm correlated most closely to what you see when you take a picture. In higher pixel-density cameras we know 30 lp/mm is better and we’ve moved to that. But we aren’t sure if 40- or 50 lp/mm might be necessary. Those have a lot more false positive results, though, which makes things more difficult.

In the meantime, those of you out there who have been saying, “just go take some pictures and see if they’re OK.” Well, right at the moment, you have the correct answer, at least for people shooting the highest resolution cameras for a photo. A lot of photos of different subjects looked at carefully are probably going to identify things we’d be arguing about in the lab.

And this is coming from a guy who spends all day, every day, testing things optically in a lab. Right at the moment, ‘take more pictures’ is correct and I am, well, less right. But give me a few months to experiment and few hundred thousand dollars worth of new testing equipment and I’ll be righter.

 

Roger Cicala