How To's

Why You Can’t Optically Test Your Lens with Autofocus

Published February 14, 2014

By far the most common email and PM I get basically says, “I think my lens is optically out of sorts. What do you think?” The second most common is, “So what’s the best way to optically test my lens without a ton of equipment.” Years ago I wrote an overly long post about testing lenses, but it was not specifically about optically testing. Plus, I’ve learned a bit since then and have been trying different things that anyone could do without a lot of equipment.

So I thought I’d write a series of posts about optically testing a lens. Not ‘determine MTF and write lab-test reviews’ kind of testing. But enough to tell if your lens is optically centered and a good copy.

We all know simply taking a few hundred images will tell you if you like the lens or not. But sometimes people don’t have that kind of time. They need to know if their lens is OK while they still have the opportunity to return it.

Other times they know they don’t like the lens, but they do want to know if that’s because they have a bad copy (especially if they’ve bought a used copy), or if that’s just how the lens is. There’s no sense trying 5 more copies of the lens if the one you tried at first is a good representation of how the other copies are going to be. But there’s also no sense giving up on a lens that everyone raves about just because you tried a bad copy. (For those who want to know, overall our experience is about 2% of lenses we buy are out of spec right out of the box.)

I’ll get into the setup I recommend for optical testing over the next couple of posts. But first, let’s talk about why I tell 90% of the people who ask me if their lens is decentered that I can’t tell anything from that image because it’s taken with autofocus.

Without Proper Focus, Testing is Useless

Critically accurate focus to test a lens is very different than acceptably accurate focus to take a picture. In order to make a nice print, everything within the depth of field is acceptably sharp. Notice the definition is acceptably, not maximally. The area of maximal sharpness is less than the entire depth of field.

Let me show you some examples using Imatest. You don’t need Imatest or any other expensive stuff to test your lens, but it does give us nice numerical data that makes a better demonstration than pictures could. Like all optical testing, Imatest is very sensitive to focus.

Here’s a simple demonstration. I set up a Canon 70-200 f/2.8 IS II on a 5D Mk II and obtained the best possible focus I could. Then I took 10 repeated shots changing nothing. This simply establishes how much variation in MTF 50 occurs when nothing about the camera or lens changes. Variables like minute lighting changes, mirror or shutter slap differences, perhaps even electrical interference or trucks driving by the building cause this slight variation.

 

 

Next I took 10 more shots, turning the focus ring to one extreme or the other and then carefully manually focusing each shot. I used a 17″ external monitor for manually focusing and there’s a specific focus target on the Imatest chart, so this is as good as I could ever hope to manually focus. Even under these ideal conditions, there’s a bit of variation compared to the 10 shots where I touched nothing between each shot.

 

 

Next I repeated the exercise, this time using the camera’s live view contrast detection AF system. The 5DII isn’t the absolute best contrast detection system I’ve worked with, but it’s not bad, either. Contrast actually is really close to my manual focus capabilities, but there is a bit more variation.

 

 

Finally, I repeated the exercise using the camera’s phase detection autofocus. Again, the 5DII isn’t the best phase detection AF system in the universe, but it’s certainly decent on a still target with center point AF and an f/2.8 lens. (And yes, we had checked microfocus adjustment and this camera and lens were spot on at this distance.)

 

 

To summarize my point, if I am checking a lens like our techs do all day, manual focus is simply much more accurate than phase-detection autofocus.

 

 

Let me again differentiate ‘testing quality focus’ from ‘picture taking focus’. If I was taking a 3-Dimensional picture, all of the images above the 850 line would be virtually indistinguishable. The two phase-detection shots that are lower would probably be acceptable; if you were comparing identical shots you might notice the two had slightly missed focus, but not horribly. But when we’re assessing a 2-dimensional test chart, that would be enough to make the lens look unacceptable.

Phase detection accuracy is different with different cameras and even with different lenses on the same camera. The pair used in this example are quite accurate; not the very best, but better than average. The very best (in these conditions) we’ve tested, the Canon 5DIII or 1Dx with certain lenses, are nearly as accurate as contrast detection AF.

Good microfocus adjustment lets the phase detection system focus at the proper location, but it can’t make the pattern of variation much smaller. (If MF adjustment is really off, the pattern can be bigger than this, but this is about as good as it gets.)

One other point that will become more apparent with the next two posts: some of the most sensitive indicators of a decentered lens are seen when it’s just barely out of focus. So the testing I’m going to describe will require evaluating the lens both in focus, and just barely out of focus in both directions. You can’t do that kind of evaluation with phase-detection AF.

So What’s Next?

This first post was to demonstrate that hand-held, autofocus optical testing isn’t very useful. If you want to test a lens using autofocus go take pictures and see if you like the lens, which is really what I recommend everyone do, anyway. But if you have a lens you don’t like, one that seems softer than it used to be, or bought a used lens that doesn’t seem nearly as good as it should, this series should be interesting.

In the next post I’ll describe a pretty simple testing setup that will you do a pretty good job of detecting a decentered or optically maladjusted lens in 10 minutes or so. It won’t be quite as good as a $150,000 optical bench, but it’s close. We not only use this setup for our basic lens testing in the intake room, we use it for some basic optical adjustments when lenses are decentered.

I expect many of you will have everything you need to set it up already. If not, you still won’t spend more than a couple of hundred dollars. Think of it as 90% of the function of an optical bench or Imatest lab for almost none of the cost.

In the third post we’ll give several examples of how to use the setup, showing what good and bad lenses look like. And if anyone is interested, we’ll bring a setup to WPPI in Las Vegas next month where you can play with it yourself.

 

Roger Cicala

Lensrentals.com

February, 2014

 

 

 

 

Author: Roger Cicala

I’m Roger and I am the founder of Lensrentals.com. Hailed as one of the optic nerds here, I enjoy shooting collimated light through 30X microscope objectives in my spare time. When I do take real pictures I like using something different: a Medium format, or Pentax K1, or a Sony RX1R.

Posted in How To's
  • Zack

    Roger, Thank you so much for these articles. Every one of them is outstanding and unique… and each adds just one more thing for us obsessive gear heads to stress over 😉 Please keep them coming!!!

  • Roger Cicala

    HI Jon,

    1) I think it does, a bit, but the difference is small enough that I don’t know if I could actually prove it.
    2) Yes, but 10 seconds. We’re paranoid 🙂

  • Jon H

    Hi Roger,

    1) Does the 5D3 have much more accurate liveview (contrast AF) than the 5D2? I was under the impression that CDAF on the 5D3 was very accurate for a high contrast test target.

    2) Did you use mirror lockup with 2 sec timer to eliminate mirror slap in your tests?

    Thanks

  • Jon H

    Hi Roger,

    Very interesting analysis. Really appreciate these articles you are making available.

    1) Does the 5D3 have much more accurate liveview (contrast AF) than the 5D2? I was under the impression that CDAF on the 5D3 was very accurate for a high contrast test target.

    2) Did you use mirror lockup with 2 sec timer to eliminate mirror slap in your tests?

    Thanks
    Jon

  • AJ

    Hi Roger,
    Thanks for the response.
    Yes that’s more or less it – actually that’s ‘part le deux’ – another possibility.
    In essence I’m thinking that once the AF motor gets ‘up to speed’ – in other words maximum acceleration attained by virtue of the distance needed to be traveled by the motor to obtain focus – it’s difficult to stop at the correct focus point.
    A by-product of those wanting ‘infinitely fast’ AF speeds I guess.

  • Roger Cicala

    Hi AJ,
    Not sure if this is what you mean, but with some lenses we’ve found they’re more scattered coming from one direction in focusing (near to far, for example) than another.

  • AJ

    Focus accuracy of modern digital cameras with AF has always been something that I’ve wondered about.
    Thanks for the time to do this Roger – as usual your contributions on photography are well considered and a real asset to the community.
    One thing you might try with PD AF would be to manually move the focus ring to the extreme and then AF and see what you get – the point being that any system has inertia and the velocity achieved in AF when working at it’s maximum might have an influence – i.e. inertia involved in ‘stopping’ the focusing mechanics at the right focus point might have an influence. Just a thought 😉
    As always, best regards,
    AJ

  • Max

    I’ve almost only ever tested lenses with manual focus but was a little shocked when the Sony FE55 Imatested better using AF on both the A7r and NEX7 than I was able to do manually.

    I have always run an HDMI out to a large monitor to Imatest the NEX7, I found that the A7r viewfinder gives better detail than a 2560 x 1440 monitor. I know there are more than 1M less pixels in the EFV than the monitor but somehow it works better for me.

  • Roger Cicala

    Ah! Sorry Dave, I didn’t get that. Yes, of course, that could affect the clusters.

  • Dan Lehman

    “All 10 are at the same shutter speed, of course, but the slap vibrations could be slightly different with different shots” : Okay, but might there be different spreads of clusters of all-at-same… if the shutter speeds differ; i.e., if you take 10 shots at 1/500th vs. 1/30th –that was more my question.
    Thanks,
    –dl*

  • Roger Cicala

    Someone, that’s a good point. I should have mentioned two things. First that, yes, if I run the exact image a dozen times the calculations do come out exactly the same.

    But electrical interference is certainly a possibility. There are a couple of cell towers within 200 yards, and of course all the transformers and high power electrical lines most urban areas have. That may have more to do with the shot-to-shot variation than anything else.

  • Someone

    Roger, you said: “Here’s a simple demonstration… Then I took 10 repeated shots changing nothing… Variables like minute lighting changes, mirror or shutter slap differences, perhaps even electrical interference or trucks driving by the building cause this slight variation.”

    Sometimes it’s good to make sure our calibration & testing devices are accurate themselves. I’m not saying you don’t and you probably have done this before, but have you run the same image through Imatest multiple times and receive the exact same numbers/results?

    I agree that there could be minute external influential differences in a shop like yours, but I bet most of the variation anyone would get in an ideal situation as yours is more along the lines of what I think you meant by “electrical interference.” From the camera’s perspective, preparing the image sensor for capture, reading the results after exposure, and then performing the conversion of readings to image “data” introduces enough variation itself that the image will be different with each shot. Take a picture completely underexposed (or in complete darkness, or even with the lens/camera cap on)and watch the noise change from shot to shot.

  • Roger Cicala

    Dan, shutter slap could certainly be one of the things causing small variation. All 10 are at the same shutter speed, of course, but the slap vibrations could be slightly different with different shots.

  • Dan Lehman

    Roger, you wrote “I set up a Canon 70-200 f/2.8 IS II on a 5D Mk II and obtained the best possible focus I could. Then I took 10 repeated shots changing nothing” : is shutter speed a possible variable –you don’t mention it (but do mention shutter slap)? There is an on-going discussion about the new A7r and some possible issue it has at certain shutter speeds (around 1/100?).

    Cheers,
    –dl*

  • I suspect that many of your readers who follow your suggestion–especially if their camera is mounted on a decent tripod–will be amazed at how much sharper their images look. I’ve been shooting for over 40 years and my experience is that dodgy, inconsistent autofocus, mirror/shutter vibration, and camera shake affect image sharpness a lot more than the lens alone.

  • Roger Cicala

    Steve,

    Basically you’re doing exactly what I recommend – ISO 12233 charts are the key.

    As to the “I tell 90% of people I can’t tell anything from that image” it’s for the following reasons:
    1) They ask about the center sharpness, which is the last thing to be affected by an optical issue.
    2) They haven’t sent a lined-up, 2-D image. It’s difficult to tell equidistance in a 3-D picture.
    3) They’ve autofocused, hand-held, etc. which introduces a lot of variables.
    4) The image has simply resolution-dependent targets, like AF 1951 targets which aren’t very good for testing, especially since people usually print them on an ink-jet or low-res laser printer.

    On thing I will mention, though, decentering doesn’t just make one side different than the other. That is the most common symptom with a tilt and the second most common with a true decentering. The most common with true decentering is all 4 corners being nearly equally muddy. Spacing errors don’t cause side-to-side differences like tilt, generally, they cause overall softness, increased flare and aberrations, and fringing.

    Roger

  • Roger, I appreciate your tests and agree. However, this statement really puzzles me: “But first, let’s talk about why I tell 90% of the people who ask me if their lens is decentered that I can’t tell anything from that image because it’s taken with autofocus.”

    I assume you meant front to back focus changes as opposed to simple “decentered” where the left side is different from the right (or top and bottom, or . . .)- with proper care (everything being aligned), this is easy to test. I use three ISO 12233 charte taped to a double garage door. Newspapers can also work fine. The key here is proper alignment!

  • Roger Cicala

    Tom, for most cameras AI servo is less accurate for nonmoving, center point only than standard AF, but there are a couple of cameras that it seems to be as accurate. Of course, in the real, 3-D world, AI servo is often better.

    Hand held, especially using magnified live-view is pretty accurate for focus, but for reasons I’ll go into in the next couple of posts, it’s nearly impossible to accurately test a lens hand-held.

    Roger

  • Jim Maynard

    Thanks again for shedding light on a subject of both interest and utility. It is really helpful to see real data done correctly rather than the usual internet opinions without anything to back up the assertions. From your post, it seems that the powers that be have done us a disservice in removing such useful tools as Fresnel/split focus screens from modern DSLR cameras.

  • Roger,
    Great start to what will be an interesting and helpful series. The way you showed variation in each focus method was enlightening. I would be curious to see how handheld with good technique compares. Are certain cameras more forgiving, or are AF methods like back-button focus using AI servo more accurate than one-shot. I have a lot to do now with my own gear.

    Thanks,

    Tom

  • Thanks for your efforts! I am looking forward to part 2.

  • Peter Bruggemans

    Roger,
    Thanks for putting your time and energy in sharings your experience with us.

    I find it reassuring, because in my own experience, it’s only on some occasions that I doubt the performance of a certain lens.
    And after a honest(!) review of the situation I can find a lot of other causes for missing the picture besides the lens, like slight camera shake, bad light/contrast or trembling hot/cold air.

  • Keith

    Eagerly waiting for the rest of the series!

  • John Robison

    Never thought to ‘test’ my lenses. Throughout the decades I have tried a couple that were bad enough to return but mostly they seem ok to me. Of course this is only a hobby and I’ve only used a total of about 10 to 12 lenses and only 2 or 3 regularly and they are primes and all used with film. With my bathroom/darkroom set up I only make 6X9 inch prints for 35mm and 6X8 inch for 35mm half frame so not too much demand on the lens with a maximum 8X enlargement factor. However this should be a fun series to follow.

  • Roger Cicala

    Lynn,

    We have done that with some lenses using Imatest. The sheer volume of work we have to do (there are over 10,000 lenses in stock and we have to keep them all repaired and working properly) make it impossible to do Imatest results for all lenses, but we usually publish a group of 40 to 60 when we review them. There also have been changes in testing methods over time: The newer backlit film charts are needed for the highest resolution lenses and cameras, we have to repeat each lens on every camera, etc.

    We have a high power optical bench on order that we hope will let us test much more rapidly (it can take two days to run a set of 60 lenses through Imatest, the bench may do 120 a day). Our first next priority is to test every lens sold through Lensauthority.com (our used lens sale site), but once we’ve caught up with that we hope to publish a lot more group tests and develop a database like you mention.

    Roger

  • I think LensRentals could provide a real service to photographers by publishing charts of the groupings of the many, many copies you have of the same lens model. It seems like it would be interesting to have these with and without the camera body …
    * Grouped chart of 20+ Canon EF 50 f1.2L’s with optical bench; on body
    * many, many other lenses

    I’ve been impressed with the lens reviews from photozone.de, but I believe they are limited to one copy. LensRentals could take this to the next step, or further, by testing multiple copies.

  • Aaron

    This is going to be great, and a wonderful thing to know. Thanks a ton!

    Also, if you can specifically highlight any of these methods you think might work fine in a camera shop, it’d let us test a specific lens in shop to get a decent to good copy. Or at least not a bad copy.

  • A

    Thank you Roger; needless to say I await the next installment with baited breath!

Follow on Feedly