Technical Discussions

Autofocus Reality Part 1: Center-Point, Single-Shot Accuracy

Published July 18, 2012

Once upon a time I wrote an article attempting to explain phase-detection autofocus and its limitations. That was a lot like trying to explain what happened to the Mary Celeste or why Stonehenge was built.

Actually we probably have a better handle on the Stonehenge thing than we do on the Mary Celeste or phase-detection autofocus. So rather than rehashing and speculating, I thought I’d instead do some simple demonstrations and explorations.

Obviously there’s a lot of ground to cover, so rather than making a 162-page blog post, I thought I’d break this up a bit. This first part will be very simple. We’ll compare the accuracy of phase-detection AF, contrast-detection AF, and Roger-detection MF (that would be me manually focusing using 10X LiveView) on still targets using just center point AF. While we’re here, we’ll demonstrate just what AF Microadjustment does, and doesn’t, accomplish.

The Setup

We use Imatest to demonstrate AF variation using Canon full-frame cameras and 50mm f/1.4 and f/1.2L lenses. The reasons are pretty straightforward: Our setup involves shooting a still test target with star, vertical, and horizontal focusing aids in the center from a tripod, so we should be getting very accurate AF. The setup we’re using involves shooting at a 12-foot distance so the depth of field is narrow. If focus is off by just a bit, Imatest will detect the decrease in resolution.

Phase-Detection AF Accuracy with an Older Design

We started with a Canon 50mm f/1.4 lens mounted to a 5D Mk II camera. The first thing we did (deliberately a bit out of focus so it would show up easily) was set the lens on manual focus and take eight repeated images without touching anything. This gave us a baseline (represented by blue diamonds) of the variation in testing procedures if nothing else changed.

Then we took eight repeated shots using LiveView and manual focus (represented by red squares), spinning the focus ring between each shot to either infinity or absolute close up. The red squares demonstrate Roger Units (how well Roger can manually focus given all the time in the world on a tripod with a perfect test target.)

Finally we did the same thing but let the camera autofocus in LiveView (represented by green triangles), resulting in contrast-detection autofocus. In theory this should be as accurate as Roger is, perhaps more so.

The graph below shows the results of those shots. For those of you who are not familiar with our Imatest graphs, the numbers reflect the sharpness of the image in Line Pairs / Image Height. The sharpness in the center is shown on the X-axis and average sharpness on the Y. Higher is better and, in this test, better focus equals higher sharpness.

 

The graph shows that there’s a little bit of variation when we take several shots without changing anything (blue diamonds) which probably reflects tiny movements from mirror slap, minute changes in lighting, or possibly even tiny fluctuations in sensor performance. Remember the blue shots are purposefully a bit out of focus so they’d show up separately in the graph.

When we tried to manually focus accurately (red boxes) or let the camera do it with contrast detection there is a bit more shot-to-shot variation. But the variation is still small, as is the difference between the camera and Roger. To put some numbers to it, the standard deviation of the baseline shots was 5.5; for the Roger-focus shots 11.5; and for the LiveView AF shots 8.4.

Now let’s throw away the repeated shot results from above since we’ve made that point, and replace them with standard (phase-detection) autofocus shots. These are taken in exactly the same way as the live view AF shots: take the image, spin the focus ring to one extreme, let the camera refocus, save the image.

 Two things become obvious:

1) When phase detection AF gets it right, it’s every bit as in focus as contrast detection or Roger detection.

2) Four of these 10 shots aren’t quite as accurate as the other six. That sounds like a ridiculously high miss rate but let’s put the numbers in a bit of perspective.

If we believe in Subjective Quality Factor (SQF) then an SQF difference of about five is needed to see a difference in a reasonable print or pixel peeping at 50 percent, given that the two lowest blue diamonds (phase AF shots) are going to show up as a bit soft or out of focus if we’re really critical. The third lowest might or might not look different from the “good shots”–it’s right on the edge.

Still, that gives us a “missed focus” rate of 20 percent in this little test, and that’s using center point AF with a still target. That’s not a big sample size obviously, and we repeated it with several other 50mm f/1.4 lenses and got “missed focus” rates of between 10 and 20 percent for all of them.

That’s not horribly out of focus, but it’s definitely pixel-peeping out of focus. For those of you who like numbers, the standard deviation for phase-detection AF ranged from 25 to 44 on different runs (always greater than the 5 to 15 we got with LiveView AF.)

Demonstrating Microfocus Adjustment

Next on our trial list, we thought we’d compare with a newer designed lens. The Canon 50mm f/1.4 doesn’t have a real USM motor and is known to be a bit difficult to focus. Not wanting to make the test too easy on the camera (and because we were set up at 50mm), we grabbed a few Canon 50mm f/1.2 lenses. These lenses have a modern design but again are known to be a bit “focus challenged”, to use the politically correct term.

One of the first copies we shot gave us the opportunity to demonstrate microfocus adjustment. We took our usual LiveView autofocus and manual focus charts, followed by a set of phase detection autofocus shots. When we graphed the results, we had to change the axis range because the phase-detection AF shots were so out of focus. Some of the shots still don’t show up; they’re even worse than the expanded axis shows. Unlike the above example, every single one of this is obviously out of focus at a glance–no pixel peeping necessary.

 

What happened was readily obvious: This copy of the 50mm f/1.2 was backfocusing badly on this camera. We did a quick microfocus adjustment (12 points) and reshot the lens with the results below.

Here is a superb example of what microfocus adjustment accomplishes. After adjustment phase detection AF is now very accurate, although there still is going to be shot-to-shot variation. For the two or three of you who like to scream, “I don’t want to us microfocus adjustment! The lens should be perfect out of the box!“, we went ahead and put the same lens on a different body with no microadjustment.

On camera No. 2 the same lens autofocuses accurately. The other 50mm f/1.2 lenses we tested all autofocused accurately on the first camera. The lens is perfect, just not with the first camera. The first camera is fine with all the other lenses we tested. Sometimes a given lens that’s fine doesn’t match up with a given camera that’s fine. So it goes.

The conclusion is pretty obvious: If you want to shoot wide aperture prime lenses and you don’t want to use microfocus adjustment, you just refuse to cope with reality.

We did several runs with 50mm f/1.2 lenses and found similar results all the way through. Standard deviations for sets using LiveView AF were 11 to 20, while phase detection ran from 20 to 35. These results are similar to what we saw with the 50mm f/1.4.

Conclusions

This really wasn’t a “conclusion” post; our intention was to just demonstrate some autofocus basics that most people know already.

  • LiveView (contrast-detection) AF on a still target is more accurate than phase-detection AF. It should be so. Contrast detection is using the actual sensor to determine focus; phase detection is not. Overall we found about one shot in 10 was out of focus with phase detection.
  • LiveView AF is as accurate as Roger View MF. You may be better than this, or you may not.
  • Phase-detection AF has more shot-to-shot variation than contrast detection. It’s not huge, but it’s real. This shouldn’t surprise anyone. Phase detection was developed for fast AF speed and to detect subject movement. It wasn’t developed to be more accurate.
  • Microfocus adjustment pulls good phase-detection AF results up to a par with LiveView, but it doesn’t eliminate the small amount of shot-to-shot variation that phase-detection AF has.

I’ll be expanding on this first article in some subsequent posts. Next I will be comparing the “two beep” AF method (where you push the shutter halfway down to focus then repeat before taking the shot) to the “single beep” method we used here. (I think the “two beep” method is more accurate for no reason other than when I started people told me it was so.  We shall see.)

We’ll certainly do some comparisons with different lenses on the Canon cameras and try to see if newer designs, STM, or other lens changes reduce the AF variation. Obviously we’ll have to get a larger database to be able to detect subtle changes, but that’s just a matter of repetition. (What we showed today were just examples. We’ve done a larger number of these lenses already.)

We will also look at different camera systems’ AF to see if they seem to have less phase-detection variation. I had thought about comparing contrast-detection systems in other cameras, but it’s obvious we’ll struggle to find one that’s better than Canon’s.

I’m not saying Canon’s is the best, but it’s as good as what I can do, so it may be difficult to detect “better” in this situation. We might be able to detect worse, of course.

Finally, within limits, I’m asking for suggestions about other things we can look at.

Remember when you make suggestions, that this model requires a fairly narrow depth of field to detect differences. Very long focal lengths (over 200mm full-frame equivalent) test at longer distances which may increase depth of field too much for us to detect small differences. Narrow maximum apertures (f/5.6 or so) would have the same effect.

Given those limitations, though, I’m open to suggestions!

 

Roger Cicala

Lensrentals.com

July 2012

Author: Roger Cicala

I’m Roger and I am the founder of Lensrentals.com. Hailed as one of the optic nerds here, I enjoy shooting collimated light through 30X microscope objectives in my spare time. When I do take real pictures I like using something different: a Medium format, or Pentax K1, or a Sony RX1R.

Posted in Technical Discussions
  • lexvo

    Great article!
    I think it would be great do to the same test with the Canon 5DmarkIII to show how much AF has improved.

  • Lester

    Argh, what were my fingers thinking, that should read, “bet the farm on CDAF”…!

  • Lester

    You say that Canon’s live view PDAF is as good as it gets, but I’d like to join the (small) chorus that asks for some runs using an Olympus (or Panasonic) mFT system — they’ve bet the farm on PDAF, and it would be very interesting to see whether your tests might show a difference… Great blog!

  • * Longer focal lengths don’t alter DoF in a significant way. They affect perspective, so the out-of-focus background seems more out-of-focus, but with what you’re doing here there should be no extra issues (you’re shooting a flat surface, there’s no background).
    http://www.similaar.com/foto/dof/dof_notes.html

    * I’d like to see if in-plane phase detection helps (it doesn’t require AFMA, and may be constrast-detection-enhanced) or is a problem (having to be buried between the photosites may make some tradeoffs arise).

  • Esa Tuunanen

    Max, at least Olympus offered possibility for separate focus adjustment value for zoom’s widest and narrowest focal length with camera interpolating values for in between focal lengths.
    That feature must have been too advanced for other makers if they don’t have similar…

    Integration of PDAF into sensor would solve at least inaccuracies coming from SLR PDAF measuring focus in elsewhere than actual focal plane and beyond rather complex optical path with lots of parts needing very precise positioning to have any changes for correct focus.

    CDAF’s accuracy comes from measuring focus from real focal plane/imaging medium.
    But unlike PDAF which has from the first sample good idea how much and to which direction focus must be moved CDAF can only see that image is either probably blurred or maybe not blurred (even with spot on focus it can’t know it from first sample) which slows things down because it needs lens to move focus in tiny steps while sampling focus constantly to know if things are going for better or worse untill highest contrast is found.
    Faster sampling rate (more images per second from sensor) allows giving focus commands at faster pace if lens can handle it.
    Also from enough consequent samples more complex focus algorithms can estimate how much further focus can be moved directly to get close to correct focus speeding process by skipping over number of smaller focus movement steps and also lessens focus overshoot.
    That’s how m4/3 cameras have increased their focus speed.

    But of course all that speeding of CDAF focus process by estimations flies out of window if target keeps moving.
    Which is why Nikon 1 is only mirrorless which can properly track moving target because its phase detection can tell instantly from next focus sample at what rate and to which direction target is moving.

    Applets here show very well operating principles of imaging sensor CDAF and SLR PDAF.
    http://graphics.stanford.edu/courses/cs178/applets/applets.html#autofocus

  • Arun

    A while back the FoCal proprietor had tweeted that the 5d3 had one of the most consistent AF his software had ever encountered. So, if you could repeat the tests you did with the 5d2 with the 5d3, I’d much appreciate it.

  • Budi

    Roger: as long as the tools being used are calibrated/consistent, they don’t have to be 100% accurate. That would allow correct statistical analysis. Since you test every lens you have, it’d natural to know what you could get from that huge database. 🙂
    I’m no expert, but I know a little about experiment design and reliability life calculation, email me if you want more info – gratis! 🙂

  • Volker Herrmann, I like your suggestion of auto alignment considering Roger’s article shows that contrast focus is pretty close to the best you can get.

    In regards to suggestions, I’m surprised nobody’s brought up the Canon 5D mark ii. I see so many forum postings about it having awful focusing. I think the test I’m thinking is regular light vs low light accuracy, comparing it to something with a comparable middle focus point like the 50D I believe. Also people talk about the the assist focus points and I’d be curious to know if those actually do anything to help or hurt. I guess one other possibility would be testing if it’s mostly the camera that affects the focus accuracy, the lens or a combination of the two. That’s all I got.

  • Roger Cicala

    Budi,

    I like what you have in mind. I’m hoping we’ll end up with a bit of a database that shows us how accurate different lenses, cameras, etc. are. I may be off the mark, the ‘tool’ we’re using may not be accurate enough. Or perhaps there isn’t a difference to detect. But I want to know that, too.

    I think FoCal is a really good choice: it’s not perfect but it’s pretty accurate and very easy to use.

    Roger

  • Roger Cicala

    Hi Gary,

    I like Fo-Cal very much, although I find it better if I actually am looking at what it does, rather than just putting it on full autopilot.

    One thing that’s important to remember, though: it’s going to set your lens up for perfection at one focusing distance (and one focal length if it’s a zoom). But the results will vary a bit at other focal lengths or focusing distances. The bottom line is it’s best to adjust to the distance and focal length you tend to shoot at most.

    Roger

  • Max

    It would be very interesting to see how the same camera/lens behave at short vs. long distance. Example: trying to shot a target ad 10 foot, and then the same target at 100 foot.

    And then check what kind of difference autofocus microadjustment can make.
    If I microadjust focus for a lens using a 10 foot target, what happen when I shot the same target ad 100 foot? does the lens, microadjusted at 10 foot, retain its accuracy even at 100 foot?
    And what happen if I microadjust a lens at 100foot, and then shot a target at 10 foot?

    And what happen when I microadjust a zoom lens? Is better to microadjust zoom lens at their widest position? or ah their tele position?

    And what happen if I’m using a real macro lens (like the sigma 180 macro)?
    Does the focus microadjustment retain it’s precision even when I’m using the lens for shotting closeup/macro subject? (i know that real super-macro lenses (M-PE 65) dosen’t have focus ring… but I’m not going to setup a tripod and a camera rail just to shot an apple 🙂 )
    And what happen when I’m using a near-macro zoom lens? (like the sigma 17-70mm)

    Thanks for writing these kind of article, I find them very-very-very interesting 🙂

  • intrnst

    ‘m joining the chorus, is there any (easy) way to test predictive PDAF?
    If so, how better is the efficiency against a very fast CDAF system, bearing in mind this Subject Factory… Quality… thingy.

    Just keep writing.

  • Just bought/downloaded the Fo-Cal system to test out my camera (5d mk ii) & lenses – it seems to do testing akin to yours for phase-detect, and comes up with the best MA setting, all auto-magically. It was therefore really interesting to read this article as it crystallised many of my thoughts I’d had doing it …

    Everything is going OK so far (haven’t calibrated the longer focal lengths yet as the weather has been so bad), but interestingly enough it was concerned over my Sigma 70mm macro and said it was too inconsistent! It was happy with my short Canon L zooms, and I’m looking forward to see what it makes of my old 50mm 1.8, which works amazingly well for copying artwork as it’s so sharp out to the edges (using MF & liveview x10).

    Would be very interested to hear your views on the Fo-Cal system,

    Thanks again,

    Gary

  • Armis

    I’ll echo Ed’s and Walter’s requests: I’d be really interested to know how bad the focus accuracy drops ou outside points, and I’d be hugely interested to see if there’s any statistically significant accuracy drop when using AI Servo.

    I’d actually probably be ready to pay for a database that plots, say, 100 AF shots (one-shot and AI Servo, if one can dream) on each cameral-lens pair. I’ve been looking for a 50 (or 50-ish) with a large aperture and AF performance that doesn’t suck, and after trying Canon’s f/1.8 and f/1.4, let’s just say I’m still in the market. And I’d really prefer not having to actually try all of them.

    Brilliant post regardless, I’ll be waiting for the follow-ups with baited breath.

  • Volker Herrmann

    Why is there no funktion in our cameras, that do an lens auto-calibration between the focus difference of phase and contrast AF shooting a given testchart?
    I could imagine this is only missing software.

    Thanks for your intersting articles!

  • Stv

    Found out about your blog just recently. I really love it. All the details, but easy to follow. I started with the postings about the new mFT lenses, your Canon lens tests. And now this series about autofocus types is super exiting.

  • Budi

    Roger: it’d interesting to take the raw data and make a statistical distribution plots to analyze whether there are other influential factors or even interaction factors, that contribute to the event referred to as missed focus. If we remove all the nuissance factors (operators, etc) and all the related ones, Lenth could be used to calculate which factors are dominant (focal length, motor type, aperture, contruction, temperature, etc).

    Back to fun, what tools/software would you recommend for a proper and easy to use AF Micro Adjustment?
    Imatest sounds like a good tool, but not many of us could justify the cost of owning one.

  • Walter Freeman

    Two things:

    1) Some people have suggested that the high speed of the contrast AF on newer Micro Four Thirds cameras, particularly the EM-5, make them less accurate than the “slow deliberate” contrast AF on the Canons (or whatever). You might repeat this test with the an EM-5 and see what happens.

    2) Does “AI Servo” give the same accuracy as single-shot AF on Canons? (My father has never been able to figure this out with his 7D.)

    3) Olympus claims to use an “interleaved/hounds’-tooth” layout of AF-sensor pixels that yields greater accuracy in their 4/3 SLR’s. In my experience, it definitely does seem to be pretty good, with less variability in some cases than Canon AF. I guess I am one of eight people in the world that still cares about legacy 4/3, though. 😛

    4) I notice that my macro lens (Zuiko 35/3.5, a great little lens otherwise) can have serious problems with phase-detect AF accuracy at long distances. Are macro lenses consistently worse in this regard since they have a longer range to worry about, or different gearing?

    Thanks again for all your hard work!

  • Nate

    Macro extension tubes would probably let you use longer focal lengths. And possibly get you insanely shallow DOF, depending what combination you ended up choosing.

    I don’t think that this would change the overall characteristics and render the whole exercise pointless…?

  • asad

    @Bret Douglas — testing BBF is pointless. BBF vs. halfway pressing the shutter button is just a change in what button tells the AF system to do its thing, not in actual hardware. Why would you even think BBF is more or less accurate?

  • Chris K

    High precision versus low precision AF points might be an interesting test. Back when I was paying attention to such things I remember Canon spec’ed high precision AF points as “within one third of the DOF” while low precision points were “within the DOF”. I don’t know what their definition of CoC is… I suspect they are using the same CoC they used with their film SLRs, which isn’t appropriate for a camera like the 5D2, IMHO.

    I’d also love to see a Canon versus third-party EOS lens comparison. The Sigma 50/1.4, perhaps? I finally lost my faith in PDAF when I bought an EP1 that nailed focus every time, but my 5D, 40D, and 7D were all over the map. At first I thought it was my Sigma 50/1.4 and its (presumed) reverse-engineered AF algorithms, but I discovered the same lack of consistency in my Tamron 28-75/2.8 and various Canon lenses. Some more than others. Some people say Canon lenses are more accurate than third-party lenses. Is that true?

    I think there are so many questions surrounding AF that people tend to bury their heads in the sand on the issue. It’s very difficult to test AF accuracy well, but there are no lack of testers willing to submit (often flawed) data. We’ve ended up with mythology and traditions surrounding AF because of this complexity. Demystifying AF a bit is a noble cause. Thanks for all your work you do for the community, Roger!

  • Tom

    BTW, your data also supports the fact that lenses with bigger apertures are more difficult to get focused accurately.

    Thanks.

  • A

    Thankyou for confirming my feelings/experience with actual data!

    I’d be interested to see whether a 5d3 would do a better job with the same lenses – or a recent 1D-series body for that matter.

    I’m also mildly curious about the lower end cameras Rebel/Kiss/nn0D too (or whatever it is Canon chooses to name them in the various parts of the planet).

    I do wish the camera manufacturers would stop changing product names every few yards around the world. Seriously guys, one product, one name; it’s not that difficult!

  • Tom

    Great article with very interesting observations.

    I would be more interesting to see a comparison between something like Nikon 50mm f/1.4 AF D and Nikon 50mm f/1.4 AF-S G. Thus, they have the same f-value but different focus motor.

    Thanks.

  • jim thomson

    1. My next camera will definitely have MFA.

    2. Test some of the third party lenses OSM, USD. See how they perform on the different camera makers AF systems.

    3. How good is the contrast detect focus on your OM-D?

  • Siegfried

    Roger,
    I think it’d be interesting to see the DOF within which different camera+lens combinations assume themselves to be in-focus when using phase-detection AF method. I’m no way a canon boy and don’t exactly know how it really looks like in Canon, but I believe that there’re some AF assistance from camera when you are manually focusing (like beeping a beep or lighting up a dot in the viewfinder). Obviously, there is a range with-in which camera’s PDAF engine assumes that we’re in-focus. And that’s what I’m curious about – that very range vs different camera+lens combinations (if you’re stuck with Canon brand, then let them be 5D MkII as the baseline, MkIII to see the progress, 7D as comparison with amateur line and something pro-level to see what pro’s are paying for).

    You could move the target to measure that range in distance units (feet or centimeters) or per cents (range / focusing distance). Or you could twist the focusing ring and refer to distance scale and focusing ring angle play.

    I believe you’ve got the idea.

    Thanks,
    Zig

  • Tim

    Thanks for the information, especially on the phase vs constrast autofocus.

  • Bret Douglas

    Please also test the Back Button Focus accuracy vs. the Shutter button halfway down approach.

  • Ed

    Roger, the next obvious test would be to look at the other focus sensors away from the center. After that, compare cross-type to non-cross-type sensors.

    As usual, your posts are first rate.

  • Steven

    I think it would be interesting to rig up a moving target, towards the camera (on a motorised Dolly), and getting the camera to take 5 successive photos, timed intervals (so that there was sufficient movement) – and plotting the results on an above chart.

    Suggestions for lens – 85 1.8 please 🙂

Follow on Feedly