Technical Discussions

Autofocus Reality Part 3A: Canon Lenses

Published July 27, 2012

Sometimes I get to write cool, new stuff. Sometimes there’s just grunt work to be done, usually of my own making. This is one of those times.

To bring you up to date if you’ve landed here first, this is how I got here. Autofocus Reality Part 1 showed that on center-point, single-shot autofocus, standard phase detection was less accurate than manual focus and LiveView contrast-detection AF. No surprise there.

Autofocus Reality Part 2 showed that two of Canon’s newer lenses, the 24mm f/2.8 IS USM and 28mm f/2.8 IS USM, seemed to have much more accurate phase-detection AF when shot on 5D Mark III bodies but not on 5D Mark II bodies. That was pretty surprising. To me at least.

That made the next step very obvious and not particularly fun.

We need to find out what other Canon lenses have the more accurate AF. If there are any, we’ll see if we can determine when this change occurred. Then we need to compare the other Canon bodies.

This post will simply compare Canon lenses (shot on the 5D Mk III) to see if we can find which ones autofocus more accurately.

Assuming what we’re seeing is real, I made the assumption that recently released lenses would be more likely to autofocus accurately. (I have a background in science. Meaning, until someone besides me reproduces this finding, it’s just a blip. Perhaps there are other explanations to what I found. Maybe the Imatest charts I use are just more accurate at 24mm or something.)

So I started testing the most recent releases and then worked backwards. That, I should point out, makes a big assumption.

My method assumes that the lenses are designed and developed in the order they are released. That’s probably not 100 percent accurate, particularly given Canon’s habit of announcing things about a generation before they actually appear on the street.

Today’s Test Subjects

I set up the following lenses for testing on 5D Mk III cameras over several days, grouping them by year of release:

2012 – 28mm f/2.8 IS, 24mm f/2.8 IS, 40mm f/2.8
2011 – 300mm f/2.8L IS II (I would assume all the “II” telephotos will behave the same way. The others are too long for me to test accurately in our lab).
2010 – 70-200mm f/2.8L IS II, 70-300mm f/4-5.6L IS
2009 – 100mm f/2.8L IS Macro
2008 – 24mm f/1.4 Mk II

Previously I’ve been putting up graphs of the individual results. They’re pretty and all, but they take up a lot of space.

A quicker way to compare for this article will be to look at the standard deviation (SD), a measurement of how much the numbers vary. In all of the previous testing, LiveView AF groups with 10 test shots had SDs between 10 and 20, as did phase-detection autofocus on the Canon 24mm and 28mm f/2.8 IS lenses.

Phase-detection AF sample groups on all other lenses had SDs of well over 20, usually over 30. I added one older lens, the Canon 70-200 f/2.8L NON IS, because I would be testing zooms for the first time and wanted a baseline. This is a rather blunt tool, so it is not going to be 100 percent accurate scientifically.

I’ll add that I don’t particularly have the time to perform (nor do I expect you have the inclination to read) a scientific statistical analysis of the data involved. I consider this just a screening test–I’m just looking to see if there might be something worth investigating further.

Results

For brevity I’ll simply place the results in a table instead of showing a bunch of graphs.

The standard deviations are the average SD of (a minimum of) three separate runs of each lens for each type of focus. I have started off with some of the lenses from the previous AF posts and continued on to today’s results. I’ve listed the lenses by year of release, since I assumed more recent lenses were more likely to have improved AF.

Lens	LV SD	AF SD	year
Canon 50mm f/1.4	9	34	1993
Canon 70-200 f/2.8	15	31	1995
Canon 50mm f/1.2	15	27	2007
Canon 24mm f/1.4	12	29	2008
Canon 100mm f/2.8 IS L	13	34	2009
Canon 70-200 f/2.8 IS II	14	24	2010
Canon 70-300 IS L	11	13	2010
Canon 24mm f/2.8 IS	10	12	2012
Canon 40mm f/2.8	13	16	2012

At a glance if you look down the LV SD column, which is the standard deviation of the LiveView contrast-detection focus runs, you see the variation is pretty consistently in the teens for all of the lenses tested.

Looking next at the standard deviation of autofocus we see there are accurate focusing lenses (which I’ve marked in bold) and other lenses. The accurate focusing lenses are very, very close to LiveView results.

I didn’t show it in this graph, but the 28mm f/2.8 IS is also an accurate focusing lens. There’s a bit of a surprise in that the 70-300mm L appears to focus with the accurate group, while the 70-200mm f/2.8 IS is kind of on the border. Not in the teens certainly, but it’s the best of all the other lenses.

Both of these were released in 2010, and I’m not sure what this means.

It might be that the 70-300 seems more accurate because it’s being tested at f/4 rather than f/2.8, but we’ve tested other lenses at f/4 and they weren’t this accurate. The 70-200 f/2.8 IS L may be better than the major group of other lenses but not quite as good as the newer lenses. Or maybe not. Testing three copies probably isn’t sensitive enough to make that call.

And the 300mm f/2.8L IS II

There was one last task to be done. The new Canon Supertelephoto II lenses are all recent releases: 2011 or 2012. They replace lenses designed in the 1990s, which were considered some of the best lenses Canon had made. So we have to check them.

We tested the 300mm f/2.8 IS II for a couple of reasons, but first among them was that our Imatest setup starts struggling at 400mm. I felt the 300mm should be representative of the group.

Because I felt supertelephotos were different lenses than the others we tested, we checked both the 300mm f/2.8 IS and IS II on both the Canon 5D Mk II and 5D Mk III.

As expected, neither lens proved to focus more accurately with the 5D II camera, so I won’t go into that further.

We will post graphs, like in the previous articles, of the results of a sample series (one copy of each lens on the same 5D III) as an illustration–they are pretty and all. But I repeated the tests with other copies and received the same results.

First is the 300mm f/2.8 IS. There was a bit of frontfocus (the difference in average between the two groups), which I could have helped fix with microfocus adjustment, but I didn’t have the time.

The standard deviation for manual focus was 20. For phase detection AF, it was 35. I’m not certain why there was difficulty with contrast-detection AF with this lens (most other lenses were in the teens as the table above shows), but it showed about the same number for all three copies tested. Clearly, though, phase detection AF was less accurate, as was the case with all other older designs.

The 300mm f/2.8 IS Mark II, though, did behave differently on the 5D Mk III.

As expected, the 300mm f/2.8 IS II has more accurate phase-detection AF than the older version 300mm f/2.8. In the example above, it is absolutely as good as the contrast-detection LiveView focus. If you prefer the standard deviation numbers, the 300mm f/2.8 IS II had an average SD of 17 with Live View and 17 with phase autofocus.

As an aside, in case you haven’t noticed, I had to change the scales of the resolution graph for the 300mm f/2.8 IS II. It’s an amazingly high resolution lens.

The older 300mm f/2.8 may look bad in comparison, but those numbers are very good. The original version is sharper, for example, than the 70-200 f/2.8 IS II. But the new version is amazingly good.

So What Did We Learn Today?

Well, we learned that on 5D Mk III bodies, the newest Canon lenses seem to autofocus more accurately, nearly as accurately as LiveView contrast-detection focus. And of course, standard phase-detection AF is much faster and more convenient than Live View.

As best as I can tell, the three lenses released in 2012 (24mm, 28mm, and 40mm f/2.8), the 70-300 IS L lens released in late 2010, and the new Mk II supertelephoto lenses are more accurate.

Maybe the 70-200 f2.8 IS II is, but probably it isn’t. I’m not able to make the call since it seems to be somewhere in between the “best” group and the other lenses.

Oh, yeah: we learned the 300 f/2.8 IS II is amazingly, bitingly sharp.

Next up, obviously, will be comparing with other Canon bodies.

I think we can safely assume the 1Dx will be at least as good as the 5D Mk III. We can also assume the 5D Mk II and previous cameras with the same AF system are not. It will be interesting to see where the 1D Mk IV, 1Ds Mk III and 7D fit in this picture.

Roger Cicala

Lensrentals.com

July 2012

Author: Roger Cicala

I’m Roger and I am the founder of Lensrentals.com. Hailed as one of the optic nerds here, I enjoy shooting collimated light through 30X microscope objectives in my spare time. When I do take real pictures I like using something different: a Medium format, or Pentax K1, or a Sony RX1R.

Posted in Technical Discussions

Autofocus Reality Part 2: One vs. Two, Old vs. New

Imatest Macro Results (With Apologies to the Nikon 105 VR Micro)

Roger Cicala

Kenneth,

You can email me at Roger@wordpress.lensrentals.com Unfortunately I may be of little help there. The 7D in AI Servo mode is a place I just never, ever go.
kenneth

Hi Roger,

Could I have your email address to write to regarding an focusing issue i have with my Canon 7D?
For the same low light level shooting condition, the camera’s AF system somehow just fails to focus in ‘AI Servo’ mode but achieves focus in ‘One Shot’ mode – even with a static subject!!!

From my limited experimenting and problem determination, this phenomena seems to occur for Single Point AF, AF Point Expansion, Zone AF & 19 Point AF, although it works slightly better in 19 Point AF mode with the center focus point locking in focus at times.
I would like to send you some images to illustrate my point.
Would you know what could be going on here?

For a semi pro (hi performance) camera that can work up to ISO 6400, one would expect the camera to be able to achieve focus under similar low light shooting conditions, be it ‘One Shot AF’ or ‘AI Focus AF’ for static or low movement subjects in conjunction with wide aperture lens e.g. Canon 50mm f1.4.
Wouldn’t you agree? Otherwise, Canon (Marketing?) should have advised accordingly.

As the AF achieves focus in One Shot AF but not AI Servo AF, I think the camera’s hardware is working OK, but something is not quite right (bug?) with the AI Servo AF code/firmware implementation.

Just so you know, I have contacted local Canon support regarding this but thus far have not had a reply from them yet. Canon tech i am liaising with tried on other Canon cameras (7D, 5Ds)and experienced the same phenomena.

Hope you can advise and assist with an explanation for this Canon AF phenomena I have stumbled on.

Enjoy your insightful articles very much.

Many thanks and very best regards,
Kenneth
Steve Runyan

from recent testing of 3 50 1.4’s, i suggest that comparing AF performance of 1.4 lenses wide open vs 2.8 lenses wide open is apples and potatos. while the consistency of the 50 at 1.4 is crap, it is far better at f2, and essentially consistent at f2.8. is this a consequence of Canon’s feedback tolerance re DOF on older lenses? different response of different lenses not accounted for in software? something else?

i also tested my (cherry picked) 85 f1.8 which appears to have extremely consistent and accurate AF.

note that none of your consistent lenses have apertures larger than 2.8
plevyadophy

Oh OK cool.

Please accept my apologies for my presumptiousness (presuming you didn’t know LC).

Well, the cool thing is you have dealt with the electro-mechanical element i.e. the AF modules, and LC the practicalities and techniques of and for obtaining good focus. Between the two of you, you have got the subject of accurate focus completely covered in my view.

Now what I would like to see is a publication including your combined efforts. That would be the ultimate treatise on accurate focus.

Get to it guys!! The photographic community needs it!!

And if you “name and shame”, that is, publish a list of those lenses that work well with certain bodies and those that don’t, I will lay money on it that you will start to see Canon and Nikon rather quickly updatedating their lenses or at the very least improving their AF fine tune systems (they should allow AF fine tune of at least three apertures per lens to be of any real use). In London, oh I dunno, I think 20 or so years ago, the police started publishing a list of the cars that were most easily broken into and stolen (based on their figures as a result of call out, chases, and recoveries etc). The motor industry cried foul; amongst other things they suggested that the police were by definition engaging in product endorsement which they shouldn’t be doing. Anyway, the motor industry lost the argument, buyers began to gradually shun cars on the police blacklist and then very rapidly indeed car security improved. ;o)

Given your excellent study, and as you alluded to in one of your comments, really the camera manufacturers ought to be open and tell us all which of their lenses work best with which cameras (if they know, and I doubt that they don’t) instead of giving the impresssion (by saying nothing) that they are all pretty much as capable as each other (give or take an update here, and a tweak there).

Warmest regards,

And thanks again for the education.

plevyadophy
Roger Cicala

Hi Plevyadophy,

All the lenses were tested wide open to avoid any possible focus shift. The 70-300 you mention was f/4 when wide open at the focal length tested. But I totally agree with all of your points. I correspond with Lloyd Chambers regularly (and subscribe to his blog. He is, indeed, and invaluable resource and has helped me on many occasions. Not the least of which is “Lloyd’s already done that, so it doesn’t need to be repeated.”.

Best regards,

Roger
plevyadophy

Firstly, I would like to applaud you on an excellent study. Well written and well executed.

Additionally, I would like to add, perhaps, one minor, or major depending on one’s view, criticism; then a suggestion for further study and/or collaboration; and finally offer my take on the AF accuracy in low light issue.

Criticism
==========

In your study you seem to have overlooked the issue of focus shift. Have you over-looked it, or is it not entirely relevant to your study?

Anyway, what made me think you have overlooked it is your comment above where you say: “It might be that the 70-300 seems more accurate because it’s being tested at f/4 rather than f/2.8, but we’ve tested other lenses at f/4 and they weren’t this accurate.”

If a lens at it’s widest aperture, or near widest aperture is being tested against a lens that has to be stopped down one or two stops to make the same aperture, then it is understandable that the stopped down lens may show less focus accuracy. This may be so due to the issue of focus shift due to aperture change.

Collaboration
=============

To me, the leading authority, well the leading authority of the all the bloggers I have seen (and I have read lots) on the issue of focus shift is Lloyd Chambers.

The man is positively anal about lens/image quality. I don’t say that as a citicism of him by the way. If he says a lens is good then beleive me it’s good.

However, having read his numurous posts and after reading your excellent study what I find is that both of you are studying the same thing, namely focus accuracy but coming at it from different angles. Also, it does appear to me that there a gaps in your understanding, which he can fill and vice versa.

I really think you should collaborate with each other on a major study of focus accuracy. What the two of you have produced seperately is already excellent but the two of you working together would produce something awesome, that’s assuming the camera manufacturers don’t have you both assassinated first! :o)

His website is here: http://www.diglloyd.com/ and the subscription only section of his site, which to my mind is a MUST read is here (Making Sharp Images): http://www.diglloyd.com/index-msi.html

Declaration: I am an avid reader of his blog and a subscriber.

Comments
=========

Reading your study was an eye opener with regard to the problems that may be taking place as manufacturers try to make their AF modules work in lower light.

I think this is a classic example of “if it ain’t broke, don’t fix it” and “newer ain’t always better”.

According to Rob Galbraith and otehrs, the AF reliability, at least for continuous AF and tracking AF is/was far more reliable on the “old” Canon 1D Mark IIN than on the newer 1D Mark III and IV models hence why some sports shooters refuse to “upgrade”.

With that in mind and having read your comments, it seems to me that Canon, and maybe Nikon too, have had the solution to the problem for years but refused, for reasons best known to the themselves, to provide it. The solution is simple. In the case of Canon, all they had to do was leave the AF module of the 1D Mark IIN well alone and simply add an AF illuminator to the new Mark III camera body. To make it less intrusive they could have provided a laser type AF assist as Sony once did with their F series cams instead of a red or orange lamp.

It beats me why both Canon and Nikon seem to think that their respective 1D and D3 series cameras don’t need an AF assist lamp but must instead rely on a big lump of a flash gun atop the camera instead to provide AF assistance. Had Canon at least done this, that is provide an AF assist on the camera body, then maybe most of their 1D Mark III problems would have disappeared (that’s of course assuming the beam had long enough reach for sports shooters).

Warmest regards,

And thanks for the education,

Plevyadophy
Slava

Excellent work, sir! Very nice writing! I wonder if you will have time to do the same test for 5dIII+ new tamron 24-70 with VC. Sorry, if you already answered that question. Thank you for your effort. I will try to keep up with your blog!
Roger Cicala

Hi Sean,
I totally agree. That was exactly the point Dave made in one of our conversations that started me down this path.
Sean C

High ISO performance has increased dramatically. Photographers taking advantage of that want autofocus, so the camera need to focus in lower and lower light. Perhaps the ISO range improvements of the sensor have outpaced similar improvements in the AF sensor, leading to a dynamic range problem for the AF designers. They’d have to choose between bright and low light AF performance.
Kenneth

Hi Roger,
Thanks very much for your informative articles.
Your recent articles on AF accuracy have prompted
me to ask why is it that in low light shooting situations
One Shot AF is much more accurate (and useable) as
compared to AI AF for the Canon 7D (or perhaps other
Canon bodies as well).

Very best regards,
Kenneth
Roger Cicala

Greg,

I hope to compare Canon bodies this week, then move on to Nikon. It should be interesting.

Roger
Greg Fischer

Roger –

It is very helpful to see results aggregated across samples. Traditional product testing has ignored the issue of sampling variability. Yours are much more informative because they directly address such variability within and between bodies and lenses.

I was surprised to see how much autofocus precision has improved on the latest Canon body with new lenses. Do you plan something similar for Nikon lenses and bodies? Autofocus may be the one area where the latest Nikons fall short of Canon. I hope not, because superior autofocus was one of the reasons why I chose Nikon FX five years ago.

Greg F
Bob Ewell

Roger,

That would explain it, if they pushed too far for low light performance they could lose it in bright light. This could particularly be a issue if the subject was reflectiveorts uni) and that took it beyond the sensors range(I don’t know enough about how these sensors work to know if it is the ambient or reflected light that impacts sensor, but I would guess reflected). If this is true would a slower lens(say 4-5.6) possibly work better in these conditions by bringing the light back within range of sensor? If I recall correctly Rob Galbraith seemed to have the most issues in bright light with the Mark iii, but said it was extremely quick to grab initial focus indoors. Possibly the enhanced sensitivity also made it more skitterish while tracking, particularly in bright light. Maybe they need a way to change these settings in camera depending on subject and light level

Or you might have to bring in some slower glass.(my guess is many shooters(particularly Canon) moved away from the 600 f4 with the 1.3 crop sensor bodies.

Bob
Roger Cicala

Bob,

I have no idea, but Dave Etchells at Imaging Resource brought up something the other day that I hadn’t thought about: AF sensors are still just sensors and have a dynamic range like any other sensor. His thoughts were that they might be tuned for low light, bright light or something in between. The comments you’ve made make me wonder if, since everyone was complaining about poor AF in low light, that perhaps the manufacturers went too far and made the sensors better in low light at the cost of bright light accuracy.

Roger
Bob Ewell

So Roger, as if you don’t have enough going on I wonder if in these adventures you have come across any reason why Canon, and now it appears Nikon current generation AF cameras appear to struggle at times with bright lighting? From what I can tell the mark iii appeared to struggle in these situations at times, and it seems not everyone was sold on The IV either. On the Nikon side I read something by a prominent reviewer indicating that bright lighting may not bring out the best AF in D4. Any ideas as I am looking to sell some older cameras but might keep them if they AF is better in bright light. Could it be that there is a point of diminishing returns in sensitivity. If I recall the marki iii was lauded for quick initial focus acquisition in low light, likewise I have read the D4 has improved in this area, but now also may be a little skittery in bright light.

Thank you

Bob

Thank you

Bob

Thank you

Bob
Roger Cicala

HI Greg,

Thank you for that: it was well thought out and very logical. I’m definitely going to do the T4i and I’ll add the 60D.
I’m interested in the 1DsIII largely because I suspect it was an AF dead-end with the separate processing unit. I’m hoping, since this test isn’t very stressful given the single nonmoving center point, that it actually is better. But the more I think about it, the more I suspect you are correct. It will be fun finding out.

Roger
Greg

Roger, I like that you will be looking at additional bodies, but I would like to add to your work load a bit (sorry) by including in your comparison the 60D and T4i.

I think there are multiple variables pertinent to the bodies beyond release date including PDAF sensor, algorithms, and processors.

The 50D (9/2008) and 5D MkII (11/2008) were the first single Digic 4 cameras (with different PDAF sensors) and both received firmware updates 2/28/2012 (for CF card reasons). The single Digic 4 line ended with the 60D (9/2010) and T3i/T3 (3/2011) (again with different PDAF sensors). Of these, the most recent firmware update was for the 60D on 6/19/2012 (for serial number reporting). The 60D focusing algorithms also incorporate color and luminosity data from the metering sensor for additional accuracy. If a single Digic 4 body were to show the PDAF performance improvement you’ve reported, I think it would be the 60D. I would like it if you included the 60D in your test group.

I would agree the 1DX with it’s dual Digic 5+ and same sensor as the 5D MkIII with it’s single Digic 5+ would probably perform similarly.

It would additionally be interesting to compare results with the T4i with a single Digic 5, similar/same PDAF sensor as the 60D (but different metering sensor).

You have already named both dual Digic 4 cameras, the 1D MkIV (12/2009) and 7D (10/2009). I note that the 7D firmware has not been updated since v1.2.5 on 4/25/2011 (and would like to see v2 compared when released next month). Curiously, the firmware for the 1D MkIV (v1.1.1) was released 3/29/2012 specifically to address exposure with the 24 & 28 f/2.8 IS USM lenses! (Wonder if PDAF algorithms were updated too.)

The 1Ds MkIII is older with dual Digic III and a “separate processing unit devoted solely to autofocus calculation”. Its current firmware (v1.2.0) was released 12/6/2009. I will be very surprised if you see the improved PDAF performance with this body, at least without a firmware update.

Thanks Again,
Greg
Stv

Super. Please keep your postings coming. This is all very, very interesting.
jim thomson

Roger,
Don’t forget to include the new T4i with its hybrid AF system. Be interesting if it does better with the new lenses.

It would also be interseting to know how the 7D performs before and after the firmware upgrade that supposed to be available next month.
Roger Cicala

Thank you, Kirneh. Criticisms of that point were certainly valid. I had oversimplified in a place where I shouldn’t have. Constructive criticism is always welcome: it helps me learn and helps everyone else understand.

I’m glad you like this series. I’m having fun, largely because I have no idea, really, what is going on or what we’ll see next. And I’m enjoying ‘telling it as I do it’ rather than waiting until I know what the ending is.

But it’s been a long while since a history article. I’m feeling the itch to do another.

Thank you,

Roger
Roger Cicala

Thank you Greg. You’re correct, I’ve fixed that error.

Roger
Greg

Fascinating info — THANKS!

I wonder (in advance) if your results for the 7D might change with the v2 firmware due out soon. Looking forward to your body tests.

Greg

(Note: In your list under “Today’s Test Subjects” for 2008, I think you meant 1.4L instead of 2.8)
L.P.O.

I recently made some criticism of some DoF issues in a previous article of yours. Although I also told there how much I usually respect and like your articles, I just want to point out that for me these new AF articles are pure magic (my other favourite have been your photography history articles, particularly the coatings episode).

Looks like you’ve started with testing something that is as clear as day: “Of course LV AF is more accurate than PD AF”, and suddenly dropped in a mirror world.

After all, AF accuracy is an aspect DPReview and other test sites and magazines never formally test – and you seem to have learned something genuinely new. Can’t wait to see how this will turn out!
ShooterMcGavin

Oh, and hooooooooray about doing the new Nikon stuff 🙂 Looking forward to it!
ShooterMcGavin

You guys should provide a service for folks to ship you their whole kit, and do a fine tune with all their glass on their bodies with the Imatest setup. That’d be pretty sweet.
Rory

Whoops, missed the above comments .
Rory

Thanks for this interesting and timely research Roger.

I have noticed a significant difference between LV and phase focus on Nikon equipment, including the D800 and D4 with 300mm and 600mm lenses. If you get a chance, it would be interesting to confirm this. It appears that Canon has made a real advance here, which could be very significant for sports and wildlife shooters.
Roger Cicala

Hi Dave,
Yes, they’re all copies less than a year old, but designed back when. No question the manufacturers will change things during the lifespan of the lens: we’ve noted changes in mechanical parts, certainly. I suspect changing the electronics wouldn’t be that hard, but it doesn’t seem to have happened.

Shooter and Dave: Nikon is on deck for sure: as soon as I finish the Canon cameras we’ll be moving on.

Roger
ShooterMcGavin

Roger, great detail and interesting research as always! Any plans to do the Nikon system with their most recent bodies and lenses?
Dave

For the older lenses, I assume these were recently purchased older design? Meaning, Canon wouldn’t slip a tweak in an older design and improve the same circa 1993 lens bought today would they?

Also, since you are getting so good at this, how about Nikon’s new AF system in comparison?

Roger, Thanks again for all your hard work, your blog posts are awesome, and I love when you start peeling the onion 😉