LensRentals.com

Autofocus Reality Part 4: Nikon Full Frame

Posted by

I started this series to simply demonstrate a well-known fact; that contrast detection autofocus in Live View was more accurate than standard Phase Detection autofocus for still subjects. It ended up being a bit more than I’d bargained for just with the Canon cameras. After that series was done, I promised several people that I would take a similar look at Nikon cameras. I also got several pertinent suggestions and criticisms in the first series about gathering more data and presenting more examples. I promised to do that, too.

You ever notice how easy it is to promise to do some work later on? Only then do you realize that what you’ve promised is going to take about 30 hours to get done. That causes an immediate disturbance in the force, and your day job workload triples because there’s a holiday weekend coming up. Plus, some cool stuff that you really want to play with need to carefully evaluate comes in. So for the 3 or 4 people who were waiting for this, I apologize for taking so long.

The Conclusion

Several people wanted to see more data presented, so I’ll try to show more this time. Most people, though, simply want to know what I found. A few, of course, want to know how they can work in what I found into some raving Fanboy post or other. Sorry, Fanboys, there really isn’t any Canon – Nikon forum fodder here. These tests are fairly blunt tools that are only looking at one tiny aspect of autofocus.

Anyway, if you just want the results, they are pretty straightforward:

  1. The Nikon full-frame cameras all do well with center-point, single-shot AF.
  2. The newer bodies seem a bit better than the older ones.
  3. There don’t seem to be dramatic changes with a few camera-lens combinations like the newer Canon bodies and lenses showed.

There are a few rather interesting things we found this time around. Additionally, my “write as you go, not when you know” technique in this series has gotten several knowledgable people to email more thoughts and information. That helped me accomplish my main purpose: find out more about how phase detection AF works. Which lets me speculate a bit more at the end of the article.

Data

We tested a larger number of lens-camera combinations this time around. As always, we used cameras and lenses that had recently been tested and found acceptable. One series of shots, discussed here, was removed and not included in these results since the lens used was found to be decentered.

I’ll show some example graphs of various lens-camera runs to give an idea of what the actual data points look like. This space is too small to try to give a graph of every data point, so there will still be some summary numbers. The graphs show resolution (MTF50 measure by Imatest) for different lens-camera combinations. One lens tested on one camera is a run – then we changed both lens and camera for the next run, etc. For each run I show the results for 10 shots taken with Liveview (contrast AF) as diamonds, 10 shots with standard phase AF as crosses. The vertical axis is average resolution at 13 points on the sensor (corners, edges, 4 midpoints, center) while the horizontal axis is resolution at the center point (all in line pairs / image height from unsharpened raw files).

Each run (represented by one color) is one copy of the camera with one copy of the lens; no lens or camera repeats. For example, when we tested Nikon D800s with Nikon 28mm f/1.8 lenses, we tested 4 different lenses on 4 different D800s. I’m only showing two runs in the graphs because they get too crowded and run over each other, but all 4 are used in determining the average numbers. I repeat this because tomorrow someone will be saying “they only tested two cameras” because, well, some people apparently find reading too hard and they just look at the pictures.

One other thing is worth mentioning. We look at the standard deviation as a convenient measure of variation between the sample sizes. That’s a fairly valid and convenient measurement when we’re comparing D700 contrast detection to D700 phase detection. When we start comparing D700 numbers to D800 numbers, though, the resolution numbers are quite different. The absolute numbers of the D800 are much higher, so the same perecentage of variation is going to make a slightly larger standard deviation.

Some Visual Examples

The range of the axis on the graphs below change depending on which camera is used. In the Nikon D700 graph, for example, the peak resolution range is 400 to 900 LP/IH. With the D800 the range shown is 700 to 100 LP / IH.

For the D700 and D8000 I only graphed two of the 4 runs to clearly show how contrast detection and phase detection varies. For the Nikon D4 and D800 I graphed 3 runs. It makes the graphs a little more crowded, but in both of those graphs the runs colored red make a point I want to use. They also show why I didn’t try to put 4 runs on one graph.

 

The D700 shows the difference betweeen phase and contrast detection similar to what we’ve seen in the better Canon bodies. Contrast detection gives a very tight pattern, about as good as I could do using careful live view manual focusing with magnification. Phase detection gives some shots every bit as good as contrast, some not quite as good numerically but not different enough that you could easily tell in a photograph, and an occasional shot that has just missed focus.

 

 

The Nikon D3x gave us similar results. I should mention that out of the 4 runs with these two cameras we had 3 completely miss focus with the D700 (one shows in the graph above) and 4 with the D3x out of 40 shots with each. That’s about what we’ve seen with most Canon cameras and similar to the missed focus rate (around 10%) others have reported with phase detection.

 

 

The D800 example above shows something worth demonstrating again, because so many people still don’t get it. For the green and blue runs phase detection is similar to the previous cameras: a bit wider spread (larger standard deviation) but generally close to the contrast detection results.

The red run (run 2), though, shows the phase detection average is clearly a bit lower than the contrast detection. In other words, the pattern is similar to the phase pattern of the other runs, but the average is further away from the best shots obtained with contrast detection. That’s because this lens require microfocus adjustment of -5 on this particular camera. When we made that adjustment and repeated the run, the red pattern looked similar to the others.

The point: microfocus adjustment brings the average up of phase detection AF up closer to the contrast detection range, but it doesn’t reduce the variation seen with phase detection shots. The D4 graph below shows a similar example. For purposes of the averaged numbers I’ll talk about below, we repeated those runs after making appropriate microfocus adjustments and used the adjusted runs for our actual calculations.

 

You’ve noticed in the graphs, I’m sure, that both the D4 and D800 seemed to have fewer ‘bad misses’ than the D3x and D700 did. The D800 had 2 of 40 missed shots, the D4 1 of 40. That’s not clearly significant, but certainly an interesting trend.

Data Summary

The summary of multiple runs using different camera and lens each time for 28mm f/1.8 lens shot at f/2.8 on the various cameras is shown in the table below.

CameraContPhaseAvg Res
D41422689
D8001524834
D3x1636741
D7001230621

It doesn’t look like much, I know, but that table represents 320 Imatest shots. Before we start discussing it, let me be clear: the numbers we’re looking at aren’t a statistical analysis and there’s anyone who tries to say “Camera X was 30 and camera Y was 17, so camera Y has a better autofocus system” is missing the point entirely. We’re looking at one tiny part of the AF system: how well does the phase contrast system compare to contrast detection looking only at the center point, with a still target.

Before we started we all knew contrast was more accurate. When we tested Canon cameras, most had a standard deviation about twice as large for phase detection as for contrast. The Canon 1D Mk IV was more accurate than that. When mounted with a select set of the newest lenses, the Canon 5D Mk III and Canon 1Dx were just about as accurate in phase detection as with contrast detection. With older lenses, though, they were about as accurate as the 1D Mk IV.

With the Nikon full-frame cameras we see that the D700 and D3x seem similar to the Canon cameras: phase detection has a standard deviation about twice as large as contrast detection. The newer cameras, the D800 and D4, seem clearly better, very similar to the newer Canon bodies on most lenses.

Of course, we did check other Nikon lenses to see if there was a subset, like the newest Canon lenses, that seemed as accurate with phase detection as with contrast on the D4 and D800. We did smaller runs (2 lenses on 2 cameras) with the  70-200 f/2.8 VR, 35mm f/1.4 G and 85mm f/1.4 G. Results were similar to the 28mm f/1.8. We’ll check back with the next new Nikon lens release to see if it has improved accuracy on these cameras, whenever that is.

 Discussion

There’s not really any difference between Canon and Nikon cameras for the lenses we all use every day. Both do pretty well, and the newer cameras are a bit better than older models. Of course, that doesn’t mean there’s a huge difference in real life.

A better real life comparison would be “how many shots were badly missed” like the ones you see in the D700 graphs. That degree of miss would be obvious in the photograph. The variation in standard deviation probably would only be detected by pixel-peeping. Unfortunately, it would take several hundred more shots with each camera to get large enough numbers to determine statistical significance for number of missed shots. At roughly an hour per Imatest run, there’s no way I’ll have enough time to ever do that.

These results don’t translate to outer focus points, Servo focus, etc. It simply hints that the manufacturers are doing something that makes phase detection more accurate on still shots using the center points.

There does seem to be a small group of Canon lenses (24mm and 28mm f/2.8 IS, 40mm f/2.8, 300mm f/2.8 IS II) that are more accurate on just the newest bodies. We’ll have to wait for the next new Nikon lens release to see if there is a similar improvement with newer Nikon lenses.

On the other hand, there’s one thing worth considering about the “more accurate” group of new Canon lenses. All are f/2.8 prime lenses. It will be very interesting to see if the the Canon 24-70 f/2.8 II lens, and any future fast prime lenses, are also more accurate. And yes, I was able to get the Imatest shots to check that, it will  take me a while to get the numbers run, though.

I know some of you want to know how the crop sensor Nikons did. I’ll be honest; this is incredibly time consuming work and I just can’t get to more cameras right now. I’m comfortable than none of the crop sensors will be better, and some will probably be worse than the full-frames. That’s what we saw on Canon cameras and I don’t think it will be hugely different for Nikon. But it’s certainly possible that Nikon crop cameras are just as good as Nikon full-frames.

I’ve also had several people ask me to compare contrast detection accuracy between SLRs and mirrorless cameras. That’s kind of a fool’s errand. Contrast detection on SLRs is as accurate as I can measure. We can’t detect more accurate than we can measure.

I do think it would be worthwhile for someone to compare the speed of contrast detection in different systems. While I expect all contrast detection to be as accurate as we could measure, doing this little experiment made me very aware that the speed to lock on using contrast detection is very different with different cameras. The most practical way would probably be to record the AF motor sound and then use that to measure time.

Notice I’m giving this suggestion out for someone to do — because I’m not going to be able to get to it any time soon. If one of you wants to do it, we’d be happy to publish the results. Heck, if one of you wants to come here and do it with our gear, we’d be happy to set you up.

 

Roger Cicala

Lensrentals.com

September 2012

 

 

 

29 Responses to “Autofocus Reality Part 4: Nikon Full Frame”

ginsbu said:

Did you test AF accuracy with any screw-driven lenses? I would imagine that would be of interest to some Nikon shooters.

LensRentals Employee

Roger Cicala said:

Ginabu, a very good point, but we didn’t.

salim madjd said:

Roger thanks for spending your valuable time and sharing your findings. It makes your blog one of my favorite and informative photography blogs that I frequently visit.

Joachim said:

After all that time-consuming testing and as well, writing down the results, I’, just curious: How would you guess the results would change using wide open apertures? Like f/1.8, f/1.4.

Worse or better (because the AF units get more light, but find it tough to hit the center of a shallow depth of field)? A guess from you is, after all this testing, to me as much worth as more tests.

Thank you again.

Danny said:

Another great article, Roger. Thanks for all of your hard work on this. It sounds like you are ready to move onto a different project :) I know you have built up quite a mass of data of Imatest data on variety lenses, so hear me out on this idea: many people (you included) claim that Leica M glass has a special “look,” but I have never seen a convincing article on what causes this phenomenon. I wonder if you could make some graphs and/or run some stats on Leica Imatest data vs. existing Canon/Nikon/Oly etc and see if there is a clear difference in some metrics. Maybe Leica maintains a specific center/edge resolution ratio or has a special contrast falloff from center to edge. Who knows?! But everyone seems to agree there is something special about Leica glass. It would be so great if you could quantify at least one thing that was tangibly different about M lenses.

Jim Thomson said:

Thanks for another interesting article.

dave steinberg said:

Thanks for all the work in this informative test. Did you test the outside focus point accuracy on any of the Phase auto focus systems? I am curious because of the Nikon D800 problems with the left auto focus points. I have had 2 defective cameras.

Jack said:

Thanks for another excellent publication.

Please suggest to Norman to enhance the Imatest, so it can measure and graph the Leica “special look”, Zeiss µcontrast and pick up any holly glass because these questions will persist : -)

Falk Lumo said:

Great article, thanks a lot.

However, there is a minimal glitch in your analysis. You are right that the variation of defocus error does not depend on the AF micro adjustment parameter. However, this does not directly translate into MTF50 figures because they don’t scale exactly proportionally. The details are a bit tricky but you’ll find the math on my site. I studied MTF50 variation as a function of the AF micro adjustment parameter (AFMA) and you may be curious enough to ave a look:

-> http://www.falklumo.com/lumolabs/articles/D800Focus/SensorArray.html Fig.12

(I actually did that to find out how AFMA corrections transate into physical units).

Samuel H said:

Great article, great info. Thanks a lot.

Rory said:

Thanks for doing the Nikons Roger. Very much appreciated.

alek said:

Great job as always Roger – this engineer enjoys to looking at data.

Love the clever solution to determining time to autofocus – “The most practical way would probably be to record the AF motor sound and then use that to measure time.” … also wondering if you might be able to do something with the EXIF since at least in the Canon, they record to the hundredth of a second. But you need accurate measurement of AF start … plus assuming that timestamp is when the picture is actually taken, it includes mirror up, shutter open, etc.
alek

P.S. I think this this sentence needs an extra “0″ – ‘With the D800 the range shown is 700 to 100 LP / IH.’

Jim said:

Hey Roger, quick question for ya
I just got my fast DSLR(a D3200) paired with 18-200mm lens. While PDAF is dead accurate from 80mm and above, it’s very inconsistent at wide angel when focusing on medium to far away object. I confirmed this by reading the distance scale on lens barrel.

I tested on a tripod, center point, single focus, lens set at F3.5 18mm. I tried to focus on a license plate 7 meters away with the lens set at 18mm F3.5. 10 different PDAF focuses set the lens focus distance anywhere from 6m to near infinity(!). While all the images are acceptably sharp, you can definitely tell the difference by 50% magnification. With live view CDAF, it focus the lens to the 7 meter mark reliably every time. I also tried MF using the green dot indicator and surprise to find the tolerance to be quite huge at 18mm.

Is this a normal behavior for PDAF is wide angel? or there is something wrong with my camera/lens?

Jim said:

Hey Roger, quick question for ya
I just got my fast DSLR(a D3200) paired with 18-200mm lens. While PDAF is dead accurate from 80mm and above, it’s very inconsistent at wide angel when focusing on medium to far away object. I confirmed this by reading the distance scale on lens barrel.

I tested on a tripod, center point, single focus, lens set at F3.5 18mm. I tried to focus on a license plate 7 meters away with the lens set at 18mm F3.5. 10 different PDAF focuses set the lens focus distance anywhere from 6m to near infinity(!). While all the images are acceptably sharp, you can definitely tell the difference by 50% magnification. With live view CDAF, it focus the lens to the 7 meter mark reliably every time. I also tried MF using the green dot indicator and surprise to find the tolerance to be quite huge at 18mm.

Is this a normal behavior for PDAF is wide angel? or there is something wrong with my camera/lens?

LensRentals Employee

Roger Cicala said:

Hi Jim,

It sounds like somethings wrong with the distance encoders or AF motor frequency or voltage in the lens. All zooms will vary a bit from one end to the other, but if the variation is really extreme it means the focus group isn’t moving like it’s supposed to. That is most likely an AF motor adjustment, but some mechanical things can do it to.

Roger

yln said:

Some quick questions about the result.
What is the actual deviation? D700 seems to have an outlier. This could askew the data and if far out enough, it is justified to remove from the sample set. Interestingly the newer camera misses does not look like statistical outlier and seems to be just misses. Maybe more data is needed?

LensRentals Employee

Roger Cicala said:

Yin,

As the text goes over, most of the runs have an outlier – that’s the point of the exercise. There is a very real incidence with phase detection of missed focus. Other people who’ve studied this simply count the ‘miss focus’ rate looking at images and it seems to run from 10% to 15% depending on the cameras.

You’re suggesting we throw out the bad focus shots, but that’s what we’re actually looking for.

Roger

yln said:

Hi Roger,

I was thinking somewhere along those lines: http://en.wikipedia.org/wiki/Outlier

Basically if a data point is way way skewed, and have close to 0 chance of happening in the sample size, it is justified to throw out the data point since it will mess up the result. The D700 data set seems to have whole bunch of points clustered at one place which is expected and one really far out. Not sure if that is out-lier or not. : )

LensRentals Employee

Roger Cicala said:

I understand Yin, but it’s not an outlier: around 10% of shots done with phase detection are that way. As I said, if I wanted to do an accurate statistical analysis, the best way I can think of would be 400 or so shots and determine just how many were outliers.

Perhaps I can word it better this way: we’ve taken hundreds of shots both ways. We’ve never had an outlier in contrast detection. We have about 8% outliers in phase detection.

Roger

Jeff Loughlin said:

Thanks for taking the time to do this testing. I found all 4 parts very interesting.

Robert said:

“On the other hand, there’s one thing worth considering about the “more accurate” group of new Canon lenses. All are f/2.8 prime lenses. It will be very interesting to see if the the Canon 24-70 f/2.8 II lens”

No, you forgot about the new 70-300mm, which is not f/2.8! It seems Canon has implement this new approch to every single new lens (I’m pretty sure the new 24-70mm wasn’t left out, and most probably the yet to be officially announced 200-400mm.

What will be intersting is how the new 6d performs (especially since it got a supersensitive -3EV center point). Hope this body is as accurate as the 5d3 and 1dx as it will probably be my first FF. But my hopes are not high as you’ve tested the t4i and is not at the same level as the 5d3 and considering the 6d is a really crippled cameras performance wise…

obican said:

Fun time: Let’s get a split screen viewfinder and put that into a 5dm3. Now mount a 85/1.2L in front, just to keep things challenging and try manually to focus that magnificant piece of glass. I wonder how consistant and accurate the results would be :)

mark lowden said:

Really interesting and as you said you haven’t the time as you have a business to run. Why can’t the camera mags who purportedly need content to differentiate themselves from what is freely available on the web do this.
I for one would buy those magazines. I currently rarely bother to buy camera magazines as the content is mainly derivative and subjective. Thanks for a most interesting article.

LensRentals Employee

Roger Cicala said:

Mark,

I don’t know if the mags think this kind of stuff has a wide enough audience. I’m also cynical enough to wonder, if I ran a magazine, if I’d be willing to publish something guaranteed to piss off one of my really large advertisers. I’m in a lucky position: I have no advertisers to please, I don’t have to beg the companies for lenses or cameras to review or preview, etc.

Roger

Luke said:

Roger,
I just wanted to thank you for doing the test that I was waiting my whole DSLR life for. Thanks a lot. You should be given a Nobel Prize for this.

mark lowden said:

Roger,

Thanks for your response. I think you are probably right and just about sums up the usefulness of the magazines. None other than of course putting trivia between adverts.

Mark

malchon kao said:

Thanks for all the analysis. I learned lots of what I can understand.

Auto Focus is not a Focus.
The focus of Sensor is on the Vertical Strips.

I used Manuel Focus all the times in last 3 years.
It makes me more concentrate before the shoot pre set the focus range. The result is happy.
By the way, I am very happy about D800 as oil painting. D700 as a water coloring
Thanks again.

Martin said:

Thanks Roger for your great work and time. Another great article. I would only dream about comparison AF precison with lens like 1.2-2.0 for both Nikon and Canon.

Dave Sucsy said:

Thanks Roger. Great stuff. Very helpful. We’re all deeply indebted for all the time and work you put into this.
Apparently, Canon’s newest generation of PD focus is leading Nikon’s. Interesting. I’m sure Nikon will catch up. I shoot both. No bias. Each has various pros and cons.

Leave a Reply