As some of you know, I've been testing a large number of Nikon cameras and lenses as part of the Autofocus Reality series. (Nikon full frame results will be up in a day or so.) I noticed an abnormality in one copy of a lens we were testing that some of you might find interesting.
We were using 28mm f/1.8 lenses (that had all previously passed resolution tests on our D3x test bodies) on D800 bodies. Each camera and lens combination was shot 10 times using Live View (contrast detection) autofocus, 10 times using standard phase detection autofocus. On the graph below, each color represents one run with one copy of the 28 f/1.8 mounted to one copy of the D800. Contrast detection are the diamonds, phase the crosses. If you don't understand what any of this is, there's a blurb about the units below, you can read the Autofocus Reality series, or just muddle along.
On the graph below, Runs 1 and 3 show about what we've come to expect. Contrast detection has a very tight pattern. Phase detection has a slightly looser pattern that's not quite as accurate, although still acceptable. Run 2, though, shows slightly lower resolution and very loose patterns for both phase and contrast detection. After some careful further testing we found that the lens used in this run was just a bit decentered.
Why did it show up now, and not before?
Several reasons. First, you can see that center resolution (the horizontal axis) is just as good as the other lenses. The average resolution is lower because the edges and corners are weaker. Unless a shooter was carefully evaluating the corners of his images it wouldn't be noticeable. In fact, even when we knew it was a problem we had to compare it to other copies to tell it was 'not as good'. If you just looked at an image from the lens, you would just think the corners were a bit soft.
Second, the higher resolution of the D800 exposes the weakness a bit more. It would still be a bit weaker lens in the corners of a D3X or D700 image, but less noticeably so. Please don't start a D800 resolution-war argument - we all realize every lens resolves better on a D800 than on a D700, OK? That doesn't mean every lens wrings out all the resolution a D800 has to offer.
The final reason is testing methods. Like all lens testers and reviewers using Imatest or DxO our testing method involves careful manual focus, focus bracketing several shots on either side of that focus point, then taking the best shot. We have never tested "how accurately does the lens autofocus?" If I just checked the best image this lens made, it comes in at the lower end of passing numbers. If we hadn't happened to be testing multiple shots for AF accuracy we wouldn't have noticed this.
So what did we learn today?
We learned a couple of things today. The most important part is that a decentered lens not only doesn't resolve as well as a good lens, it focuses less accurately. So, the best shot possible with the decentered lens has less resolution than the good lenses, but autofocus is less likely to get you the best shot possible with that decentered lens.
It seems logical when I think about it: autofocus, whether phase detection or contrast detection, requires pixels on a sensor evaluating an image. If the lens has some spherical aberration, decreased contrast, or (especially for phase detection) difference in side-to-side light paths, focus accuracy could well be affected.
The second thing is that even some lenses that seemed OK on a D3x (our standard test camera) may have flaws exposed on the higher resolution of a D800. This particular lens passed our standard resolution tests on a D3x, although it was at the lower end of 'pass'.
Finally, we'll be able to use this finding to improve our testing. Currently we do the same thing most lens testers do: Carefully manually focus, bracket on either side of that point, and use the best result for analysis. At least for borderline lenses, we'll start testing repeated autofocus accuracy to hopefully pick up these 'almost, but not quite OK' lenses more effectively.
Every day I hear from one photographer or another that one of their lenses doesn't autofocus accurately on their camera, but their other lenses seem to. It's usually the usual: Poor technique or simply not microfocus adjusting the camera to the lens.
I wonder, though, if sometimes they're experiencing what we've seen here: A lens that seems reasonably sharp but has just enough decentering to interfere with AF. This particular lens had been used several times by experienced photographers and seemed acceptable to them. In the center (horizontal axis on the above graph) it's just as sharp as the other lenses. When evaluating it closely, it's just a bit soft in all 4 corners, and a bit softer on the right side. But if I didn't have other copies to compare it to, I'd simply think "OK, it's a bit soft in the corners, and I probably wasn't lined up perfectly on the test target."
For us, and people like us, this will hopefully lead us to testing for autofocus accuracy in lenses just like we test for resolution. For regular photographers, if one of your lenses seems to miss focus a lot, it may be worth some careful evaluation to make certain there's no tilt or decentering present.
Graph Explanation: These are the Imatest points for 3 runs of Nikon D800 with 28mm f/1.8 lenses, shot at f/2.8. The vertical axis is average resolution at 13 points on the sensor (corners, edges, 4 midpoints, center) while the horizontal axis is resolution at the center point (all in line pairs / image height from unsharpened raw files).