Comments on: Why Aren’t the Damn Numbers the Same????

By: Max

Max — Wed, 02 Jan 2013 03:16:31 +0000

Roger, great information as always. I can understand the variations you write about in Imatest testing and results but can you please explain how the same lens and camera tested by Photography Life scores 3000+ in Imatest while you score it at 1000 ? If there lenses 3x better out there – then I MUST have them. Max

By: Roger Cicala

Roger Cicala — Sat, 13 Oct 2012 20:29:47 +0000

Gert,

I’ve been wondering when someone would ask. These numbers are for a Phase 180 digital back medium format camera — 80 megapixel CCD sensor.

Roger

By: Gert Weber

Gert Weber — Sat, 13 Oct 2012 10:41:27 +0000

Hi Roger,

This is a very interesting article!

For me it left only one question:
The original Imatest-chart you are showing shows roughly a double resolution for a very good lens/camera-combination or four times as much as a normal zoomlens. (2700 lp/ph in the Chart, about 1200 lp/ph for a good prime, 500 – 850 in the comparision here)

Why are the numbers so different?

With kind regards

Gert

By: Nikola Runev

Nikola Runev — Sun, 07 Oct 2012 01:15:11 +0000

Great article, thanks for taking the time and trouble to write it.
The way I judge a lens is by trying it on my camera, I don’t do all that pixel peeping stuff. I go out with the lens and make a day out of it, later when I have a look at the RAWs I make my decision.
I’ve been lucky to have good copies of Sigma lenses ( 24-70/2.8 300/2.8 150/2.8 12-24 F/whatever 10-20mm ) but I always tested them with my eyes.
When I was in Japan buying the 24-70 I made the chaps at Yodobashi Cameras open every single camera they had then argued for about 2 hours for the price got a great deal
Always compared the shots with my mates.All that matters are prints !
Most of the lens issues can be eliminated in post. If you’re not a pro that prints larger than A0 , the difference is hard to tell from the viewing distance.

By: Bob

Bob — Wed, 03 Oct 2012 13:48:41 +0000

Great article. What I take from this is you don’t know how a lens will perform for you until you mount it on the camera you will be using it on and take some images at your typical apertures and distances. The only challenge with this approach is that over the years you use different digital cameras and so it is hard to obtain a benchmark as a lens may appear softer than another, but it really isn”t, it just appears that way because your current camera creates softer images as processed than your previous one. Maybe the solution is to buy a high resolution camera and keep it as long as works for comparative lens testing. This may seem extreme, but I have spent tons of time and mental distress trying to figure out why a new lens doesn’t look as sharp as I expect, and this approach would hopefully eliminate the camera body as a possible reason

Bob

By: Randy

Randy — Tue, 02 Oct 2012 17:09:50 +0000

Roger – Very interesting and candid, as usual. Not to beat this to death, but having had some experience on the manufacturer/distributor side, things can be a bit more subtle. If you receive an item prior to its being available to the public, you can be sure it has been thoroughly tested to make sure the manufacturer is putting their best foot forward. “Pre Release” implies unfinished but who really believes a manufacturer would subject their product to a test, knowing it isn’t ready.

The other scenario is that a lens tests so poorly that the tester is uncomfortable publishing the results and requests another sample. If that sample doesn’t check out they decide not to publish the review. Of course you’re in a different position because LR can simply choose not to carry the lens.

By: Jim Thomson

Jim Thomson — Mon, 01 Oct 2012 22:01:32 +0000

Nicholas
Why do you want to shut down all of the photography forums?
Many may read, but few will understand and be allowed to continue posting.

By: Nicholas Condon

Nicholas Condon — Mon, 01 Oct 2012 00:04:53 +0000

“I grew up in biological and medical research and took it for granted everyone expected different tests to have slightly different results.”

Heh. As a physical scientist, I’m always surprised to see replicates of bio-med experiments that are only slightly different…

Snark aside, this is a great, great article. Everyone on the internet on any comment thread about any lens should be forced to pass a test proving that they have read and understoond this information before being allowed to post.

By: Dieter

Dieter — Sun, 30 Sep 2012 23:12:22 +0000

Again, a big thank you for answering my questions. I had aimed at “would you notice it in real world photographs” – and if I understand correctly then one could if one would look very very carefully. That’s what matters in the end result (the image). Thus owning any one of the 70-200 lenses used in the above test, for example, would not lead to noticeable differences in real world photographs unless I scrutinize the images very carefully (i.e. pixel peep).

Of course, the “would I notice it on some at-home optical testing” is also quite interesting – and you answered that question as well. Still surprised at the magnitude of the deviations but knowing those numbers now helps putting things in perspective.

I was aware of the variability of the measured value with distance and focal length and the also the dependency on the actual camera used. It seems to me that my approach of running every new lens through some quick “real life” tests to check for obvious flaws is quite sufficient. To exaggerate a little: if I can’t see it in an image, then I don’t really care if it could be detected on the optical bench.

By: Roger Cicala

Roger Cicala — Sun, 30 Sep 2012 22:28:06 +0000

Dieter,

We may be comparing apples and oranges. I was trying to compare “would you notice it in real world photographs” and I think perhaps you were talking “would I notice it on some at-home optical testing”. Correct me if I’m wrong, I don’t want to assume.

I think the +/- is about 7.5 – 8%.

Now a couple of factors:
1) If I set up a camera and lens and autofocus on the chart taking repeated shots, nothing else difference, the deviation will be about =/- 2%.
2) If I take the highest testing lens and put it on a different camera, it will generally drop down towards the middle quite a bit. If I take the worst and put it on a different camera it will be quite a bit better. This isn’t an AF variation, it’s with careful bracketed manual focus, taking single best shot. Usually a good 3-4% change for extreme cases like we’re discussing.
3) If I do give you the very best camera lens and very worst camera lens combination you would see the same difference I see – if you were shooting at that focal length and shooting distance. But if the results are for 70mm on a 70-200, there will be no correlation between doing well at 70 and doing well at 200. So the best lens at 70 will not be the best lens at 200.
4) A similar variation exists in focusing distance: less obvious but if I run this batch on a small chart at 9 feet and a big chart at 15 feet, the lenses won’t finish in exactly the same order.

As confirmation, on at least the lenses I know very well I can optically adjust them to be superb at one end of the zoom range, but this adjustment often weakens the other end. Not all adjustments do this (centering improves both, for example).

So I guess it’s a two part answer: If you shoot the same focal length and distance as my test, with the same lens and camera, you’ll notice the difference no question. If I sent you just the lenses on your camera the difference might be much smaller. If you change focal lengths or shooting distance, the difference may be gone, or maybe reversed. The sharper lens at 70mm and 15 feet may be the softer lens at 200mm and infinity.

Roger