Introducing the Lensrentals Numerical Analysis

It’s been obvious for a long time that people don’t want to read long drawn out blog posts about lenses. They want a numerical score so that they can make quick comparisons. We have pounded the drum for a long time that things like validity, relevance, transparency and scientific methodology are of critical importance. But The People want a single number, and they want it right damn now! Because reading is hard; even when I include pictures.

First, The Airing of Grievances

As you know, I’ve been critical of single-number evaluations. For one thing, what does that number mean? 76.23 of what? People who liked it? Beers you drank while testing it? Then there’s the whole ‘proprietary methods’ thing. I mean I wouldn’t want to go to a surgeon who was going to tell me “I do this operation differently from everyone else, but I can’t tell you how, exactly, because it’s proprietary. Besides, you wouldn’t understand anyway.” Even if the operation rates a 76.23 on their Surgery Scoring Scale, I’m not going to participate without some further research.

And how about that weighted thing? OK, so you measured 22 things and then put them in a weighted formula; meaning you picked the ones YOU thought were most important. Did number 3 weigh more than all the other numbers combined? And don’t even get me started on the worst of the worst; the pseudoscientific weighted number. So you measured five things in your lab, and then rated another eight things that were your impression from 1-10. And then you weighted some of these measurements and some of those impressions and came up with a number. Impression, suppression, whatever. You threw a dart at a board or spun a big wheel, didn’t you?

But you have a number and people want that. One. Number. To describe everything. So obviously if we want to be relevant, we have to do a single number summary of our testing.

So Let’s Do Science!!!

We sat down and made a list of things that would be needed to make a really valid, single-number, lens evaluation.

1. Reproducible measurement methods. ‘We rated it subjectively from 1-10’ is neither. It’s just assigning a number to ‘Mikey liked it.’
2. Transparency of all calculations, weights, and formula used. If we don’t give you our calculations, how can you say they’re worthless when your favorite lens measures poorly?
3. A complete evaluation of the lens, with stuff that’s useful in the real world.
4. Testing of multiple samples. Because of variation.

Then we sat down and looked at our capabilities. We have a lot of equipment and findings here that we don’t use in our everyday blog evaluations. Combining that into a single number made sense, since nobody would want to look at that stuff. We can test MTF of course because that’s what we do. We can convert the MTF numbers into something that shows a real-world impact on image resolution, like Popular Photography’s SQF.

We can also test for lateral and longitudinal color, distortion, field-of-focus curvature, and focus shift.

 

Without too much difficulty we can do vignetting, light transmission, and check for true apochromatism. Because we use pinhole collimated light for some test, it’s relatively easy to defocus the pinhole and evaluate how the out-of-focus highlights are rendered and present that mathematically. How spherical, smooth, and colored the OOF highlights can be measured for both foreground and background bokeh.

 

After looking at our capabilities for a bit, the obvious occurred to us: our machines measure everything in metric units. The light wavelengths are in nanometers, focusing distances in fractions of a millimeter, tilts in microns, etc. So can we interconvert them all? So instead of just saying our rating is 76.23 for this lens, we can say it’s 76.23 NANOMETERS. What’s better than a number? Well, duh, a number with Units of Measurement. Obviously, we were well on the way to creating a meaningful number!!!!!

Weighting the Measurements

Well, we talked about this for days and days. What’s the most important? Is resolution three times as important as bokeh? Or 2.7 times? Does one bokeh unit equal one degree of distortion? Or 2? How about field-of-focus curvature? Does the weight of the lens get included? If so, then how do we do that one? Is zero weight ideal? But then we don’t have a lens. Maybe the heaviest weight is best because people will say it’s built like a tank. Finally, we simply decided to go socialist and count all of our measurements equally. Why should f/2.8 get more emphasis than f/16? Are we not all brothers-in-aperture?

And who are we to arrogantly tell you what is important in your photography? Because we don’t know what’s important to you (and well, because it makes the math easier) we threw out weighing altogether. If it’s worth measuring, it counts the same.

Presenting the Formulae

This was an argument in and of itself. Because if you have more than one formula, you can call it ‘formulas’ or ‘formulae’ or ‘more than one formula’ or, where I live, ‘bunch o’ cypherin’ and be correct. We went with formulae cause we thought it sounded more professional and all. But we want to be transparent so that you can check our figures at home.

MTF, of course, is the simplest. While I’m sure you all know this by heart, I’ll repeat here for completeness that we use the standard MTF formula we all learned in 3rd grade (upper left) with consideration, of course, of correction by Fourier transformation, impulse response function, and confirmation with Strehl’s ratio as described by Optikos and others.

 

Bokeh, sadly, is not considered mathematically by many photographers. We use the formula so nicely described by Marcus Winter for our calculations, and recommend you keep this handy; it makes bokeh calculation in the field a breeze.

 

Other formulae and measurements are presented in the appendix in order to keep the length of the article reasonable.

Let’s See This in Practice!!!

We selected a well-known lens, the Canon 35mm f/1.4 L Mk II, to be our first lens tested. As is our practice, ten copies were tested, and the results averaged for each of the following tests, all done at f/1.4, f/2.0, f/2.8, f/4, f5.6, and f/8.

• 4-axis MTF
• Lateral Chromatic Aberration
• Longitudinal Chromatic Aberration
• Distortion
• Field-of-Focus curvature
• AF speed (near-far and far-near)
• Bokeh (spherosity, smoothness, size)
• Focus shift

We did the measurements at six apertures. After each was properly calculated out, we converted numbers and weighing everything equally derived the Lensrentals Numerical Lens Score. If you’d like to look at the individual test results you can see them here, but be patient; it’s a bit of a long download.

We are pleased to announce that the Canon 35mm f/1.4 is our highest rated lens ever tested under this methodology, with a score of 42.42 Nanometers!!!! Canon has really outdone themselves here. No other lens in our Lensrentals Lens Score Database has ever achieved such a high score, and we are comfortable with saying it is the Best Lens We’ve Ever Tested and has achieved the Highest Lensrentals Test Score Ever.

You should rush out and buy the lens today before they see this review and raise the price. Please use the link below because all this testing costs money and I depend on your using the affiliate link to put gas in my Tesla. Plus we’ve shopped around and used our resources to find the Best Price on the Canon 35mm f/1.4 L.

So What’s Next?

Well, this kind of testing takes about a zillion man hours. So you’ll probably see results for the next Lens Rentals Numerical Lens Score in about a year.

 

Roger Cicala and Aaron ‘we be rollin’ Closz

Lensrentals.com

April 1, 2017