Note: This is a Geek article. If you aren’t into geeky stuff, you won’t be into this.

“We take a step back so that we may leap further.” African proverb

I wrote a post a couple of weeks ago about our first copies of the Sigma 50mm Art lens, and promised to follow up when we got more samples. Unfortunately, for the first time in history, Tyler just can’t get any copies, so I’ve been unable to look at sample variation any further.

It’s always been my practice to not write anything when I don’t have anything to write about — somewhat unusual for a blogger, I know. But it’s been so long since I posted that several people have asked if I was OK. So I thought I’d show you why we’ve been so busy, and why we should start posting some very cool things quite soon.

More Equipment

Right after we moved into new space, we took delivery of OLAF and started improving our lens adjustment and assessment techniques. We spent most of April learning what OLAF could show us and how we could use it.

Our Imagemaster optical bench was delivered from Trioptics the first week in May, arriving in a crate that was larger than some hotel rooms I’ve stayed in. Unpacking it was fun – although we had to wait for Byron, the Trioptics engineer, to arrive before they’d let us open it.

Byron wouldn’t let us touch much stuff.

Even warning labels seem much more serious when they’re in German. I thought this meant ‘have a beer before setting up this equipment’, but apparently it’s just ‘Handle with Care’.

Set next to OLAF, I thought our new toy testing equipment looked very nice and I was ready to play with it.

The equipment in our optical testing room now is worth slightly more than my house. I’d appreciate it if nobody tells my wife, OK?

But assembly took a lot longer than I expected. Judging from his expression, I think it took longer than Byron expected, too.

Once it was finally together, I thought we’d get to play with it. But there was lots of calibration and recalibration left to do.

Followed by two full days of rather intense training complete with a 200-page manual to study. I was afraid there’d be a test and we wouldn’t get to keep the machine if we failed.

Aaron (on the left) takes notes during training. I rely on sheer memory power — and access to Aaron’s notes.

And I’ve always wanted to have a machine with an emergency stop button. I can’t imagine what we would need to emergently stop testing a lens for, but we’re absolutely ready if it happens.

Of course after taking 3 days for that stuff, we were so backed up in repair that we had to spend a couple of more days catching up. So it was last week before we really got to start playing with using our new toy equipment.

So What Does This One Do?

First and foremost, of course, it measures the MTF of the lens. As opposed to Imatest, it measures with the lens focused to infinity (depending on focal length, Imatest measures at distances from 5 to 30 feet). Having both capabilities is important because many lenses perform differently at infinity than they do close up.

It also measures just the lens, where Imatest and DxO measure a lens-camera combination. That’s important because it lets us separate out many variables, especially when so many cameras are now processing even raw files in-camera. And it does it quickly, as you can see in the video below.

MTF testing with the TRIOPTICS ImageMaster from LensRentals.com on Vimeo.

It does a lot of other very nice things, too, like measure field-curvature, distortion, actual focal length, lateral and spherical chromatic aberrations, and shoe size. The bottom line is we now have access to more optical testing equipment than – well, anyone in the U. S. except a few government installations and some Universities.

We’ll be using it for lens tests and reviews going forward, but before we start doing that I have to assemble a comparative database: right now I have data for thousands of lens copies using Imatest, so I know things like expected sample variation, acceptable ranges, etc. Until we get some similar data using the Imagemaster I won’t be comfortable making generalizations. But in the meantime I thought I’d show you some examples.

Some Optical Bench Examples

MTF Across the Field

Remember that in some ways optical bench results are very different from Imatest results, as I blogged about earlier and more like the MTF maps that manufacturers give for their lenses. With the major difference that these are measurements of real lenses, not computer-generated ideal-o-graphs.

Below is an example for Imagemaster results for a very good lens, the Zeiss 21mm f/2.8. Notice the graph shows 10, 20, 30, and 40 line pairs / mm. (Since we’ve previously reported MTF50 in line pairs / image height, we can multiply by 24mm to get the numbers per image height, so if you’d prefer this is the graph for 240, 480, 720, and 960 lp/ih.) The solid lines are sagittal resolution (lines going from the center of the image toward the edge); the dotted lines are tangential (lines at right angles to sagittal lines). The difference between the two is astigmatism.

Probably the first thing you notice is this lens has a bit more astigmatism on the far right side. This is just minor sample variation, every lens is slightly different. For comparison the image below shows another copy of the Zeiss 21mm that was decentered enough to be noticeable when carefully shooting a test chart. Not only is the astigmatism much more severe, but the absolute MTF numbers are lower (notice the 20 lp/mm lines are around 0.9, or 90% in the good lens, while they’re largely below 0.8 in the bad lens).

Frequency Graphs

Frequency graphs show us how a single point on the lens (usually the center) resolves as the lines get smaller and smaller. Here is a frequency graph comparing the centers of the two lenses in the graphs above. The better lens is the red curve and again, solid lines are sagittal and dotted lines tangential. You don’t have to understand what this is to realize it’s a tool that does a nice job of comparing lenses.

Distortion

It doesn’t take an optical bench to show distortion, but it’s nice to be able to check it at infinity. The Canon 50mm f/1.2, for example, shows the same amount of distortion (1.45%) at infinity as it does close up (the dotted line here shows a normalized curve, the solid line actual plot points).

It’s also nice to be able to test multiple copies, because there is a bit of copy-to-copy variation with distortion, too (note the scale of the graph has changed here). The graph below shows 3 copies of the Sigma 50mm f/1.4 lens.

Field Curvatures

One of the coolest, and perhaps almost overwhelming, features is the ability to plot out field curvatures. It’s the only really time consuming test, taking 5 to 6 minutes while the others are done in a minute or less, but it’s well worth the wait.

Obviously, knowing the field curvature of a lens is helpful when framing a shot, but it’s helpful when we’re adjusting a lens optically, too, since a field tilt often requires different adjustments than other causes of ‘soft on one side’.

Astigmatism

Another test result that’s interesting in comparing different lenses, but also very useful when we’re optically adjusting lenses is an astigmatism plot.  The first two examples below just show different lenses, one in which astigmatism occurs even slightly off-center, the second in which it occurs only near the edges, and a third in which it is abnormal because of decentering.

Actual Focal Lengths

The Imagemaster contains a grid that lets us measure the exact focal length without a lot of calculations. It isn’t particularly important to note that the Sigma 50mm f/1.4 Art lens has an actual focal length of 49.5mm, while the Zeiss 50mm f/1.4 is 51.5mm and the Canon 50mm f/1.2 is 51.8mm. But I found it amusing when I saw the Zeiss 55mm Otus is actually 54mm, because I’d just read a comment that someone found the Canon 50mm focal length much more usable than the 55mm of the Otus. Really, 52mm is much more usable than 54mm?

It does become slightly interesting with zooms. For example, the 24-70 f/2.8 zooms are between 66 and 67.5mm at the long end, while all 70-200 f/2.8s we’ve tested start at 74 to 76mm.

So Why Aren’t You Posting Some Tests, Roger?

Time, of course, is the major factor. We’ve spent most of the last two weeks just learning the equipment; testing lenses we already know well to make sure our results are accurate and reproducible. Our second priority (which justifies spending all of this money) was getting the optical bench inserted in our testing protocols and learning how to use it to help us with optical adjustments.

We will be starting to do some testing and comparisons, soon, but before that I’ve got to figure out the best ways to present the data from multiple copies — the major contribution I make when evaluating lenses is showing sample variation and the range of expected results. The sheer mass of data generated by the Imagemaster makes that challenging. An Excel spreadsheet with the results of one MTF run has about 500 cells of data. Multiply that by two to three runs per copy of 25 copies and pretty soon it’s a lot of numbers.

For example, below are composites made from MTF runs of 5 copies of the Zeiss Otus 55mm and 5 of the Sigma 50mm f/1.4 Art, each run at two angles.

Showing sample variation in so crowded a graph is going to be difficult. Standard deviation bars are probably the best solution, although they’ll get crowded and aren’t very elegant. But I’m open to other suggestions as I try to figure this out, so if you’ve got some great ideas please let me know.

I know many of you have 632 awesome ideas for things we should test tomorrow, but please be patient a bit longer. We still don’t have all of the various lens mounts machined and calibrated yet, so it will be a while before we can test those Sony and micro 4/3 lenses without a camera modifying the images. Canon and Leica comparisons should be coming out fairly soon, and hopefully we’ll use those first posts to work out the best ways to present all this data to you.

Roger Cicala and Aaron Closz

Lensrentals.com

May, 2014