LensRentals.com

Introducing the Optical Bench

Posted by

Note: This is a Geek article. If you aren't into geeky stuff, you won't be into this.

"We take a step back so that we may leap further." African proverb

I wrote a post a couple of weeks ago about our first copies of the Sigma 50mm Art lens, and promised to follow up when we got more samples. Unfortunately, for the first time in history, Tyler just can't get any copies, so I've been unable to look at sample variation any further.

It's always been my practice to not write anything when I don't have anything to write about -- somewhat unusual for a blogger, I know. But it's been so long since I posted that several people have asked if I was OK. So I thought I'd show you why we've been so busy, and why we should start posting some very cool things quite soon.

More Equipment

Right after we moved into new space, we took delivery of OLAF and started improving our lens adjustment and assessment techniques. We spent most of April learning what OLAF could show us and how we could use it.

Our Imagemaster optical bench was delivered from Trioptics the first week in May, arriving in a crate that was larger than some hotel rooms I've stayed in. Unpacking it was fun - although we had to wait for Byron, the Trioptics engineer, to arrive before they'd let us open it.

Byron wouldn't let us touch much stuff.

Even warning labels seem much more serious when they're in German. I thought this meant 'have a beer before setting up this equipment', but apparently it's just 'Handle with Care'.

Set next to OLAF, I thought our new toy testing equipment looked very nice and I was ready to play with it.

The equipment in our optical testing room now is worth slightly more than my house. I'd appreciate it if nobody tells my wife, OK?

But assembly took a lot longer than I expected. Judging from his expression, I think it took longer than Byron expected, too.

Once it was finally together, I thought we'd get to play with it. But there was lots of calibration and recalibration left to do.

Followed by two full days of rather intense training complete with a 200-page manual to study. I was afraid there'd be a test and we wouldn't get to keep the machine if we failed.

Aaron (on the left) takes notes during training. I rely on sheer memory power -- and access to Aaron's notes.

And I've always wanted to have a machine with an emergency stop button. I can't imagine what we would need to emergently stop testing a lens for, but we're absolutely ready if it happens.

Of course after taking 3 days for that stuff, we were so backed up in repair that we had to spend a couple of more days catching up. So it was last week before we really got to start playing with using our new toy equipment.

So What Does This One Do?

First and foremost, of course, it measures the MTF of the lens. As opposed to Imatest, it measures with the lens focused to infinity (depending on focal length, Imatest measures at distances from 5 to 30 feet). Having both capabilities is important because many lenses perform differently at infinity than they do close up.

It also measures just the lens, where Imatest and DxO measure a lens-camera combination. That's important because it lets us separate out many variables, especially when so many cameras are now processing even raw files in-camera. And it does it quickly, as you can see in the video below.

MTF testing with the TRIOPTICS ImageMaster from LensRentals.com on Vimeo.

It does a lot of other very nice things, too, like measure field-curvature, distortion, actual focal length, lateral and spherical chromatic aberrations, and shoe size. The bottom line is we now have access to more optical testing equipment than - well, anyone in the U. S. except a few government installations and some Universities.

We'll be using it for lens tests and reviews going forward, but before we start doing that I have to assemble a comparative database: right now I have data for thousands of lens copies using Imatest, so I know things like expected sample variation, acceptable ranges, etc. Until we get some similar data using the Imagemaster I won't be comfortable making generalizations. But in the meantime I thought I'd show you some examples.

Some Optical Bench Examples

MTF Across the Field

Remember that in some ways optical bench results are very different from Imatest results, as I blogged about earlier and more like the MTF maps that manufacturers give for their lenses. With the major difference that these are measurements of real lenses, not computer-generated ideal-o-graphs.

Below is an example for Imagemaster results for a very good lens, the Zeiss 21mm f/2.8. Notice the graph shows 10, 20, 30, and 40 line pairs / mm. (Since we've previously reported MTF50 in line pairs / image height, we can multiply by 24mm to get the numbers per image height, so if you'd prefer this is the graph for 240, 480, 720, and 960 lp/ih.) The solid lines are sagittal resolution (lines going from the center of the image toward the edge); the dotted lines are tangential (lines at right angles to sagittal lines). The difference between the two is astigmatism.

Probably the first thing you notice is this lens has a bit more astigmatism on the far right side. This is just minor sample variation, every lens is slightly different. For comparison the image below shows another copy of the Zeiss 21mm that was decentered enough to be noticeable when carefully shooting a test chart. Not only is the astigmatism much more severe, but the absolute MTF numbers are lower (notice the 20 lp/mm lines are around 0.9, or 90% in the good lens, while they're largely below 0.8 in the bad lens).

Frequency Graphs

Frequency graphs show us how a single point on the lens (usually the center) resolves as the lines get smaller and smaller. Here is a frequency graph comparing the centers of the two lenses in the graphs above. The better lens is the red curve and again, solid lines are sagittal and dotted lines tangential. You don't have to understand what this is to realize it's a tool that does a nice job of comparing lenses.

Distortion

It doesn't take an optical bench to show distortion, but it's nice to be able to check it at infinity. The Canon 50mm f/1.2, for example, shows the same amount of distortion (1.45%) at infinity as it does close up (the dotted line here shows a normalized curve, the solid line actual plot points).

 

It's also nice to be able to test multiple copies, because there is a bit of copy-to-copy variation with distortion, too (note the scale of the graph has changed here). The graph below shows 3 copies of the Sigma 50mm f/1.4 lens.

Field Curvatures

One of the coolest, and perhaps almost overwhelming, features is the ability to plot out field curvatures. It's the only really time consuming test, taking 5 to 6 minutes while the others are done in a minute or less, but it's well worth the wait.

Obviously, knowing the field curvature of a lens is helpful when framing a shot, but it's helpful when we're adjusting a lens optically, too, since a field tilt often requires different adjustments than other causes of 'soft on one side'.

Astigmatism

Another test result that's interesting in comparing different lenses, but also very useful when we're optically adjusting lenses is an astigmatism plot.  The first two examples below just show different lenses, one in which astigmatism occurs even slightly off-center, the second in which it occurs only near the edges, and a third in which it is abnormal because of decentering.

Actual Focal Lengths

The Imagemaster contains a grid that lets us measure the exact focal length without a lot of calculations. It isn't particularly important to note that the Sigma 50mm f/1.4 Art lens has an actual focal length of 49.5mm, while the Zeiss 50mm f/1.4 is 51.5mm and the Canon 50mm f/1.2 is 51.8mm. But I found it amusing when I saw the Zeiss 55mm Otus is actually 54mm, because I'd just read a comment that someone found the Canon 50mm focal length much more usable than the 55mm of the Otus. Really, 52mm is much more usable than 54mm?

It does become slightly interesting with zooms. For example, the 24-70 f/2.8 zooms are between 66 and 67.5mm at the long end, while all 70-200 f/2.8s we've tested start at 74 to 76mm.

So Why Aren't You Posting Some Tests, Roger?

Time, of course, is the major factor. We've spent most of the last two weeks just learning the equipment; testing lenses we already know well to make sure our results are accurate and reproducible. Our second priority (which justifies spending all of this money) was getting the optical bench inserted in our testing protocols and learning how to use it to help us with optical adjustments.

We will be starting to do some testing and comparisons, soon, but before that I've got to figure out the best ways to present the data from multiple copies -- the major contribution I make when evaluating lenses is showing sample variation and the range of expected results. The sheer mass of data generated by the Imagemaster makes that challenging. An Excel spreadsheet with the results of one MTF run has about 500 cells of data. Multiply that by two to three runs per copy of 25 copies and pretty soon it's a lot of numbers.

For example, below are composites made from MTF runs of 5 copies of the Zeiss Otus 55mm and 5 of the Sigma 50mm f/1.4 Art, each run at two angles.

Showing sample variation in so crowded a graph is going to be difficult. Standard deviation bars are probably the best solution, although they'll get crowded and aren't very elegant. But I'm open to other suggestions as I try to figure this out, so if you've got some great ideas please let me know.

I know many of you have 632 awesome ideas for things we should test tomorrow, but please be patient a bit longer. We still don't have all of the various lens mounts machined and calibrated yet, so it will be a while before we can test those Sony and micro 4/3 lenses without a camera modifying the images. Canon and Leica comparisons should be coming out fairly soon, and hopefully we'll use those first posts to work out the best ways to present all this data to you.

Roger Cicala and Aaron Closz

Lensrentals.com

May, 2014

59 Responses to “Introducing the Optical Bench”

Roger Moore said:

If you have a lot of curves for a given lens, you might just try superimposing all of them. The areas of overlap will add, giving greater density. You'll wind up with a wide line that's stronger around the average and weaker at the extremes. You could probably do this by having each graph as a layer in Photoshop and using an appropriate method of merging them.

LensRentals Employee

Roger Cicala said:

Lars, working out a comparison number is one thing I'm working on right now. The simplest thing will be, I think, to generate MTF50 numbers on the bench from the frequency graph, then comparing those to the Imatest numbers to get some baselines. Of course the bench will offer more information than Imatest so we'll have to extrapolate a bit.

The good news is that in another 2 months we'll also be able to test finite conjugates (shorter distances) on the optical bench so we'll be able to have true apples to apples comparisons there.

LensRentals Employee

Roger Cicala said:

Tony, thank you. And that's exactly what we plan to do. We've already started it for some prime lenses.

Tony said:

Roger, take a look at "Processing" language for presenting the mountains of data you'll soon be collecting. It's extremely flexible and pretty darn capable. You can create charts with the plotted points having customized colors, shapes and sizes, pulled directly from a .csv file, with something like 20 lines of code. One option that came to mind was to use ellipses to present the astigmatism results. The best part could be the price (exactly $0).

another thought is that your friends over at LensAuthority could use the new tool to generate pedigree papers for the lenses they sell. Or at least someone might more fully explain the meaning of the lpm numbers that are sometimes shown.

Lars Jansen said:

"Followed by two full days of rather intense training complete with a 200-page manual to study. I was afraid there’d be a test and we wouldn’t get to keep the machine if we failed."

Yep, sounds like the Germans. I would not have been surprised if they actually did have a test to ensure you can be allowed to use their machine.

You mention that you plan to use Imatest for short(er) distance measuring and LARS (like the name, Dave!) for infinity measurements. Will these numbers be fully comparable? It would be interesting to test lenses at several different distances, independent of their angle of view.

Peter K said:

Roger
Looking at your fourth picture I probably understand how you arrived at this interpretation

LensRentals Employee

Roger Cicala said:

Peter, I love the "No Throwing" warning. I think at one point the engineer got frustrated enough that it probably was pertinent.

Peter K said:

In fairnes to German Let me translate the Warning signs more or less verbally:

"Bruchgefahr!" = Breaking Danger! (you know, they tend to write consecutive words in a heap...)

"Nicht werfen!" = No throwing! (considering the size of the thing this warning is probably unnecessary)

"Vor Nässe schützen!" = Protect from Humidity! (i.e. keep it dry - makes sense)

Dave M said:

I'm still looking for a name to go with OLAF.

OLGA. Optical Lens Grading Assistant.

OLE. Optical Lens Enhancer.

THOR. Tangential Harmonic Optical Resource.

LARS. Lens Astigmatism Resolving System.

LEIF. Lens Enhancing Image Focuser.

A Ronald Gallant said:

Roger,

How about lightly shading each graph with ± 1 std shading the same color as the graph. The overlap of Tan and Sag shading would show up as slightly darker shading, which would be interpretable to the eye.

Ron

Christoph Breitkopf said:

@Nquina certainly not "impossibly expensive", if you come up with a lens design that enough people will buy. E.g. Brian Caldwell's CoastalOptics 4/60 UV/IR lens. And Brian apparently came up with the idea for OLAF, so there's already a lens designer available (sort of...)

But the hard question is: what lens do you miss really from the big manufacturers? And would enough people be willing to pay for it? Given the success of the Zeiss Otus, maybe quite a lot, but I'm still sceptical.

For example, I'd like a pure macro lens, about 100mm focal length, going from about 1:3 to 2:1 magnification (i.e. no infinity focus). I imagine that'd be much too niche to be commercially viable.

Freeze said:

Roger, for the data-plotting, perhaps you can give the 95% interval? As in: calculate the interval for each measured point, and plot a line through the upper and lower points. This would increase cluttering of the graph, but allows overlapping ranges for different lenses (which would be hard to see if the interval was solid color). Interpretation is quite easy, lines close together = little variance, lines far apart = large variance. Also, it allows people to pick either the upper or the lower line to make their point ;)

Mihail said:

Hello Roger,

From your descriptions I gather that the machine spits out data in xls files. Which is nice, but I don't think that Excel is very convenient for handling large data sets. It might be more useful to load the raw data in some kind of database and crunch the numbers in there. Relational databases are really good for handling large amounts of tabular data, after all this is what they are designed for :) . After that the graphs could be visualized more easily with the proper tool.

I'm an oracle dba, so my firs suggestion would be of course Oracle Database Express for the database and Oracle APEX for the visualization layer (both free) :) I'm not really sure if APEX will be able to handle the more advanced graphs ( field curvature for example, which is actually similar to a heat map ), but some other tool might.

Actually, my real suggestion is what Globules said - collect some more comprehensive data, make it available and start an open-source project for building a tool for visualizing and comparing the data. Again, I think that it should be something database-driven. 50 spreadsheets times thousands of lens copies almost amounts to a small data warehouse. And when you can test a lens in 10 minutes and have historical data, there are no limits for creating interesting graphs and reports.

I am sure that you can easily find a team of photo geeks with the proper experience and fund the whole project with spare lens elements :) (make a contest, overseas shipping would be nice). The end result should be that you will enter a lens model and serial number, push a button and watch all the graphs go by.

Of course, the bench's own software might already have similar functionality, in which case I'm just rambling :)

Regards, and thanks for the great articles,
Mihail.

isaac said:

Roger, Thank you for being so intrigued (and thorough) as I only dream of being. Please know that we watch you posts fully ad really listen to what you have to say. Thanks for taking the effort to move to the next step. ib

Nqina Dlamini said:

^I was think of a similar thing, minus the "frankenlens" part. More along the line that Roger&Co, have so much know how and equipment, they might be tempted to do the next step.....Design their own lens. Wishful thinking and probably impossibly expensive.

Great article as usual.
Thank You

Joe said:

Hey Roger, with all these tools, when are you going to start building frankenlenses? I mean, between all the 'parts' left over from repairs, there has to be a LR, fully manual lens in the future right? It may be a fixed focal length, and manual focus, but optical perfection is within your grasp now!

Dave M said:

Since you already have OLAF, your new toy should be named TROLL. TRioptics Optical Lens Luminator.

Jon said:

Hi Roger

I have read nearly all of your lens test articles and found them very insightful.

You mention center point PDAF is just as accurate as LVAF in several of Canon's newer lenses (24/28 2.8 IS, 40 2.8, 70-300L).

Would it be possible to check the 16-35 2.8 II, 24-70 4.0 IS and 70-200 4.0 IS? I realise f/2.8 lenses have greater AF accuracy than f/4 lenses when used with certain AF sensors so would be interesting to see how these f/4 lenses fare.

LensRentals Employee

Roger Cicala said:

Globules, that might work out. But the raw data would be a 30 or 40 page Excel spreadsheet. Still, I get some macro wizard could figure something out. Good idea.

Globules said:

> but before that I’ve got to figure out the best ways to present the data
> from multiple copies

Once you have some raw data you could make it available, then hold a contest
to see who can come up with the "best" visualization.

LensRentals Employee

Roger Cicala said:

Russell, probably not. But I do reserve judgement a bit. If we see, for example, that certain lenses look great not on a camera, but then fade with certain types of sensors, we might start learning how to make such predictions. ALthough certainly not for a while, and certainly not certainly.

Roger

brandon said:

Thanks.

RussellInCincinnati said:

'twould be so fine, if Roger could cogitate for us and figure out what, if any, optical bench info could foreshadow which lenses will have more trouble with darkening/color shifting corners, on some not-ideal sensors. Uh, for example, would lenses with more astigmatism tend to have worse corners on a Nex 7? Maybe another way to ask this is, can the optical bench ever predict which lenses will "be the most tele-centric" with annoying sensors like the Nex 7?

This is not imply that the optical bench isn't wonderful.

Josh said:

Always an interesting read Roger. Can't wait to see what can be learned about various lenses in the coming months!

Siegfried said:

Roger,
good luck with your new baby!

As for the graphs - I can't think of anything better than graphs with variable thickness, though I know no standard graph libraries which have this feature implemented. If I were you and googling came up with nothing, I'd design my own (there are some benefits of having some s/w design experience in the background).

Cheers,
Zig

P.S.
Just curious: what do Mordor warning signs look like to Elves? I mean like this one: http://www.domznak.ru/c8/sosulki.jpg
Or this one: http://govp.info/sites/default/files/musor07061310.jpg

James Scholz said:

Thanks for the vision and investment to push the limits of solid technical information. As an architectural photographer things like curvature of field and distortion and infinity results are often as important to me as lens resolution. You are enabling us to look at more sides of lens performance then just one dimension, and that is a welcome amount of information.

Thanks again for pushing the envelope.

Lee Saxon said:

Fascinating stuff! Can't wait to see what you learn!

LensRentals Employee

Roger Cicala said:

Don, those were temporary to move it where we wanted, of course. The actual mount is a 1" rubber vibration dampener. The machine self-levels within it's base, expecting that over time there will be some changes. It does need to be on a concrete slab (which is what's under our carpet) - the only problems they've had are when people use it on upper floors of frame structures. It even does a vibration time out after each movement before taking the reading, although our floor is solid enough that it's only for a fraction of a second.

Dom said:

It looks like you've drunk some beer, as coasters were left under the machine :)
With equipment that precise I think it is actually bad idea to not to install it on hard (and flat!) surface: when off-level, device's frame will warp, misaligning whatever (everything?) is attached to it.

CarVac said:

Wow. Very data. Much lens.

You have so many dimensions to present now, and only two to output on...

Leave a Reply