A Data Processing Contest

OK, when I introduced our new optical bench, I mentioned that it generates a lot of data. We gathered data for a nice comparison test (I mean who wouldn’t want to know how the Zeiss Otus and Sigma 50mm f/1.4 Art stack up against some nice Leica 50mm lenses). Since then I’ve spent about 20 hours wading through data and trying different ways to present it. I’ll wallow through that article, but I can’t say I’m totally happy with my efforts.

A number of people have left posts that if I’d furnish them some raw data they’d take a shot at graphing and presenting it. Which seems like a wonderful idea, especially after I went cross-eyed over the holiday weekend trying it myself. So this morning Aaron collected a small data set that anyone who wants to play with is welcome to.

Basically we’ve tested 5 copies of the Sigma 35mm f/1.4 Art lens, and 5 copies of the Canon 35mm f/1.4 lens. We’ll limit this to just MTF numbers across the field (from side-to-side). Each lens is tested 3 times, rotating from 0 to 45 and then 90 degrees. (Since the optical bench makes one ‘slice’ across the lens, we need to take several slices to make sort of a 3-dimensional picture.) We’ve considered making a 4th slice at 135 degrees (180 degrees takes us back to zero), but have found after testing a lot of lenses we’ve never seen anything revealed at 135 degrees that the other three cuts don’t show us.

So if you want to participate, you’ll receive 2 Excel spreadsheets, one for the Canon lenses, one for the Sigma. Each spreadsheet has 5 lenses on separate pages, each page looking like this:

Object angle is simply the degrees off axis (“0” being the center of the lens). The other data is Tangential and Sagittal MTF at each position for 10, 20, 30, and 40 line pairs / mm.

The goal is to create a graph that shows the MTF across the field for each type of lens, along with some way of describing variation, and astigmatism. Then we should be able to compare the two types of lenses and detect any significant difference between the Canon and Sigma lenses.

Five copies is not a huge sample size, particularly for demonstrating range, but I wanted to start small and simple. We usually try to test about 10 copies of each lens when doing comparisons. I don’t expect anyone to find a statistical analysis that will actually differentiate scientifically with 5 copies, but I’m open to any attempts.

To make things more interesting, the 5 Canon samples all passed our standard testing (including Imatest) easily. The Sigma samples include one lens that was borderline and another that failed our other tests. My hope is that being able to detect the Sigma outlier will make certain your efforts are sufficiently sensitive to bad lenses. (Had this been an actual test, the one Sigma copy would not have been used.)

Let’s make it a Contest

I know some people will just enjoy playing with the data (btw, you’re free to use it anywhere you like, just mention that it’s our data when you do). But I’m hoping to get something out of this (an exceedingly easy, automated way to present data so that I can write blog articles by just pushing a button). So if someone comes up with a method I’m going to use, there will be a prize.

Actually, since I’m a history buff, let’s model this after the original French Society for the Encouragement of National Industry contest for a new photography lens and offer several levels of prizes.  We may not award all prizes, or one person may win all three prizes.

Platinum Medal Prize: $500

To win the Platinum Prize you must write a program or macro that fulfills my wild fantasy – Aaron spends hours collecting data on dozens of lenses, then I push a computer button and beautiful, logical graphs are generated that are easy to follow, present the data and variation, and offers an easy way to compare different types of lenses. I will be a bit flexible on what constitutes pushing a button, but if it takes me hours, it’s not winning.

Gold Medal Prize: We will do the lens test of your choice (within limits of our mounts for the optical bench and lens availability) and a $100 rental credit

So if you’ve always wanted a comparison between the Canon 18-55 IS kit lens and the Olympus 16-50, or just want your ownt lenses tested, here’s your chance. To win the Gold Medal Prize you develop a fairly simple way to create logical, easy to understand graphs that demonstrate the variation of copies for each type of lens, and offers an easy way to compare different types of lenses.

Bronze Medal Prize: Our undying gratitude and we’ll test 2 of your lenses on our optical bench and provide you printouts. 

To win the Bronze Medal you simply have to demonstrate a graphing method that presents the average (mean) results for multiple copies of a lens in a way that’s easy to follow and understand. If we use your graph methods, you win a Bronze Medal.

Enter Soon, Enter Often

To enter, just email me at Roger at Lensrentals dot com and I’ll send you the spreadsheets. Let’s put a June 15th deadline on submissions. And remember, an actual test run will usually be 10 samples or more of each lens, and for zooms will include test runs at 3 zoom positions, so whatever you think of has to be able to handle a lot more data than what we’re using on this trial run.

 

Roger Cicala and Aaron Closz

Lensrentals.com

May, 2014