Well before our Imagemaster optical bench was delivered, I knew what I wanted to do first: compare some Leica lenses with the best SLR lenses on a level playing. I did one comparison involving Leica lenses a long time ago, but Leica lenses are difficult to test in our Imatest lab. It’s nothing to do with Imatest or the lenses; simply that mounting and aligning rangefinder cameras in a testing setup designed for SLRs is difficult and very time consuming.

That was a couple of weeks ago. Usually I have the write up done within a day or two of testing. But I’ve never processed and graphed this much data before — the optical bench generates a lot more numbers than Imatest. I tried a few things. I hated them. I tried some more. Finally I asked the good Geeky people who tend to read this blog to help out. I’m pleased to say over 40 people are currently working with another set of data, trying to help me find a good way to overcome my mathematical and graphical limitations to handle all of this data. But that’s going to take some time.

In the meantime, I still have a bunch of information about those 50mm lenses I want to tell you about. So I’ve decided to go ahead and post using the fairly crude methods I’ve come up with so far. Consider this a very preliminary effort since I hope the next set of tests we do will have a far better presentation. Please take it for what it is – a beta trial of a new technology we’re learning to work with.

I’m going to use a mixture of graphs and tables today, but please give me some feedback on what seems clumsy, what seems good, and what I’ve left out that you’d like to see.

Meet the 50mm Contestants

The 50mm lenses we tested range in price from under $1,000 to $11,000. I’ve included the measured focal lengths in this table for those who want to say the Otus isn’t a 50mm lens. It isn’t, of course, but neither are any of the others if you want to be picky.

LensPriceFocal Length (mm)
Leica 50mm f/0.95 Noctilux$10,99552.5
Leica 50mm f/1.4 Summilux M ASPH$4,25051.4
Leica 50mm f/2 Summicron M$2,35051.8
Leica 50mm f/2 APO Summicron M ASPH$7,35051.5
Zeiss 50mm f/1.5 Sonnar$1,20050.3
Zeiss 50mm f/2 Planar$86051
Voigtlander 50mm f/1.1 Nokton$99951.6
Sigma 50mm f/1.4 Art$94949.5
Zeiss 55mm f/1.4 Otus$3,99054.2

While I always like to test multiple copies it’s not always possible with some of the more unusal lenses. We only have a few copies each of the Noctilux and APO Summicron and most were out on rental so I only had 1 copy of each of those available. For the other lenses I tested 3 copies.

Test Results

Distortion

The 50mm focal length is unusual in that lenses can have either barrel or pincushion distortion, although they usually don’t have much of either. Positive numbers are pincushion distortion, while negative are barrel.

LensDistortion %
Leica 50mm f/0.95 Noctilux-1.0
Leica 50mm f/1.4 Summilux M ASPH+0.3
Leica 50mm f/2 Summicron M+0.25
Leica 50mm f/2 APO Summicron M ASPH+0.3
Zeiss 50mm f/1.5 Sonnar+0.8
Zeiss 50mm f/2 Planar-0.7
Voigtlander 50mm f/1.1 Nokton-0.7
Sigma 50mm f/1.4 Art+0.2
Zeiss 55mm f/1.4 Otus-0.6

None of these lenses have a great deal of distortion, but the Sigma Art and all of the Leica’s (with the exception of the Noctilux) have the least distortion of the bunch. A few of these numbers are a bit different from what has been reported elsewhere, probably because these are measured at infinity focus rather than 8-12 feet where chart distortion measurements would be taken.

Frequency Graphs and MTF50

I’m sure people would like to compare optical bench tests to the computized chart analysis (Imatest and DxO) they’re used to seeing. It’s not completely possible; the optical bench tests just the lens at infinity, while the chart analysis tests a lens-camera combination at close focusing distances. But the optical bench is fully capable of generating MTF50 numbers so I thought I’d try presenting those.

As an example, here’s a frequency graph showing MTF numbers for a lens at 3 positions: Center (red), halfway between center and edge (green), and at the edge (blue). The solid lines show sagittal MTF, the dotted lines tangential (AKA meridian). If you don’t know the difference, it’s explained here about halfway through the article, but the takeaway message is a large difference between sagittal and tangential lines shows astigmatism.

 

If you notice the black bars I put along the left side of the graph, they show where the MTF30, 50 and 70 would be read. The optical bench reads them in line pairs/mm, but simply multiplying that by 24 (the height of a full-frame sensor in mm) gives results in line pairs/image height, which is what most review sites use.

In order to keep things manageable, we’ll show the center of the lens, mid-way between center and edge, and at the edge. I initially gave both sagittal and tangential numbers but that got confusing, so I’ll list the higher of the two MTF readings, along with the astigmatism (difference between sagittal and tangential MTF) at that point.

Each lens was tested at three rotations (0, 45 and 90 degrees) and the average of the three readings used for that lens. Except for the Noctilux and APO Summicron three copies of each lens were tested.

Remember, Optical bench MTF50 number are much higher than Imatest numbers. The bench is testing just the lens. With Imatest we have additional loss of detail through the Bayer array, AA filter, and sensor cover glass; then we’re analyzing pixelated data with some inevitable electrical noise.

Here are the MTF50 results given in line pairs/image height for the lenses, each tested at widest aperture. (The various shades of orange indicate my impression ranging from ‘that’s good’ to ‘hey look at this!!!!)

The Summicron APO kicks some very serious butt in this test, as it should for $7,000. It’s sharper at the edge than all but one other lens are in the center. The Leica Summilux was exceptional, too, especially when one considers it’s shooting at f/1.4 in this test while several other lenses are at f/2. The Summicron, Sigma f/1.4 Art, and Zeiss f/1.4 Otus also did very, very well.

The Leica Noctilux and Voigtlander Nokton, of course, are wide-aperture specialty lenses and aren’t designed to give the best resolution. In this test they are additionally handicapped by shooting at f/0.95 and f/1.1 respectively.

To even things out a bit, I retested all of the lenses at f/2.0.

The Summicron APO still has the best resolution numbers, although several other lenses narrow the gap when closed down to f/2. The Summilux is just a bit behind the APO in central resolution and actually does just as well at the edges. The Sigma lags the best Leicas just a bit in the edges, but it is superb in the center, nearly catching the Summicron APO.

Take a moment to look at the astigmatism numbers, though. Remember, for this test I gave the highest reading from either the Tangential or Sagittal lines, and the difference between the two lines as astigmatism. The Zeiss f/1.4 Otus, in particular, doesn’t have the highest absolute resolution, but has amazingly little astigmatism. (If this had been Imatest, I would have averaged the Sagittal and Tangential lines and used that in my report, which would have made the Zeiss and APO look better and the Summilux look a bit worse.

The Normal f/2 Summicron is also excellent, especially considering its more reasonable price compared to the other Leica lenses. Then again, when you consider it still costs 2.5 times what the Sigma costs, I guess the price doesn’t seem so reasonable. From an MTF50 standpoint, the Voigtlander does surprisingly well, at least in the center and middle regions of the lens, given it’s low cost and wide aperture.

So How is This Different than What I Would Show You on Imatest?

I tried to make this part as similar to one of my Imatest posts as I could. There are some slight differences: this test is at infinity focusing distance, rather than 10-15 feet where we would test this focal length on Imatest, and this test has fewer data points. I’m showing center, edge and corner data, where for Imatest I would show center, average of the entire lens, and corner data.

The absolute numbers are higher, because we are looking at just the lens, rather than the lens-camera interpretation. But if we somehow shot all these lenses on the same test camera, had a theoretically perfect adapter and sensor microlenses, and a billboard sized Imatest chart that let us shoot at infinity, the lenses should rank the same way. Basically, I’d be done now (and since this article has been hanging on my desktop for weeks, that sounds really good). But the optical bench can show us a lot more than MTF50.

More Optical Bench Results

So far I’ve just used the MTF50 number for all of the tables, which is what I give you when I test with Imatest. With the optical bench, though we get a lot more information from a lens than the MTF50.

The graph below shows the MTF chart from one side of the lens to the other (“0” on the horizontal axis is the center of the lens). The red, green, blue, and purple lines represent 10, 20, 30, and 40 lines per mm; the solid lines are sagittal and the broken lines tangential. Notice most of this information occurs at lower frequencies than the MTF50. So does much of the information in your photograph.

The graph above is a single side-to-side cut of a single lens. In our testing, we’d also rotate it 45 and 90 degrees and repeat the readings, letting us look at the lens in two dimensions rather than just one single line from side-to-side. (In discussing this with others, we probably should also do 135 degrees for completeness. We didn’t in this test because our first work with the optical bench was looking at bad copies, and just testing at “0” and 45 degrees was sufficient for that.)

As we average multiple angles and multiple copies, the graphs become nearly identical on each side of “0”. That is why the camera companies only show one side of this kind of graph when they present their MTF charts for a given lens – the other side is simply a mirror image (assuming the lenses are accurately adjusted). I’ll follow that pattern here.

All of these lenses are tested with the aperture wide-open, so you’ll have to take into consideration that for some pairs one lens is at a slightly different aperture than the other.

You don’t have to really understand MTF charts to know that higher is better, and less separation of the solid and dotted lines is better.

The Ultra-wide Aperture Lenses

First let’s compare the Nokton / Noctilux lenses. As we’ve already seen, they’re designed for widest aperture, not highest MTF readings. The Voigtlander, given it’s low cost, certainly holds up well in an MTF comparison against the Leica Noctilux, although neither would be considered a sharp lens wide open.

The Zeiss M Mounts

Next we’ll compare the two Zeiss M-mount lenses. Even taking into account the difference in aperture, the Planar f/2 looks significantly better than the Sonnar f/1.5. The MTF50 results above suggest stopping down to f/2 doesn’t help the 50mm f/1.5 a great deal so the 50mm f/2, from an MTF standpoint at least, seems the better lens. It’s also the less expensive lens.

The Moderately Priced Leicas

Next, let’s look at the ‘lesser’ Leicas, the Normal 50mm f/2 Summicron and the 50mm f/1.4 Summilux. Again, the Summilux is tested here at f/1.4, so it’s spotting the ‘Cron a fair amount of aperture.

You don’t have to know anything more than ‘higher is better’ and ‘lines of the same color closer together are better’ to appreciate that these lenses are really good. In the center the Zeiss 50mm f/2 Planar is certainly comparable to these more expensive lenses, but off-center it has a lot more astigmatism. When you consider the Summilux is doing this at f/1.4 you see why some people are willing to pay a premium for it. It’s at least as good, and perhaps a bit better, at f/1.4 as the other lenses are at f/2.

The SLR Lenses

Next up, we’ll compare the SLR lenses: Sigma 50mm f/1.4 Art with the Zeiss Otus 55mm f/1.4.

In the central part of the lens (out to about 10 degrees off- axis) the Sigma is a bit better, while the Otus is superior in the outer areas. They’re slightly different, but purely from a resolution standpoint they are pretty close, and both are pretty close to the Leica Summilux. Each lens has areas where they are a bit better, but the difference between them would be one of personal preference or Fanboy hair-splitting.

I think it’s pretty cool that two SLR lenses are able to hang right there with the f/1.4 Leica lens, the Summilux. Price-wise, you would hope the Otus would, but certainly the Sigma, one of the least expensive lenses in the test, has to take honors for hanging with the best at a tiny fraction of their price.

I think this is the important take home point of this type of testing. If we just looked at the MTF50 numbers in the first part of the article the Summilux probably seemed the best of the three. It had higher MTF 50 numbers in the center, middle, and edges of the lens, although it also had higher astigmatism. Looking at the MTF curves with a slightly different test, the case isn’t quite as clear and you could certainly make an argument that you preferred any of the three lenses.

(One point I want to make clear is the testing methods differ – the MTF50 tests are done at ‘best focus for that point’. In other words the bench individually focuses at the center, mid point, and edge to generate the highest possible numbers. With the MTF tests we’re showing here, the bench focuses in the best position for the center of the lens and all other measurements are done at that focus. So field curvature will affect these graphs, but not the MTF50 tests done earlier.)

The APO Summicron

And now for our last lens. The MTF50 results in the first part of the article suggest the Leica APO Summicron f/2 is the best lens of the bunch, so let’s see if the more general MTF testing bears that out. I chose the Otus for a comparison graph, but it could just as easily have been the Sigma 50mm Art or the Summilux.

I can see some arguments here or there, but I do think the Leica APO Summicron looks amazing. I wish I’d done this comparison with the Otus at f/2.0, but I don’t think it would have evened things up entirely. The Summicron is the most expensive lens of the bunch, so you would expect it to be the best – and it is.

I won’t pretend to tell you the APO Summicron is worth $3,000 more than a Summilux or an Otus. It certainly is to some people. It would never be a consideration for others. But as the most expensive lens (other than the Noctilux) it should be the best.

Going Forward

Please take this first optical bench test for what it was: a trial to help work the kinks out as we develop our testing and graphing methods on this equipment. It’s going to be fun to be able to compare lenses from different camera mounts without guessing what effect the cameras have. Testing at infinity, rather than close distances, has some practical usefulness, too.

We’re certainly will have a lot more data from every test than we’ve ever had before. Today I just showed a part of it. I haven’t even begun to consider how we’ll present field curvature data. In a few weeks will have an accessory to the optical bench that allows it to test close distances as well as infinity, giving us even more data.

Hopefully after another trial run or two we’ll have things settled down into a standard format. I’m sure some of the contest entries will help us make this more standardized and automated. But in the meantime, please send me any suggestions you have that might make things easier to understand.

Roger Cicala and Aaron Closz

Lensrentals.com

June, 2014