LensRentals.com

Comparing Rangefinder and SLR 50mm Lenses. (Version 0.7)

Posted by

Well before our Imagemaster optical bench was delivered, I knew what I wanted to do first: compare some Leica lenses with the best SLR lenses on a level playing. I did one comparison involving Leica lenses a long time ago, but Leica lenses are difficult to test in our Imatest lab. It's nothing to do with Imatest or the lenses; simply that mounting and aligning rangefinder cameras in a testing setup designed for SLRs is difficult and very time consuming.

That was a couple of weeks ago. Usually I have the write up done within a day or two of testing. But I've never processed and graphed this much data before -- the optical bench generates a lot more numbers than Imatest. I tried a few things. I hated them. I tried some more. Finally I asked the good Geeky people who tend to read this blog to help out. I'm pleased to say over 40 people are currently working with another set of data, trying to help me find a good way to overcome my mathematical and graphical limitations to handle all of this data. But that's going to take some time.

In the meantime, I still have a bunch of information about those 50mm lenses I want to tell you about. So I've decided to go ahead and post using the fairly crude methods I've come up with so far. Consider this a very preliminary effort since I hope the next set of tests we do will have a far better presentation. Please take it for what it is - a beta trial of a new technology we're learning to work with.

I'm going to use a mixture of graphs and tables today, but please give me some feedback on what seems clumsy, what seems good, and what I've left out that you'd like to see.

Meet the 50mm Contestants

The 50mm lenses we tested range in price from under $1,000 to $11,000. I've included the measured focal lengths in this table for those who want to say the Otus isn't a 50mm lens. It isn't, of course, but neither are any of the others if you want to be picky.

Lens Price Focal Length (mm)
Leica 50mm f/0.95 Noctilux$10,99552.5
Leica 50mm f/1.4 Summilux M ASPH$4,25051.4
Leica 50mm f/2 Summicron M$2,35051.8
Leica 50mm f/2 APO Summicron M ASPH$7,35051.5
Zeiss 50mm f/1.5 Sonnar$1,20050.3
Zeiss 50mm f/2 Planar$86051
Voigtlander 50mm f/1.1 Nokton$99951.6
Sigma 50mm f/1.4 Art$94949.5
Zeiss 55mm f/1.4 Otus$3,99054.2

While I always like to test multiple copies it's not always possible with some of the more unusal lenses. We only have a few copies each of the Noctilux and APO Summicron and most were out on rental so I only had 1 copy of each of those available. For the other lenses I tested 3 copies.

Test Results

Distortion

The 50mm focal length is unusual in that lenses can have either barrel or pincushion distortion, although they usually don't have much of either. Positive numbers are pincushion distortion, while negative are barrel.

Lens Distortion %
Leica 50mm f/0.95 Noctilux-1.0
Leica 50mm f/1.4 Summilux M ASPH+0.3
Leica 50mm f/2 Summicron M+0.25
Leica 50mm f/2 APO Summicron M ASPH+0.3
Zeiss 50mm f/1.5 Sonnar+0.8
Zeiss 50mm f/2 Planar-0.7
Voigtlander 50mm f/1.1 Nokton-0.7
Sigma 50mm f/1.4 Art+0.2
Zeiss 55mm f/1.4 Otus-0.6

None of these lenses have a great deal of distortion, but the Sigma Art and all of the Leica's (with the exception of the Noctilux) have the least distortion of the bunch. A few of these numbers are a bit different from what has been reported elsewhere, probably because these are measured at infinity focus rather than 8-12 feet where chart distortion measurements would be taken.

Frequency Graphs and MTF50

I'm sure people would like to compare optical bench tests to the computized chart analysis (Imatest and DxO) they're used to seeing. It's not completely possible; the optical bench tests just the lens at infinity, while the chart analysis tests a lens-camera combination at close focusing distances. But the optical bench is fully capable of generating MTF50 numbers so I thought I'd try presenting those.

As an example, here's a frequency graph showing MTF numbers for a lens at 3 positions: Center (red), halfway between center and edge (green), and at the edge (blue). The solid lines show sagittal MTF, the dotted lines tangential (AKA meridian). If you don't know the difference, it's explained here about halfway through the article, but the takeaway message is a large difference between sagittal and tangential lines shows astigmatism.

 

If you notice the black bars I put along the left side of the graph, they show where the MTF30, 50 and 70 would be read. The optical bench reads them in line pairs/mm, but simply multiplying that by 24 (the height of a full-frame sensor in mm) gives results in line pairs/image height, which is what most review sites use.

In order to keep things manageable, we'll show the center of the lens, mid-way between center and edge, and at the edge. I initially gave both sagittal and tangential numbers but that got confusing, so I'll list the higher of the two MTF readings, along with the astigmatism (difference between sagittal and tangential MTF) at that point.

Each lens was tested at three rotations (0, 45 and 90 degrees) and the average of the three readings used for that lens. Except for the Noctilux and APO Summicron three copies of each lens were tested.

Remember, Optical bench MTF50 number are much higher than Imatest numbers. The bench is testing just the lens. With Imatest we have additional loss of detail through the Bayer array, AA filter, and sensor cover glass; then we're analyzing pixelated data with some inevitable electrical noise.

Here are the MTF50 results given in line pairs/image height for the lenses, each tested at widest aperture. (The various shades of orange indicate my impression ranging from 'that's good' to 'hey look at this!!!!)

The Summicron APO kicks some very serious butt in this test, as it should for $7,000. It's sharper at the edge than all but one other lens are in the center. The Leica Summilux was exceptional, too, especially when one considers it's shooting at f/1.4 in this test while several other lenses are at f/2. The Summicron, Sigma f/1.4 Art, and Zeiss f/1.4 Otus also did very, very well.

The Leica Noctilux and Voigtlander Nokton, of course, are wide-aperture specialty lenses and aren't designed to give the best resolution. In this test they are additionally handicapped by shooting at f/0.95 and f/1.1 respectively.

To even things out a bit, I retested all of the lenses at f/2.0.

The Summicron APO still has the best resolution numbers, although several other lenses narrow the gap when closed down to f/2. The Summilux is just a bit behind the APO in central resolution and actually does just as well at the edges. The Sigma lags the best Leicas just a bit in the edges, but it is superb in the center, nearly catching the Summicron APO.

Take a moment to look at the astigmatism numbers, though. Remember, for this test I gave the highest reading from either the Tangential or Sagittal lines, and the difference between the two lines as astigmatism. The Zeiss f/1.4 Otus, in particular, doesn't have the highest absolute resolution, but has amazingly little astigmatism. (If this had been Imatest, I would have averaged the Sagittal and Tangential lines and used that in my report, which would have made the Zeiss and APO look better and the Summilux look a bit worse.

The Normal f/2 Summicron is also excellent, especially considering its more reasonable price compared to the other Leica lenses. Then again, when you consider it still costs 2.5 times what the Sigma costs, I guess the price doesn't seem so reasonable. From an MTF50 standpoint, the Voigtlander does surprisingly well, at least in the center and middle regions of the lens, given it's low cost and wide aperture.

So How is This Different than What I Would Show You on Imatest?

I tried to make this part as similar to one of my Imatest posts as I could. There are some slight differences: this test is at infinity focusing distance, rather than 10-15 feet where we would test this focal length on Imatest, and this test has fewer data points. I'm showing center, edge and corner data, where for Imatest I would show center, average of the entire lens, and corner data.

The absolute numbers are higher, because we are looking at just the lens, rather than the lens-camera interpretation. But if we somehow shot all these lenses on the same test camera, had a theoretically perfect adapter and sensor microlenses, and a billboard sized Imatest chart that let us shoot at infinity, the lenses should rank the same way. Basically, I'd be done now (and since this article has been hanging on my desktop for weeks, that sounds really good). But the optical bench can show us a lot more than MTF50.

More Optical Bench Results

So far I've just used the MTF50 number for all of the tables, which is what I give you when I test with Imatest. With the optical bench, though we get a lot more information from a lens than the MTF50.

The graph below shows the MTF chart from one side of the lens to the other ("0" on the horizontal axis is the center of the lens). The red, green, blue, and purple lines represent 10, 20, 30, and 40 lines per mm; the solid lines are sagittal and the broken lines tangential. Notice most of this information occurs at lower frequencies than the MTF50. So does much of the information in your photograph.

The graph above is a single side-to-side cut of a single lens. In our testing, we'd also rotate it 45 and 90 degrees and repeat the readings, letting us look at the lens in two dimensions rather than just one single line from side-to-side. (In discussing this with others, we probably should also do 135 degrees for completeness. We didn't in this test because our first work with the optical bench was looking at bad copies, and just testing at "0" and 45 degrees was sufficient for that.)

As we average multiple angles and multiple copies, the graphs become nearly identical on each side of "0". That is why the camera companies only show one side of this kind of graph when they present their MTF charts for a given lens - the other side is simply a mirror image (assuming the lenses are accurately adjusted). I'll follow that pattern here.

All of these lenses are tested with the aperture wide-open, so you'll have to take into consideration that for some pairs one lens is at a slightly different aperture than the other.

You don't have to really understand MTF charts to know that higher is better, and less separation of the solid and dotted lines is better.

The Ultra-wide Aperture Lenses

First let's compare the Nokton / Noctilux lenses. As we've already seen, they're designed for widest aperture, not highest MTF readings. The Voigtlander, given it's low cost, certainly holds up well in an MTF comparison against the Leica Noctilux, although neither would be considered a sharp lens wide open.

The Zeiss M Mounts

Next we'll compare the two Zeiss M-mount lenses. Even taking into account the difference in aperture, the Planar f/2 looks significantly better than the Sonnar f/1.5. The MTF50 results above suggest stopping down to f/2 doesn't help the 50mm f/1.5 a great deal so the 50mm f/2, from an MTF standpoint at least, seems the better lens. It's also the less expensive lens.

The Moderately Priced Leicas

Next, let's look at the 'lesser' Leicas, the Normal 50mm f/2 Summicron and the 50mm f/1.4 Summilux. Again, the Summilux is tested here at f/1.4, so it's spotting the 'Cron a fair amount of aperture.

You don't have to know anything more than 'higher is better' and 'lines of the same color closer together are better' to appreciate that these lenses are really good. In the center the Zeiss 50mm f/2 Planar is certainly comparable to these more expensive lenses, but off-center it has a lot more astigmatism. When you consider the Summilux is doing this at f/1.4 you see why some people are willing to pay a premium for it. It's at least as good, and perhaps a bit better, at f/1.4 as the other lenses are at f/2.

The SLR Lenses

Next up, we'll compare the SLR lenses: Sigma 50mm f/1.4 Art with the Zeiss Otus 55mm f/1.4.

In the central part of the lens (out to about 10 degrees off- axis) the Sigma is a bit better, while the Otus is superior in the outer areas. They're slightly different, but purely from a resolution standpoint they are pretty close, and both are pretty close to the Leica Summilux. Each lens has areas where they are a bit better, but the difference between them would be one of personal preference or Fanboy hair-splitting.

I think it's pretty cool that two SLR lenses are able to hang right there with the f/1.4 Leica lens, the Summilux. Price-wise, you would hope the Otus would, but certainly the Sigma, one of the least expensive lenses in the test, has to take honors for hanging with the best at a tiny fraction of their price.

I think this is the important take home point of this type of testing. If we just looked at the MTF50 numbers in the first part of the article the Summilux probably seemed the best of the three. It had higher MTF 50 numbers in the center, middle, and edges of the lens, although it also had higher astigmatism. Looking at the MTF curves with a slightly different test, the case isn't quite as clear and you could certainly make an argument that you preferred any of the three lenses.

(One point I want to make clear is the testing methods differ - the MTF50 tests are done at 'best focus for that point'. In other words the bench individually focuses at the center, mid point, and edge to generate the highest possible numbers. With the MTF tests we're showing here, the bench focuses in the best position for the center of the lens and all other measurements are done at that focus. So field curvature will affect these graphs, but not the MTF50 tests done earlier.)

The APO Summicron

And now for our last lens. The MTF50 results in the first part of the article suggest the Leica APO Summicron f/2 is the best lens of the bunch, so let's see if the more general MTF testing bears that out. I chose the Otus for a comparison graph, but it could just as easily have been the Sigma 50mm Art or the Summilux.

I can see some arguments here or there, but I do think the Leica APO Summicron looks amazing. I wish I'd done this comparison with the Otus at f/2.0, but I don't think it would have evened things up entirely. The Summicron is the most expensive lens of the bunch, so you would expect it to be the best - and it is.

I won't pretend to tell you the APO Summicron is worth $3,000 more than a Summilux or an Otus. It certainly is to some people. It would never be a consideration for others. But as the most expensive lens (other than the Noctilux) it should be the best.

Going Forward

Please take this first optical bench test for what it was: a trial to help work the kinks out as we develop our testing and graphing methods on this equipment. It's going to be fun to be able to compare lenses from different camera mounts without guessing what effect the cameras have. Testing at infinity, rather than close distances, has some practical usefulness, too.

We're certainly will have a lot more data from every test than we've ever had before. Today I just showed a part of it. I haven't even begun to consider how we'll present field curvature data. In a few weeks will have an accessory to the optical bench that allows it to test close distances as well as infinity, giving us even more data.

Hopefully after another trial run or two we'll have things settled down into a standard format. I'm sure some of the contest entries will help us make this more standardized and automated. But in the meantime, please send me any suggestions you have that might make things easier to understand.

Roger Cicala and Aaron Closz

Lensrentals.com

June, 2014

63 Responses to “Comparing Rangefinder and SLR 50mm Lenses. (Version 0.7)”

Lee Saxon said:

I have an idea, although this is really just for presentation and not data-wrangling.

Setting the MTF graphs side-by-side like only lets you compare two lenses at once, makes them super skinny, and makes the page run really long.

Maybe you can find a little rollover app like Lloyd Chambers uses on his aperture series tests to let people switch between lenses. But please please please change it to onclick instead of rollover, that part sucks.

Zlatko said:

Some of those numbers are unexpected — could there be some typos? The 50/1.4 Summilux really doesn't improve at all at center or middle when stopping down from 1.4 to 2.0? Jeff M. and Dibyendu have commented about this above. Also, the old 50/2.0 Summicron is perfectly identical (1200) at center, middle and edge?

LensRentals Employee

Roger Cicala said:

Zlatko - it's possible there are typos. I double checked when collecting and again as I entered data, but the article took so long to write as I tried different ways to present the data that I might have transposed something going from one form of presentation to another. I'll have to get the original machine tracings out and double check. But after what we've found with the sensor stack issues, I also need to retest the Sigma and Otus lenses. They might actually be better than they appear from my test results.

Roger

Dan said:

THanks so much for this test. Very impressive! It would be great to see results at f4 and f8 too some day.

hugh crawford said:

Roger,
The Noctilux Summicron show astigmatism of 24 at center. other that an out of alignment lens or a slightly cylindrical element how is that possible?
Also in the tradition of what is the sound of one hand clapping, what would be a Sagittal or Tangential line at the center of the image?

Nicholas said:

The Leica Summilux 50mm F/1.4 exhibits a significant field curvature (or wobbliness) when used on a digital camera — e.g. the Leica M9 or the M Typ 240 — which results in annoyingly distracting loss of sharpness on a 24x36mm imaging area, along a donut-shaped area about 10mm from the picture center.

As the Leica Apo Summicron 50 F/2.0's MTF curves look quite similar to the Summilux's on Leica's data sheets — the same kind of wobbly, tortured MTF curve — it's possible that the actual performance of the Apo Summicron on a digital sensor would be far less impressive than the measurements on an aerial image optical bench would suggest.

To reproduce the image degradation — astigmatism, field curvature etc. — that the cover glass of a Leica digital camera induces with rangefinder lenses with a short exit pupil to imaging plane distance, would it be possible to insert, between the lens and the optical bench's sensor, a glass plate of the same thickness (0.8mm) as the cover glass used on Leica cameras, and check how the measurement values are affected when testing the lens in an optical configuration that more closely approximates the physical layout of a Leica digital camera ?

As for the SLR lenses like the Zeiss Otus and the Sigma Art, methinks they should be measured on an optical bench with a 2mm-thick flat glass plate instead between the lens and the optical bench's sensor, to approximately reproduce the refraction effects of the sensor's optical stack (infrared filter, low-pass filter and sensor cover glass)
I wouldn't be surprised if the Zeiss' and Sigma's performance were actually improved by the presence of a 2mm-thick piece of flat glass, as their optical designers probably took into account the modern realities of how a lens is used nowadays.

LensRentals Employee

Roger Cicala said:

Nicholas, great minds think alike. Check my next two posts. . . .
That being said, I'm finding it impossible to find 0.8mm optical flats, so I'll have to make do with 1mm for that. But I plan to repeat this entire test with appropriate glass flats in the optical path.

Roger

Jeff Wright said:

Thanks, as always for these tests, Roger. And thanks for the laugh of the day, including the Leica Summilux in the "moderately priced" category. All relative, I suppose. Looking forward to the upcoming tests.

Personally, I don't find the Summilux's field curvature troubling on the M Type 240, but I suppose it might be for some measurbators. It's a much more pronounced problem on the wider Summilux lenses, however.

Kash said:

You forgot about Schneider PC TS Super-Angulon 50mm f/2.8 Tilt-Shift Lens I think this lens would be in a different level compare to these ones.

LensRentals Employee

Roger Cicala said:

Kash, I didn't forget about it. It's not close to the others as far as MTF goes, and by not close I mean it's not nearly as high. That lens is designed for things other than accutance and resolution.

Roger

James said:

bring in a Contax Zeiss 55mm f1.2

AJR said:

Seeing this article makes me glad I decided on the Sigma Art as my new lens.

Now all I need is for Sigma for release an Art 85mm...

philip_pj said:

As always a big thank you, Roger.

As a traditionalist (I guess) I'd love to see a series of traditional CZ/Leica style 10/20/40 lpmm charts at infinity, for wide open, best middle aperture and f8. Lens behaviour is well represented across the frame, to assess 'nasty' aberrations like CoF and to see fall-off rates and extreme corner res.

As an example, the Summicron M in this chart:
http://www.summilux.net/m_system/images/Summicron50.pdf
is not one I'd like to use for landscapes at f2.8-f5.6 for reasons many will see immediately. And taking the highest of sag and tan will definitely obscure its performance, as these lines cross over at several apertures.

It's a problem with summary measures. Also wish to say that several of the sites testing lenses with Imatest are using five year old 21-24Mp cameras even now for C/N (and Sigma) lenses, yet they test all FE lenses on the a7r, with predictable consequences ;-) It seems churlish to complain yet a level playing field is quite important to valid analysis.

Leave a Reply