fbpx
Lenses and Optics

Comparing Rangefinder and SLR 50mm Lenses. (Version 0.7)

Published June 2, 2014

Well before our Imagemaster optical bench was delivered, I knew what I wanted to do first: compare some Leica lenses with the best SLR lenses on a level playing. I did one comparison involving Leica lenses a long time ago, but Leica lenses are difficult to test in our Imatest lab. It’s nothing to do with Imatest or the lenses; simply that mounting and aligning rangefinder cameras in a testing setup designed for SLRs is difficult and very time consuming.

That was a couple of weeks ago. Usually I have the write up done within a day or two of testing. But I’ve never processed and graphed this much data before — the optical bench generates a lot more numbers than Imatest. I tried a few things. I hated them. I tried some more. Finally I asked the good Geeky people who tend to read this blog to help out. I’m pleased to say over 40 people are currently working with another set of data, trying to help me find a good way to overcome my mathematical and graphical limitations to handle all of this data. But that’s going to take some time.

In the meantime, I still have a bunch of information about those 50mm lenses I want to tell you about. So I’ve decided to go ahead and post using the fairly crude methods I’ve come up with so far. Consider this a very preliminary effort since I hope the next set of tests we do will have a far better presentation. Please take it for what it is – a beta trial of a new technology we’re learning to work with.

I’m going to use a mixture of graphs and tables today, but please give me some feedback on what seems clumsy, what seems good, and what I’ve left out that you’d like to see.

Meet the 50mm Contestants

The 50mm lenses we tested range in price from under $1,000 to $11,000. I’ve included the measured focal lengths in this table for those who want to say the Otus isn’t a 50mm lens. It isn’t, of course, but neither are any of the others if you want to be picky.

Lens Price Focal Length (mm)
Leica 50mm f/0.95 Noctilux$10,99552.5
Leica 50mm f/1.4 Summilux M ASPH$4,25051.4
Leica 50mm f/2 Summicron M$2,35051.8
Leica 50mm f/2 APO Summicron M ASPH$7,35051.5
Zeiss 50mm f/1.5 Sonnar$1,20050.3
Zeiss 50mm f/2 Planar$86051
Voigtlander 50mm f/1.1 Nokton$99951.6
Sigma 50mm f/1.4 Art$94949.5
Zeiss 55mm f/1.4 Otus$3,99054.2

While I always like to test multiple copies it’s not always possible with some of the more unusal lenses. We only have a few copies each of the Noctilux and APO Summicron and most were out on rental so I only had 1 copy of each of those available. For the other lenses I tested 3 copies.

Test Results

Distortion

The 50mm focal length is unusual in that lenses can have either barrel or pincushion distortion, although they usually don’t have much of either. Positive numbers are pincushion distortion, while negative are barrel.

Lens Distortion %
Leica 50mm f/0.95 Noctilux-1.0
Leica 50mm f/1.4 Summilux M ASPH+0.3
Leica 50mm f/2 Summicron M+0.25
Leica 50mm f/2 APO Summicron M ASPH+0.3
Zeiss 50mm f/1.5 Sonnar+0.8
Zeiss 50mm f/2 Planar-0.7
Voigtlander 50mm f/1.1 Nokton-0.7
Sigma 50mm f/1.4 Art+0.2
Zeiss 55mm f/1.4 Otus-0.6

None of these lenses have a great deal of distortion, but the Sigma Art and all of the Leica’s (with the exception of the Noctilux) have the least distortion of the bunch. A few of these numbers are a bit different from what has been reported elsewhere, probably because these are measured at infinity focus rather than 8-12 feet where chart distortion measurements would be taken.

Frequency Graphs and MTF50

I’m sure people would like to compare optical bench tests to the computized chart analysis (Imatest and DxO) they’re used to seeing. It’s not completely possible; the optical bench tests just the lens at infinity, while the chart analysis tests a lens-camera combination at close focusing distances. But the optical bench is fully capable of generating MTF50 numbers so I thought I’d try presenting those.

As an example, here’s a frequency graph showing MTF numbers for a lens at 3 positions: Center (red), halfway between center and edge (green), and at the edge (blue). The solid lines show sagittal MTF, the dotted lines tangential (AKA meridian). If you don’t know the difference, it’s explained here about halfway through the article, but the takeaway message is a large difference between sagittal and tangential lines shows astigmatism.

 

If you notice the black bars I put along the left side of the graph, they show where the MTF30, 50 and 70 would be read. The optical bench reads them in line pairs/mm, but simply multiplying that by 24 (the height of a full-frame sensor in mm) gives results in line pairs/image height, which is what most review sites use.

In order to keep things manageable, we’ll show the center of the lens, mid-way between center and edge, and at the edge. I initially gave both sagittal and tangential numbers but that got confusing, so I’ll list the higher of the two MTF readings, along with the astigmatism (difference between sagittal and tangential MTF) at that point.

Each lens was tested at three rotations (0, 45 and 90 degrees) and the average of the three readings used for that lens. Except for the Noctilux and APO Summicron three copies of each lens were tested.

Remember, Optical bench MTF50 number are much higher than Imatest numbers. The bench is testing just the lens. With Imatest we have additional loss of detail through the Bayer array, AA filter, and sensor cover glass; then we’re analyzing pixelated data with some inevitable electrical noise.

Here are the MTF50 results given in line pairs/image height for the lenses, each tested at widest aperture. (The various shades of orange indicate my impression ranging from ‘that’s good’ to ‘hey look at this!!!!)

The Summicron APO kicks some very serious butt in this test, as it should for $7,000. It’s sharper at the edge than all but one other lens are in the center. The Leica Summilux was exceptional, too, especially when one considers it’s shooting at f/1.4 in this test while several other lenses are at f/2. The Summicron, Sigma f/1.4 Art, and Zeiss f/1.4 Otus also did very, very well.

The Leica Noctilux and Voigtlander Nokton, of course, are wide-aperture specialty lenses and aren’t designed to give the best resolution. In this test they are additionally handicapped by shooting at f/0.95 and f/1.1 respectively.

To even things out a bit, I retested all of the lenses at f/2.0.

The Summicron APO still has the best resolution numbers, although several other lenses narrow the gap when closed down to f/2. The Summilux is just a bit behind the APO in central resolution and actually does just as well at the edges. The Sigma lags the best Leicas just a bit in the edges, but it is superb in the center, nearly catching the Summicron APO.

Take a moment to look at the astigmatism numbers, though. Remember, for this test I gave the highest reading from either the Tangential or Sagittal lines, and the difference between the two lines as astigmatism. The Zeiss f/1.4 Otus, in particular, doesn’t have the highest absolute resolution, but has amazingly little astigmatism. (If this had been Imatest, I would have averaged the Sagittal and Tangential lines and used that in my report, which would have made the Zeiss and APO look better and the Summilux look a bit worse.

The Normal f/2 Summicron is also excellent, especially considering its more reasonable price compared to the other Leica lenses. Then again, when you consider it still costs 2.5 times what the Sigma costs, I guess the price doesn’t seem so reasonable. From an MTF50 standpoint, the Voigtlander does surprisingly well, at least in the center and middle regions of the lens, given it’s low cost and wide aperture.

So How is This Different than What I Would Show You on Imatest?

I tried to make this part as similar to one of my Imatest posts as I could. There are some slight differences: this test is at infinity focusing distance, rather than 10-15 feet where we would test this focal length on Imatest, and this test has fewer data points. I’m showing center, edge and corner data, where for Imatest I would show center, average of the entire lens, and corner data.

The absolute numbers are higher, because we are looking at just the lens, rather than the lens-camera interpretation. But if we somehow shot all these lenses on the same test camera, had a theoretically perfect adapter and sensor microlenses, and a billboard sized Imatest chart that let us shoot at infinity, the lenses should rank the same way. Basically, I’d be done now (and since this article has been hanging on my desktop for weeks, that sounds really good). But the optical bench can show us a lot more than MTF50.

More Optical Bench Results

So far I’ve just used the MTF50 number for all of the tables, which is what I give you when I test with Imatest. With the optical bench, though we get a lot more information from a lens than the MTF50.

The graph below shows the MTF chart from one side of the lens to the other (“0” on the horizontal axis is the center of the lens). The red, green, blue, and purple lines represent 10, 20, 30, and 40 lines per mm; the solid lines are sagittal and the broken lines tangential. Notice most of this information occurs at lower frequencies than the MTF50. So does much of the information in your photograph.

The graph above is a single side-to-side cut of a single lens. In our testing, we’d also rotate it 45 and 90 degrees and repeat the readings, letting us look at the lens in two dimensions rather than just one single line from side-to-side. (In discussing this with others, we probably should also do 135 degrees for completeness. We didn’t in this test because our first work with the optical bench was looking at bad copies, and just testing at “0” and 45 degrees was sufficient for that.)

As we average multiple angles and multiple copies, the graphs become nearly identical on each side of “0”. That is why the camera companies only show one side of this kind of graph when they present their MTF charts for a given lens – the other side is simply a mirror image (assuming the lenses are accurately adjusted). I’ll follow that pattern here.

All of these lenses are tested with the aperture wide-open, so you’ll have to take into consideration that for some pairs one lens is at a slightly different aperture than the other.

You don’t have to really understand MTF charts to know that higher is better, and less separation of the solid and dotted lines is better.

The Ultra-wide Aperture Lenses

First let’s compare the Nokton / Noctilux lenses. As we’ve already seen, they’re designed for widest aperture, not highest MTF readings. The Voigtlander, given it’s low cost, certainly holds up well in an MTF comparison against the Leica Noctilux, although neither would be considered a sharp lens wide open.

The Zeiss M Mounts

Next we’ll compare the two Zeiss M-mount lenses. Even taking into account the difference in aperture, the Planar f/2 looks significantly better than the Sonnar f/1.5. The MTF50 results above suggest stopping down to f/2 doesn’t help the 50mm f/1.5 a great deal so the 50mm f/2, from an MTF standpoint at least, seems the better lens. It’s also the less expensive lens.

The Moderately Priced Leicas

Next, let’s look at the ‘lesser’ Leicas, the Normal 50mm f/2 Summicron and the 50mm f/1.4 Summilux. Again, the Summilux is tested here at f/1.4, so it’s spotting the ‘Cron a fair amount of aperture.

You don’t have to know anything more than ‘higher is better’ and ‘lines of the same color closer together are better’ to appreciate that these lenses are really good. In the center the Zeiss 50mm f/2 Planar is certainly comparable to these more expensive lenses, but off-center it has a lot more astigmatism. When you consider the Summilux is doing this at f/1.4 you see why some people are willing to pay a premium for it. It’s at least as good, and perhaps a bit better, at f/1.4 as the other lenses are at f/2.

The SLR Lenses

Next up, we’ll compare the SLR lenses: Sigma 50mm f/1.4 Art with the Zeiss Otus 55mm f/1.4.

In the central part of the lens (out to about 10 degrees off- axis) the Sigma is a bit better, while the Otus is superior in the outer areas. They’re slightly different, but purely from a resolution standpoint they are pretty close, and both are pretty close to the Leica Summilux. Each lens has areas where they are a bit better, but the difference between them would be one of personal preference or Fanboy hair-splitting.

I think it’s pretty cool that two SLR lenses are able to hang right there with the f/1.4 Leica lens, the Summilux. Price-wise, you would hope the Otus would, but certainly the Sigma, one of the least expensive lenses in the test, has to take honors for hanging with the best at a tiny fraction of their price.

I think this is the important take home point of this type of testing. If we just looked at the MTF50 numbers in the first part of the article the Summilux probably seemed the best of the three. It had higher MTF 50 numbers in the center, middle, and edges of the lens, although it also had higher astigmatism. Looking at the MTF curves with a slightly different test, the case isn’t quite as clear and you could certainly make an argument that you preferred any of the three lenses.

(One point I want to make clear is the testing methods differ – the MTF50 tests are done at ‘best focus for that point’. In other words the bench individually focuses at the center, mid point, and edge to generate the highest possible numbers. With the MTF tests we’re showing here, the bench focuses in the best position for the center of the lens and all other measurements are done at that focus. So field curvature will affect these graphs, but not the MTF50 tests done earlier.)

The APO Summicron

And now for our last lens. The MTF50 results in the first part of the article suggest the Leica APO Summicron f/2 is the best lens of the bunch, so let’s see if the more general MTF testing bears that out. I chose the Otus for a comparison graph, but it could just as easily have been the Sigma 50mm Art or the Summilux.

I can see some arguments here or there, but I do think the Leica APO Summicron looks amazing. I wish I’d done this comparison with the Otus at f/2.0, but I don’t think it would have evened things up entirely. The Summicron is the most expensive lens of the bunch, so you would expect it to be the best – and it is.

I won’t pretend to tell you the APO Summicron is worth $3,000 more than a Summilux or an Otus. It certainly is to some people. It would never be a consideration for others. But as the most expensive lens (other than the Noctilux) it should be the best.

Going Forward

Please take this first optical bench test for what it was: a trial to help work the kinks out as we develop our testing and graphing methods on this equipment. It’s going to be fun to be able to compare lenses from different camera mounts without guessing what effect the cameras have. Testing at infinity, rather than close distances, has some practical usefulness, too.

We’re certainly will have a lot more data from every test than we’ve ever had before. Today I just showed a part of it. I haven’t even begun to consider how we’ll present field curvature data. In a few weeks will have an accessory to the optical bench that allows it to test close distances as well as infinity, giving us even more data.

Hopefully after another trial run or two we’ll have things settled down into a standard format. I’m sure some of the contest entries will help us make this more standardized and automated. But in the meantime, please send me any suggestions you have that might make things easier to understand.

Roger Cicala and Aaron Closz

Lensrentals.com

June, 2014

Author: Roger Cicala

I’m Roger and I am the founder of Lensrentals.com. Hailed as one of the optic nerds here, I enjoy shooting collimated light through 30X microscope objectives in my spare time. When I do take real pictures I like using something different: a Medium format, or Pentax K1, or a Sony RX1R.

Posted in Lenses and Optics
  • Roger Cicala

    Kash, I didn’t forget about it. It’s not close to the others as far as MTF goes, and by not close I mean it’s not nearly as high. That lens is designed for things other than accutance and resolution.

    Roger

  • Kash

    You forgot about Schneider PC TS Super-Angulon 50mm f/2.8 Tilt-Shift Lens I think this lens would be in a different level compare to these ones.

  • Thanks, as always for these tests, Roger. And thanks for the laugh of the day, including the Leica Summilux in the “moderately priced” category. All relative, I suppose. Looking forward to the upcoming tests.

    Personally, I don’t find the Summilux’s field curvature troubling on the M Type 240, but I suppose it might be for some measurbators. It’s a much more pronounced problem on the wider Summilux lenses, however.

  • Roger Cicala

    Nicholas, great minds think alike. Check my next two posts. . . .
    That being said, I’m finding it impossible to find 0.8mm optical flats, so I’ll have to make do with 1mm for that. But I plan to repeat this entire test with appropriate glass flats in the optical path.

    Roger

  • Nicholas

    The Leica Summilux 50mm F/1.4 exhibits a significant field curvature (or wobbliness) when used on a digital camera — e.g. the Leica M9 or the M Typ 240 — which results in annoyingly distracting loss of sharpness on a 24x36mm imaging area, along a donut-shaped area about 10mm from the picture center.

    As the Leica Apo Summicron 50 F/2.0’s MTF curves look quite similar to the Summilux’s on Leica’s data sheets — the same kind of wobbly, tortured MTF curve — it’s possible that the actual performance of the Apo Summicron on a digital sensor would be far less impressive than the measurements on an aerial image optical bench would suggest.

    To reproduce the image degradation — astigmatism, field curvature etc. — that the cover glass of a Leica digital camera induces with rangefinder lenses with a short exit pupil to imaging plane distance, would it be possible to insert, between the lens and the optical bench’s sensor, a glass plate of the same thickness (0.8mm) as the cover glass used on Leica cameras, and check how the measurement values are affected when testing the lens in an optical configuration that more closely approximates the physical layout of a Leica digital camera ?

    As for the SLR lenses like the Zeiss Otus and the Sigma Art, methinks they should be measured on an optical bench with a 2mm-thick flat glass plate instead between the lens and the optical bench’s sensor, to approximately reproduce the refraction effects of the sensor’s optical stack (infrared filter, low-pass filter and sensor cover glass)
    I wouldn’t be surprised if the Zeiss’ and Sigma’s performance were actually improved by the presence of a 2mm-thick piece of flat glass, as their optical designers probably took into account the modern realities of how a lens is used nowadays.

  • Roger,
    The Noctilux Summicron show astigmatism of 24 at center. other that an out of alignment lens or a slightly cylindrical element how is that possible?
    Also in the tradition of what is the sound of one hand clapping, what would be a Sagittal or Tangential line at the center of the image?

  • Dan

    THanks so much for this test. Very impressive! It would be great to see results at f4 and f8 too some day.

  • Roger Cicala

    Zlatko – it’s possible there are typos. I double checked when collecting and again as I entered data, but the article took so long to write as I tried different ways to present the data that I might have transposed something going from one form of presentation to another. I’ll have to get the original machine tracings out and double check. But after what we’ve found with the sensor stack issues, I also need to retest the Sigma and Otus lenses. They might actually be better than they appear from my test results.

    Roger

  • Zlatko

    Some of those numbers are unexpected — could there be some typos? The 50/1.4 Summilux really doesn’t improve at all at center or middle when stopping down from 1.4 to 2.0? Jeff M. and Dibyendu have commented about this above. Also, the old 50/2.0 Summicron is perfectly identical (1200) at center, middle and edge?

  • Lee Saxon

    I have an idea, although this is really just for presentation and not data-wrangling.

    Setting the MTF graphs side-by-side like only lets you compare two lenses at once, makes them super skinny, and makes the page run really long.

    Maybe you can find a little rollover app like Lloyd Chambers uses on his aperture series tests to let people switch between lenses. But please please please change it to onclick instead of rollover, that part sucks.

  • Eric G.

    Hi Roger,

    With this new testing equipment, looks like you’ve got your hands full.
    So much data to process, which makes it very hard to present it in an easy-to-understand and meaningful way.

    Once you’ve got all your data analysis in order, and after you get the separate finite conjugate bar in place (and have enough spare time), I would like to suggest a shoot-out.

    Of the many articles I’ve read, I’ve assembled a list of the best prime lenses ever made. I would very much like you to properly test these prime lenses (in the same time frame). They have different focal lengths, which might complicate matters, but this would make a very interesting article to read nonetheless.

    Here are my five top prime lenses:
    ———————————-
    1. Zeiss APO Sonar T* 135mm f/2
    2. Zeiss Otus 55mm f/1.4
    3. Leica Summicron APO 50mm f/2
    4. Nikon AF-S Nikkor 200mm f/2G ED VR II
    5. Sigma ART 50mm f/1.4

    I really hope someday to read your full findings on these lovely pieces of glass.

  • Denis

    @Weilli
    I am not sure if this kind of test can be done for FE 55 with its focus by wire system. Since no camera used in testing thus you cannot control the lens and that’s why you just cannot focus it and control its aperture.

  • Jake

    Thank you for sharing these tests.

  • Eric

    Nice review but you could have shown a family photo with all the lenses and a table showing the specs of each lenses, especially the size & weight.
    The Leica APO-SUMMICRON-M f/2 takes 39mm filters and weights a mere 300g: clearly much smaller than the Otus and the Sigma.

  • Andrew

    Great article Roger, makes me really appreciate just how good the non-APO Summicron is for a 1979 spherical design.

  • Sam Tiata

    At the risk of appearing dumb { it is a specialty of mine} regarding the intricacies of optical testing. From a real life output viewpoint would it be possible to actually see the difference between the 1680 line pairs/image height of the APO compared to the 1560 line pairs/image height of the Sigma in any size of print.

    And in a similar vein when shooting at F2 or wider on a FF sensor would the extremely shallow DOF and even smallest of field curvature effect seriously impact these results, firmly moving them into the theoretic . It will be interesting to see what a rumored curved sensor would do with regards to resolution

  • Dan

    OUTSTANDING test Roger, thank you very much for this!

  • Roger Cicala

    Spaceman, largely to get feedback and learn how to present our data more clearly. As to lens-camera combinations, adapted lenses give hugely inaccurate results on Imatest, etc. Not only the inevitable tilts of another set of mounts plays in, but variations of sensor-stack thickness compared to the stack thickness the lens is designed for matter. The only way we can think of to clarify this is to test lenses alone and then in conjunction with different cameras to see all the possibilities.

  • John Leslie

    Are the Summilux figures really identical to the line of resolution for the centre and middle between f1.4 and f2? Seems a bit odd…

  • Spaceman

    Interesting, but why publish results when there are so many uncertainties according to yourself. To many ‘non-scientific’ tests polluting the space. Thank you for specifying that this indeed is a “beta” though.

    Also, wouldn’t it be more relevant to test a lens camera combination along with these results? People hardly take pictures only with their lenses 😉 A Sony A7R would be able to take all these lenses and you’d be able to test the FE 55mm as well.

    Interesting test though. Thank you.

  • I just used my Nikon V1 adapter and fitted the sigma art 50mm to see that it is even good enough for a 100MP DLSR see the samples at
    http://www.luminous-landscape.com/forum/index.php?topic=90563.msg737391#msg737391
    i think f2.8 is is sharpest for the center.
    I just ordered the USB dock so thank you Roger for the warning…

  • alberto

    Hi Roger.
    First of all, thank you. It is a pleasure to read your excellent and thorough article. While You as a comment, the results can not be directly extrapolated to other tests based colleagues Imatest (photozone, DxOMark Etc.), it is interesting and enlightening. And others showing such optical behavior at infinity. Also (and this I think is very important), shows the real potential that holds a lens with respect to future improvements in cameras and sensors.Thanks again ..

  • Joachim / CH

    Thanks for the help with the post, I just want to collect most of your posts as they all are a great read and I highly appreciate the passion and knowledge you share with those.

    The Nikon version is still alive and hopefully remains to stay this way. And the “problems at infinity” are less a lens problem than more one of the “autofocus guessing feature” of a normal D800. Can’t switch that off.

    Don’t know if you need to be so kind to Sigma. Even a harsh/wrong procedure of firmware update shouldn’t kill the electronic hardware – pity that an optical superb lens depends on the quality of “cheap” software. One could argue “with Zeiss / Leica this kind of failures isn’t happening – no AF, no electronics”. But even an electronically dead Sigma will still serve as manual focus. I just want to see the person getting the best focus wide open with only the matte screen. 🙂

  • Roger Cicala

    Joachim,

    The article is still up. It’s here: http://wordpress.lensrentals.com/2014/04/yet-another-sigma-50mm-art-post

    We had 3 50mm f/1.4 lenses electronically die after upgrading firmware via the dock – all Canon mount and two different docks on two different computers, so I’m certain we don’t have a bad dock. But not certain what happened. Since no one else has mentioned it, it may be something done wrong on our end.

    Roger

  • Joachim / CH

    I was just looking for the “Another Sigma 50/1.4 post”, but it appeared deleted?

    Has this something to do with the rental warning?

    “Warning: We have had multiple issues with the electronics of the lens locking up after using the Sigma USB Dock and latest firmware. We strongly suggest foregoing the use of the dock for the time-being until more information is known. With that said, rent this lens at your own risk!”

    Could you explain a bit more? What happened and waht is the “latest firmware version”, because when I checked, the “optimization software” told me, version 1.00 is the latest version. I did some customization, although I’ve no idea about the manual focus lock. So far, I made some 250 pictures which were very sharp but sometimes it doesn’t focus on infinity. Besides, AFMA is a huge difference between D800 and D7100, that’s a bit weird.

  • Roger, again a very interesting read
    I just got my Sigma art 50mm a week ago and i am impressed with what i see. (never understood why Nikon did not make such a lens while introducing a 36mp camera- clearly there is a market for it)
    From what i see in my photographs it is f 5.6 that is the best aperture overall. Probably it will be the same for the Otus and Leica Apo summicron. I very much would like to see that comparison. Since you are testing the lens only i would like to know what theoretical resolution is possible on a bayer sensor of 36MP, i have not been able to find that info yet.

  • Weiss Wang

    Great comparisons. Which digital back did you use to test these lenses?Sony Alpha 7/7r ?

  • Nqina Dlamini

    Great read. The Sigma is giving me a slight twitch on my wallet strings.
    Thanks for the effort and presentation.

  • Weiss Wang

    Great comparisons. Which digitalback do you chose to test these lens?Sony Alpha7?

  • Roger Cicala

    Christian, yes, it will soon. There’s a separate finite conjugate bar that will be here in another month to let it test at closer distances too.

Follow on Feedly