LensRentals.com

Wide-Angle Micro 4/3 Imatest Results

Posted by

Note: The results for standard range m4/3 lenses and zooms at 25mm are in the next article.

A lot of Sony users liked seeing some Imatest results for NEX lenses, and some micro 4/3 users asked us to do the same thing. That’s a more complicated undertaking for several reasons. First, there is a much wider selection of native lenses for micro 4/3. Second because there are a lot of different micro 4/3 cameras. But we wanted to tackle the project because, quite honestly, we’d never done any lab testing on m4/3 lenses. We wanted to know, too.

Making the project something we could do in reasonable time means cutting some corners. First we can’t test every camera. We chose the Olympus OM-D EM-5 for several reasons. We wanted a camera with 16-megapixel resolution, which eliminated most m4/3 cameras. Also, the physical layout of the Olympus lets us test more quickly in our lab than we can with any of the 16-mPix Panasonic cameras. There just wasn’t enough time in one day (all we could set aside for this right now) to do two different cameras.

Second, we split the test into two sets of lenses because there were so many to test. In this part we did lenses that were 25mm or wider in focal length. We’ll do the longer lenses later. Zooms with a long range were tested near their wide end in this test, and will be tested again at longer focal lengths when we do the longer lenses.

Disclaimer

The usual “don’t read this stuff and go insane” cautions apply: this is a test of lens and sensor resolution for MTF 50 (detail resolution) done at a distance of 10 to 30 feet, depending on focal length. The results might be different at 4 feet or 400 feet. The results might be difference if we tested at MTF 10 for best acutance. Focus is done manually and bracketed so it’s not a test of a camera’s autofocus ability. The tests use a controlled-lighting test-target with a hardware mounted camera at ISO 400, so photos taken in the dark of night or hand-held for 12 second exposures will give slightly different results. It doesn’t measure autofocus speed, lens size, manual focus feel, bokeh, color rendition or anything like that. It’s not a lens review; it’s one simple test of resolution.

More importantly, remember that while we are testing RAW files, both Olympus and Panasonic seem to do some in-camera corrections on their own lenses, but not the other brands. We don’t know what effects, if any, that might have on this test. Possibly Panasonic lenses might do a little better on Panasonic cameras and Olympus lenses a little worse. We’ll figure that out later, but I doubt any such corrections will affect the MTF 50 at all. If it did the effects should be minor and limited to the edges and corners. But we don’t know for sure. Yet.

Addendum 4/13/12: I had hoped that the m4/3 community might not be quite as Fanboy riddled as Canon and Nikon. Obviously I was wrong about that. Because there was such an uproar about the results of the Panasonic / Leica 25mm I’ve repeated the tests on 6 copies (all that we have) and also on Panasonic cameras, because Fanboys have been on suicide watch and full-attack-mode since they’ve taken them so out-of-context. The average of 6 lenses on both Olympus and Panasonic 16 Mpix cameras are now reported. The results are basically unchanged.

The Lenses

We had a nice selection of wide-angle m4/3 lenses in stock on the day we conducted these tests:

Imatest Results

The values are for MTF-50 (which correlates with fine detail resolution) measured in line pairs / image height. The first (higher) number is the MTF 50 measured at the center of the lens only. The second (lower) number is the average of the MTF-50 measured at 13 points including the center, 4 corners, 4 sides, and 4 mid points. I keep a point total for every time someone posts the question “what do the two numbers mean” as an indicator of the reading comprehension for each group of camera users. I have high expectations for m4/3 users, so please don’t let me down.

I’ll also mention that numbers sometimes make minor differences seem really large. A difference less than 50 lp/ih is probably not noticeable in real-world photography. So if you print a reasonably sized image from a lens resolving 860/750 lp/ih and compared it to a shot with another resolving 810/700 lp/ih the difference would probably not be noticeable. You would almost certainly notice a difference of 100 lp/ih.

At Widest Aperture

The first graph shows the numbers for each of our lenses (zooms measured at 30mm) with the lens aperture wide open. This test isn’t a level playing field, since some lenses are f/4 wide open while the primes are at anything from f/1.4 to f/3.5. But since some people tend to shoot every lens wide open, it may be useful to know how well (or not) the lenses do at their widest aperture. I’ve listed them from widest to longest focal length. The zooms were all shot at 14mm so they might be a bit better or worse at other focal lengths.

MTF 50 at Widest Aperture

Lens Max Avg
Olympus 12mm f/2.0 860 730
Panasonic 14mm f/2.5 950 785
Panasonic 7-14 f/4 965 765
Olympus 9-18 f/5 805 680
Olympus 12-50 f/3.5 825 680
Olympus 14-42 II f/3.5 720 600
Olympus 17mm f/2.8 720 590
Voigtlander 17.5mm f/0.95 565 475
Panasonic 20mm f/1.7 870 735
Voigtlander 25mm mm f/0.95 530 435
Panasonic/Leica 25mm f/1.4* 690 590

*Panasonic-Leica results on this test are the average of 6 copies. The average numbers are slightly better at f/1.4 than  the first version of this article, which reported only one copy, but the difference is not really significant.

There’s not a lot to see from this set of numbers. As usual, good primes are as sharp, or sharper, at wide apertures than consumer-grade zooms are at lower apertures. The Panasonic pancake lenses pleasantly surprised me. Pancakes are usually not the greatest lenses, but he Panasonic 14mm and 20mm are really quite good.

Stopped Down to f/2.8 or Smaller

We can level the playing field a little bit by shooting the wide primes at f/2.8, as shown in the graph below. It’s somewhat more meaningful: the zooms are still at smaller apertures, but everything else is shot at f/2.8 here.

MTF 50 at f/2.8

Lens Max Avg
Olympus 12mm f/2.8 1000 845
Panasonic 14mm f/2.8 1010 850
Panasonic 7-14 f/4 965 765
Olympus 9-18 f/5 805 680
Olympus 12-50 f/3.5 825 680
Olympus 14-42 II  f/3.5 720 600
Olympus 17mm f/2.8 720 590
Voigtlander 17.5mm f/2.8 965 810
Panasonic 20mm f/2.8 1050 875
Voigtlander 25mm mm f/2.8 995 855
Panasonic/Leica 25mm f/2.8* 960 820

*Panasonic-Leica results on this test are the average of 6 copies. The average numbers at f/2.8 are slightly better than  the first version of this article, which reported only one copy, but the difference is not really significant.

By f/2.8 the superiority of most of the primes over most of the zooms becomes pretty apparent. The Olympus 17mm f/2.8 is easily the weakest of the prime lenses, which isn’t really surprising. It’s an older design and a consumer-grade pancake lens. The two Panasonic pancakes being so good surprised me, though. They’re not much more expensive than the Olympus 17mm. I had expected a bit more out of the Panasonic Leica 25mm given it’s cost and build quality. It has other good qualities, this is simply the results for MTF 50. The Voigtlander lenses, despite being built for ultra-wide aperture (which usually results in a rather soft lens) did quite well. But as easily the most expensive lenses in the group, that seems appropriate.

Personally my biggest disappointment was in the Panasonic 7-14 f/4 zoom, which is probably my favorite m4/3 lens. But we were testing at 14mm, and that lens is actually a bit better at the wide end. And really, it’s probably a bit much to ask a lens to be this incredibly wide, and also incredibly sharp. The Nikon 14-24 pulls it off for about $2,000, but that’s the only one I can think of.

MTF 50 at f/4

We can level things out even further by shooting all of the lenses at f/4. Most of these lenses would get a bit sharper still at f/5.6, especially in the average (as opposed to center) resolution.

MTF 50 at f/4

Lens Max Avg
Olympus 12mm f/4 1040 870
Panasonic 14mm f/4 1025 860
Panasonic 7-14 f/4 965 765
Olympus 9-18 f/5 805 680
Olympus 12-50 f/4 830 685
Olympus 14-42 II f/4 730 605
Olympus 17mm f/4 735 610
Voigtlander 17.5mm f/4 1070 835
Panasonic 20mm f/4 1075 880
Voigtlander 25mm mm f/4 1030 925
Panasonic/Leica 25mm f/4* 980 850

*Panasonic-Leica results on this test are the average of 6 copies. The average numbers at f/4 are slightly lower than  the first version of this article, which reported only one copy, but the difference is not really significant.

There’s really not much new information here that we didn’t already see at f/2.8. The Olympus 9-18 and 12-50 zooms are clearly better than the little 14-42 zoom. The Panasonic 7-14, despite working at such wide angles, is better than either of the others. Overall another “you get what you pay for” situation – slapping  II on the name doesn’t make it a great lens. (BTW – although I expect m4/3 shooters to go a little less Fanboy than the Canon and Nikon guys, let’s get this out of the way: some Fanboy always says “the rental guy wants you to rent the most expensive lens”.  Actually the rental guy could care less: the profit margin is the same on a really expensive lens and a cheap one.)

Of the prime lenses, only the Oly 17 pancake isn’t very good; the Panasonic pancakes were clearly better. The Panasonic/Leica 25mm disappointed a bit: it was good but I had expected it to be THE best of the bunch and it’s not. The Olympus 12mm was just excellent, though, and the Voigtlanders provide an amazing wide aperture lens.

Part 1 Summary

Compared to the NEX system lenses I just tested, m4/3 users have a host of wide-angle, native-mount choices. There are good pancakes, standard primes, interesting (and expensive) ultra-wide aperture primes, all at reasonable prices. There’s a good f/4 ultrawide zoom and a couple of reasonably sharp zoom lenses. The big hole, as I see it, is that there’s no wide-aperture zoom, period. For my shooting style, the m4/3 begs for a wide aperture.

Remember, we’re just talking about resolution here. When I’m choosing a lens, it’s my starting point: I require good resolution. But that’s all it is, a starting point. If you’re seriously considering some of these lenses, go to a good review site where you can get more in-depth information about distortion, handling, focusing accuracy, etc.

In this part we’ve looked at most, but not all, of the m4/3 lens choices for shooting at 25mm or wider. We’ve only tested on the Olympus OM-D. We’ll do some testing on at least one of the Panasonic 16mm cameras just to make certain there’s no significant difference soon (I doubt there will be, but it’s possible). We’ll also do a second post on the longer lenses in the next couple of weeks.

Roger Cicala

Lensrentals.com

May 2012

64 Responses to “Wide-Angle Micro 4/3 Imatest Results”

LJC said:

Like the new site; hate the new blog! (Not the content of course, but the layout)

It looks like you’ve lost a lot of your formatting that aided readability (Is that a word?). Or perhaps its just interpretating line feeds differently than the last version resulting in a wall-o-text.

On the plus side, it looks as though you’ve addressed the wrapping issue that was preventing me from reading the blog earlier in the week.

Also, currently on my screen below this text box I have a big blue banner containing URL links on the left hand side, however the URLs are the same colour as the background so all I can see is the headers (In order from top to bottom; Copyright notice, address, Navigation, 8 invisible links that I can see by using select-all, Navigation (again), 3 invisible lines, Support, invisible, Follow Us then 2 invisible URLs).

I would assume that this is version 2.0 and a nice shiny 2.1 is on the cards after you’ve paid some lackey to perform some useability testing, but I’m way too cynical ;p

LJC said:

Amendment to previous message: It’s stripping out blank lines or duplicate line feeds which is causing the wall of text problem (URLs same colour as background kinda unforgiveable though ;p)

rich T said:

Hi there,
Fantastic test – thank you so much for the effort.

The only lens missing I think would be really interesting to include is the Panasonic 14-45mm – Its currently the sharpest mid-range/standard zoom available (until hopefully the 12-35 X lens will beat!)

If you get time to add the 14-45mm I (and many others) would really appreciate it.

LensRentals Employee

Roger Cicala said:

hi Rich,

I absolutely will — the only reason it isn’t on this first set is it was out of stock. Which makes sense since it probably is the best mid-range zoom.

Roger

LensRentals Employee

Roger Cicala said:

LJC – are you in IE 8? I know we’ve changed formatting and it’s causing all kinds of IE 8 problems, which they’re working on now. Like most new site rollouts, we’re doing about 100 bug fixes a day, but finding about 150. Thanks for the comments, it helps know what kind of problems people are seeing.

Dave said:

Very informative, and as you say, a good starting point. I’ m curious about why you use Iso 400 and not the camera’s base iso, not that it would change the relative results any. Also curious if image stabilization would have an effect (my understanding is you can’t turn it off on the em-5, though the usual recommendations with other systems is to turn IS off when on tripod). This might make a difference depending on shutter speed, but again wouldn’t affect the relative ratings as you have controlled for exposure. Just curious.
Lastly, with a cdaf sensor, why not use Af instead of manual for the tests? Doesn’t this remove a variable (user technique for MF)?

Richard said:

This test shows pretty much exactly what I saw with a few lenses. The Voigtlander is lame and overpriced and arguably, you can produce a better underexposed and luminance raised image with the 20mm than it. Meaning, the Voiglander’s speed is only good for defocusing the background. The 14mm also bears out what I experienced, even though I had read tests that said it wasn’t that good. Maybe there are variations in its quality?

Sandeep said:

You have overlooked the new Sigma lenses, the 19mm f2.8 and the 30mm f2.8. Looking at the MTF of the Sigma 19mm it looks like it has better edge performance than the Panasonic 20mm.

LensRentals Employee

Roger Cicala said:

Not overlooked the Sigmas, just didn’t have any to test that day. We’ve got lots more to test, but each lens is an hour or two of work, and the equipment’s primary use is testing returning lenses so we have to do these when workload allows :-)

I plan on getting to all of the lenses. First priority was to repeat a few on a Panasonic body to make sure there weren’t any major differences between cameras, then hopefully I’ll get another batch done next week.

Roger

Tom said:

Hi Roger,
Great test, thank you. I’m feeling very chuffed, as I just added the 14/2.5 to the 20/1.7 and 45/1.8. I researched this lens for ages (I’m “careful” with my money), and felt that it was at least equal to the 12/2 for resolution. Add in the size and cost, it was a no-brainer. Now you’ve made me feel very clever….
When shooting landscapes with the 20/1.7, I’ve typically stopped down to f8. I use an E-PL1, so often need to go that small to get the shutter speed low enough anyway (I live in the Middle East desert & New Zealand mountains). Do you think I’d get better results a little wider with an ND filter?

LensRentals Employee

Roger Cicala said:

Good questions Dave. We don’t know if the IS has any real effect, but shutter speeds are fast given the good lighting and we haven’t seen any effect on or off with other setups.

We use ISO 400 just because it seems a practical test point where things are pretty even. No camera is at it’s base, but none are struggling with noise, either. The decision was arbitrary but once we started there, we keep going: it’s where our database is shot at and it’s a big database.

We focus bracket every shot picking the best shot. Probably AF on this camera would be just as good, but it wouldn’t be better. It just eliminates a variable.

Roger

Sylvain said:

Roger, thanks for the nice effort.
Just a little down, I think your result would gain much being put in a simple chart.

greetings,
S.

john said:

Regarding panny 20mm f/1.7 and pannyleica 25mm f/1.4, care to explain why you got such results totally in contrast to those of slrgear (and many others tests)? There one can see the 25mm f/1.4 clearly shows much less blur units than the 20mm f/1.7 at ALL apertures.

You’re in fact the only one who says the 20mm f/1.7 is sharper than the leica 25mm!?

Esa Tuunanen said:

Roger, how about testing Zuiko 12-60mm for comparison before someone thinks m4/3 has native lenses boasting with optical quality?
Makes supposedly premium prime of 12mm look not so high quality.
http://pekkapotka.squarespace.com/journal/2011/12/30/olympus-mzuiko-12-50mm-f35-63-in-comparison.html

Tom, if you think you’ve got lots of light in desert try fully snowy landscape, that’s why E-M5′s base ISO of 200 was slight disappointment for me.
But f/8 is good compromise between small aperture and avoiding bigger diffraction softening.
With pixel size of 4/3 sensors (and 20+MP APS-C) Airy disc starts to grow larger than pixel little before that but impact starts to really grow after f/8.
Photozone has easy to read reviews where effect can be seen:
http://www.photozone.de/Reviews/overview#fourthirds
(lots of reviews for most systems)

LensRentals Employee

Roger Cicala said:

John,
Do you mean other than the explanation in the text about only having one copy to test and being concerned it may be a bad copy, planning on retesting as soon as other copies get back, asking you to take the results with a grain of salt until then, and mentioning that it might test differently on a different camera? Do you want other explanations than the ones I’ve given? Well, I guess testing on 16 megapixel cameras instead of 12 might be it, or perhaps testin on an Olympus body. But I think the explanations I already gave in the article are more likely. Oh, wait: you didn’t read the article did you? Just teed up and went Fanboy?

Nilesh said:

Thanks for the tests Roger…helps me decide which system to for mirrorless. I wish there was something like Panasonic 20 f1.7 at at that price fpr NEX.

Hate to say this but don’t care much for the new blog layout! font too small(I know I can change, but why do just for your site?), comments and content crowded etc.

Richard B said:

Thanks for the very interesting test. Do you plan to test the 45mm f1.8? The few people I have spoken with who use it seem to like it, indeed they seem to think the 12mm/25mm/45mm combo covers a great deal of their needs.

jim thomson said:

Great set of tests.
Biggest surprise was the Panasonic 14mm. This lens usually gets a negative comparison to the 20mm. Looks like it held its own in this set of test.
I’m curious to see how the Sigma 19mm stacks up given how well the 30mm did on your NEX tests.
It would also be nice to see how the Panasonic 14-42 kit compares to the Olympus.

Christian Grevstad said:

Can you test the new pz 14-42 pancake lens?

Given Panasonics claim that it is better than the 14-45 and several user tests that comfirm this, I would find this very interesting.

Rostislav Alexandrovich said:

wow.

Well done mate! thanx so much for another good source of info on the net, i was taken by the 25mm relative dissapointing performance, it was said to be as sharp as 20mm,

I hope that the extra $$ in 25mm would bring a better rendering and bokeh, but i have yet saw a good comparison between the two.

PDC said:

A thorough and very helpful review. I have the Panasonic 20mm, 7-14mm and both of the Noktons (17.5 and 25). My personal assessment of these puts them in a similar rank, thought the 7-14 should be considered a special purpose architectural lens (and it does a very good job). The Noktons are amazing, although I recently tested the 17.5 for chromatic abberation and purple fringing is definitely an issue wider than f2.
See:
http://www.drivehq.com/file/df.aspx/publish/pdcsylva/JPEGsCV17TestingPDC
or http://www.drivehq.com/folder/p9818891.aspx
for a series that tests Voigtlander Nokton 17.mm axial chromatic abberation (purple fringing artefact) in weak daytime backlight conditions.
#23, #32 F0.95
#24, #33 F1.4
#25, #34 F2.0
#26, #35 F2.8
#27, #36 F4.0
#28, #37 F5.6
#29, #38 F8
#30, #39 F11
#31, #40 F16
All processed through Lightroom 4.1 from RAW (Panasonic GH2)
For -2 series Lightroom Manual Lens Defringe AllEdges applied
For -3 series Defringe plus “Direct Positive” tone preset applied
Conclusions: Apertures wider than F2 will produce significant purple fringing in fairly weak backlighting. Lightroom Defringe marginally effective in removal.
Recommendation: Avoid backlit scenes if shooting wide open. Stopped down to F2.8 or smaller will avoid purple fringing.

David said:

Interesting test, thanks.

LJC said:

Yes I’m on IE8. It’s the most popular browser version currently in use (both worldwide and in USA according to pingdom and statcounter) so tbh you should have “words” with your software house. (

imo) New websites, especially those used to generate income, should be developed on IE8 in the first place then check it’s compatibility with other browsers and versions so that you best reduce the risk of irritating customers. But I seem to be in the minority there..

Unless this was a freebie, in which case you get what you pay for ;p

Signed,
A. Developer

David Bateman said:

Excellent work and great test. I will look forward to follow up test with GH2 (hopefully). Also I would like to see the >25mm lenses when you post it. Thinks that are way too much work for you, but I would like to see are the zooms at multiple points rather than just 14mm. To see how they fair at min, max and middle. Also hopefully you can add results with Sigma 30mm, that was the best in your Nex test and I would like to see the numbers on the EM5 for comparison. I mean are they the same (1040/840 @f4) to then add possibility to Compare across systems.

I did like that you didn’t add a paragraph as you did in the NEX7 test to compare across D800, NEX7 ete. People will do it anyways.

ginsbu said:

Thanks for doing this test. One thing that would make the numbers more meaningful to me would be to report a minimum measurement (or an average of just the corners) in addition to the maximum and average.

john said:

Roger,

Yes, I did read the article. Indeed, I noticed your randomly-spread-throughout-the-text guesses trying to explain why the pannyleica performed “poorly”. But if one of your lenses is not measured in the same conditions as the others, you should EXPLICIT say that in every table this “little” issue, for instance using an asterisk on the lens name in every table and its correspondent legend.

You just had one copy of the pannyleica. How many copies of the panny 20mm f/1.7 you used? And how many times you repeated the test for each of the above lenses?

Lee said:

Wow, lot of ill tempers today.

LensRentals Employee

Roger Cicala said:

John,

All lenses were, of course, measured in exactly the same way with best results from focus bracketing reported for each. We had a second copy to confirm results with all except the Pan/Leica, as I said.

Roger

LensRentals Employee

Roger Cicala said:

LJC,

Boy, do I wish it were free. They’re working on the IE 8 fixes, but since all the other browsers (correct me if I’m wrong, I don’t do this kind of stuff, so I’m repeating what I’m told) use certain protocols that IE 8 doesn’t they developed it in the other formats and then are doing the IE 8 fixes. I’m sure IE 8 is the single biggest, but the others together are much larger. I’m told IE 9, though, works fine.

The fixes are largely done on the website, but the blog, understandably, had back burner until the website was fixed.

Roger

Lester said:

Love your tests! I was very pleased with my Panny 7-14, I thought it might do a little better than it did in your tests. The PL 25 should certainly have done better, awaiting your confirmation of a poor copy. And my copy of the Panny 20 was poor and so never showed what you found, rats… My Oly 12-50 does exactly what you saw, as does my Panny version of the 14-42. I dismissed and sold my Panny 14-45 too quickly, I now regret it, look forward to you testing it.

George said:

Great tests, your blog has <3 articles. Just goes to confirm, imho, that micro 4/3 is the most solid mirror-less system at the moment. Looks like a ton of solid budget choices. I'm impressed by how well the 12-50mm does, really. Very acceptable. Curious to see what the tests look like on a Panny camera. As to the pannaleica results, meh, it's not terribly surprising. Remember that results may differ depending on things like: the camera (as you said) and also things like the distance the test is being performed at (some lenses are better at infinity, some better at mid ranges, and so on).

Full Lumen said:

Most interesting, thank you Roger. If you get the chance try some of the Zeiss ZM primes with a M43/LEM adaptor.
My own observation of my Zeiss 28mm, 50mm and 85mm on my GH2 and my newly acquired OM-D is they outperform
all of my Panny and Oly lenses including the Lumix-Leica 25mm. With the 25mm this is especially true at close distances and at wider apertures where the 25mm doesn’t do very well. Bear in mind this is with my particular copies of these lenses, other copies may perform differently. On the other hand the 25mm is an f1.4 lens. Mind you you the Zeiss 50mm at f1.5 easily outperforms it though maybe comparing a 25mm lens with a 50mm is unfair. Anyway give it a go if you can and see if your conclusions match mine.
Finally, I wonder how the Zeiss ZM 18mm f4 and 25mm f2.8 perform on M43? I don’t own ether of these but it would be great to find out.

ginsbu said:

Thanks for the retest of the PL25. I’m not surprised you’ve got pushback on the results there. There are plenty of m4/3 fanboys, but also your results for the lens — especially compared to the 20mm — don’t line up with previous tests from LensTip and SLRgear. I’m not sure what the explanation for that might be. In any case, the PL25 has advantages in aperture, rendering, and focusing speed that make it a fine choice even if it falls behind the 20mm in absolute resolution.

This review encouraged me to pick up the Panasonic 14mm. “Dekitted” copies are so cheap right now that it was impossible to resist what your tests show to be a very decent performer.

LensRentals Employee

Roger Cicala said:

Ginsbu – well spoken! It’s nice to have rational discussions about this.

I wonder myself about the difference, too. SLR Gear tests with DxO optics I believe, which uses DxO’s proprietary ‘blur experience unit’. While there is argument about what that actually is ( http://www.imatest.com/docs/blur_mtf/) it would seem to emphasis acutance more than fine detail (while the MTF 50 we’re testing would emphasize fine detail).

I can also understand difference with sites that measure jpgs, rather than raw images that we insist on using, especially with m4/3 and other cameras known to apply in-camera sharpening. Lenstip, on the other hand, tests in a method very similar to our own (probably why I put so much trust in their results) although they use a different chart type.

For these reasons I was hesitant about our own results which is why I wanted to retest multiple copies. The retest showed a very tight grouping so I believe our results are accurate for MTF 50. I’m not certain yet of the explanation for the difference between our results and Lenstip’s (I can understand difference with other sites using other methods). I suspect it might be that we were measuring on 16 megapixel cameras versus 12 megapixels. It may also be that they aren’t measuring MTF 50 but rather MTF 20 or 30. There’s good reason to do it that way (it’s probably a more accurate measurement of what you see on small prints and online while MTF 50 is more correlated with small detail or microcontrast that really shows up on larger prints).

At any rate, while it certainly seems to aggravate the fanboys I think it’s just another bit of useful information: I wouldn’t buy a lens solely on the basis of MTF 50 results any more than I would buy a camera solely on the basis of how many megapixels it had. It’s just one more piece of information for people to consider when making choices. There’s a reason that SLR Gear, Lenstip, Photozone, The Digital Picture, DPR, etc. are all places I go to look at results: They spend days evaluating every aspect of a lens and present thorough reviews. My contributions are not competition with that — I do superficial testing of one aspect of the lens on multiple copies (which review sites don’t get) to look at things like sample variation, or to do the same tests on a number of different lenses to provide a bit of comparison between them.

Roger

Markus Arike said:

You guys do realize that your test contradicts nearly all other MTF lens test like Lenstip and SLRGear who put the Voigtlander 25 and the Panasonic/Leica as the highest resolving lenses, with the Panasonic 20 1.7 excellent, but at most third behind the two 25mm’s. I have both the 20 1.7 and the P/L 25 1.4 and the 25 1.4 is the sharper, better lens hands down. The only thing that’s better about the 20 1.7 is the size, and it’s close-focus ability. The PL 25 is quite clearly sharper, and according to Lenstip tests, the Voigtlander is the sharpest, followed by the 25 1.4. (The Voigtlander and the Panasonic 45 2.8 Macro set resolution records on Lenstip, and I’ve seen similar results on SLRGear). I do love the 20 1.7 however. If Panasonic updates it slightly, and gives it the same focusing motor as the 25 1.4 it would be more perfect. Thanks for the test.

LensRentals Employee

Roger Cicala said:

Markus – yes, we do. That’s why I repeated it on so many copies. On the other hand, Photozone.de, whom I also greatly respect, found exactly the same thing we did.

Roger

Jeff Moore said:

As always, thanks for putting in all the work to generate these numbers. Interesting stuff, and seems to correlate reasonably well with my far-less-formal observations with the few of these lenses I happen to have used. Your testing of multiple samples certainly adds extra authority, and if you decided to present some information about the spread of your data (more formally than just the footnote on the PanaLeica 25 number), that would be really interesting, too, especially if we saw patterns by manufacturer.

While busily nominating you for extra work… well, a good part of the magic and fun of the micro-4/3s cameras is the way that with the appropriate adapter, you can bolt on nearly any lens you already have lying around. After a few decades of saving up and then squandering my lunch money, what I have a available on the shelf is mostly Leica M-mount lenses. I note you stock quite a collection of those; It’d be really fascinating to see what kind of numbers they put up in the same test as those above.

I have this suspicion (un-backed-up by proper formal testing) that the older-school symmetric designs which back up pretty close to the sensor and whose rays get pretty oblique may not get on super-well with the M43 sensors I’ve used, even moderated by the half-size sensor diagonal; but it’d be interesting to see how the Voigtländer/Cosina 12mm or 15mm get on in your center versus non-center numbers. And longer lenses like the Leica 50/1.4 ASPH or 75/2.0 ASPH (while, yes, getting a little awkwardly long, too long to match the set you test here) sure seem nice and crispy to me; it’d be great to see what that might mean in your MTF50 numbers. And the 24/2.8 ASPH, and the new floating-element 35/1.4 ASPH, and some of the Zeiss ZMs…

Well, you get the idea. I could go on and on trying to give you more work. But these would be cool things to see, I think.

(And I promise not to sulk if some of my favorite oxen end up getting gored.)

LensRentals Employee

Roger Cicala said:

Hi Jeff,

I definitely want to do that. I’ve got some more native mount m4/3 test results that should get written up this weekend. Then I hope to do that.

ginsbu said:

Roger, thanks for your detailed reply.

According to this, LensTip uses MTF50 for their resolution tests: http://www.lenstip.com/127.1-article-Why_the_Lenstip_reviews_differ_from_others_.html So it looks like that isn’t the source of the disparity. They do test at various distances and average the results. Perhaps your test distance fell at a weak spot for the PL25. (I would be surprised if the 16MP sensor is the cause since, in general, increased pixel density should result in higher MTF scores.)

I’m looking forward to your future m4/3 lens tests. I would be especially interested if you would add a test of the Panasonic DMW-GWC1 wide converter (just starting to trickle out) on the 14mm. If the converter performs decently, the prospect of an affordable 11mm option is very appealing! I imagine it would be a popular rental combination as well!

Thanks again!

LensRentals Employee

Roger Cicala said:

Distance averaging could well account for the difference. My time constraints are such that I can’t set up to test at the 4 distances we could do given our lab set up. So it’s very possible that the distance I tested at was not the ideal distance for the lens in question. I’m quite certain that was the case for the results we saw on the Nikon 105VR Micro in the D800 tests: I know the lens is better than the test results show, but can’t test it at Macro distances.

arentol said:

“Lies, Dang Lies, and Statistics”

I just want to point out that the Leica 25mm results are not quite as bad relative to the Panasonic 20mm as they first look. F4 results, max and average, then difference…
Pan-20mm = 1075 – 880 = 195
Leica-25mm = 980 – 850 = 130

With those numbers in mind, lets consider a scenario… 3 copies of lens A are tested and receive scores of 850, 820, and 820. Average = 830. Then 3 copies of lens B are tested and receive scores of 1000, 770, 750, Average = 840. Which is the better lens to purchase?

On the chart above lens A would read 850/830 while lens B would read 1000/840. So at first glance lens B looks better. However, since we know the raw data we can see that if you purchase B you are playing the “lottery”, and are probably quite likely to get a poor performing lens. Meanwhile, if you purchase A you can rely on getting at good lens, even if you don’t have a real shot at a truly ridiculous one.

A similar situation is likely happening with the Leica results. If the top outlier was removed from both lenses results the Average would likely close to about 10 points difference and the Max results would be likely be closer as well.

Presenting statistical results that tell the whole story without just providing all the raw data can be very difficult. Full time professional statisticians screw this kind of thing up all the time, let alone your average person… Not that Roger isn’t doing a great job here, just that it is truly a hard thing to do to simplify results without oversimplifying and losing some of their value.

ginsbu said:

@arentol, The max and averages Roger reports aren’t from different copies. See the comment under “Imatest Results” above.

arentol said:

True ginsbu, but the mathematical principle remains the same… If the two results are fairly close, like 880 and 850, but the center result is much higher on the 880 result then that actually means the 850 lens is more consistent across the frame, and might actually be the preferable lens to actually own.
Also because there is only one copy for all the lenses besides the Leica copy variation means that the actual average for the Panasonic 20mm could be below 850 if more were tested.
One of the massive advantages Lens Rentals has is that they are one of the few companies capable of providing results for multiple copies of each lens using the same testing process. So as much as these test results are appreciated I would actually find results from 5-10 copies of each lens more valuable to me in my renting and purchasing decisions than comparisons between lots of different, but single, lenses.

SergeyKo said:

I am just wondering why you didn’t test the lens at F/5.6 or even F/8. This would give a chance to zooms to perform better. Especially Panasonic 7-14mm.

Shawn said:

@ Jeff Moore,

You’re pretty much correct in your suspicion. I bought a handful of Zeiss and Voigtlander Leica M/screw mount lenses for my m4/3 cameras a couple years ago, all giddy that i could now use compact yet stellar rangefinder optics without the expense of buying Leica gear:

CV UW Heliar 12/5.6 (LTM)
CV Nokton 35/1.4 (M-bayonet)
Zeiss ZM C-Biogon 21/4.5 (M-bayonet)
Zeiss ZM C-Biogon 35/2.8 (M-bayonet)
CV Nokton 50/1.5 (LTM)

Long story short, they were all relatively soft in the corners except for the 50/1.5. This was much to my surprise, as i thought that the 2x crop would allow the sensor to use the central area of the image circle (thus avoiding much of the telecentricity and corner softness issues). I was wrong.

Basically, what i found was that most symmetric lenses below 50 mm had corner softness issues on m4/3. I tested a couple retrofocal wides with mixed results, with one offering superior performance:

Nikon 20/3.5 Ai was marginally better than the Zeiss ZM C-Biogon 21/4.5, but still relatively soft in the corners compared to the Panasonic 20/1.7

Zeiss ZF Distagon 25/2.8 was a pleasant surprise. It was the only non-system (adapted) lens under 50 mm i tested that was competitive with m4/3 and 4/3 system lenses in terms of corner sharpness.

Conversely, for focal lengths of 50 mm and longer, there didn’t seem to be any problems with corner softness:

CV Nokton 50/1.5 (LTM)
Samyang 85/1.4 (Nikon)
Nikon 180/2.8

Based on these results, i am guessing that the “corner softness with WA symmetric lenses” phenomenon seems to be related to the focal length vs the distance between the exit pupil and the sensor… likely an angle of incidence / telecentricity problem.

Would love to see more testing from Roger on this, and any alternative perspectives, explanations, results, and solutions as well. Hope some of you found this information useful.

Fred said:

Can anyone answer this: Is there any functional difference between the black body 14mm Panasonic stripped from a kit and the so-called silver version sold boxed and stand-alone? I just picked up one of the former, so I am very interested to know.

Mike said:

Hi Fred, yes I can help you with that.

As photons enter a so called ‘Black Body 14mm Panny” they are essentially ‘quenched’. The result is that photos taken with this lens have significantly low contrast as all photons dive towards the values of their less energetic brethren.

The silver, or ‘ethereal’ copy of this lens however is quite different. Images simply breath life, and a mere ISO 100 is often all that is needed in a darkened room to produce a bright and dynamic file.

Hope this helps.

Roger, keep up the good work..

Regards

Mike

Cameron said:

Mike, I think you’ve been spending too much time on Diablo III.

Fred, there is no difference; Mike was pulling your leg.

Troy said:

Your statement “both Olympus and Panasonic seem to do some in-camera corrections on their own lenses, but not the other brands” is counter to every photo I have ever seen of crossbrand setups. Distortion correction is applied in camera (and the data stored in the RAW file for application in post) regardless of whether it is a Panasonic/Olympus setup.

Most of these lenses (except the Panasonic/Leica models) have considerable uncorrected distortion – if bodies were truly not applying the correction, it would be quite apparent.

I think where the confusion may be coming in is that Olympus bodies do not apply chromatic aberration correction on any lens (and don’t appear to store the data for application in RAW, however LR and most others do it easily). Panasonic bodies will eliminate CA when using Panasonic lenses, but not Olympus lenses.

Interestingly, it does not appear anyone has done an objective test showing the numbers of what effect this correction has on resolution, and if there is any significant difference in how Olympus and Panasonic apply these corrections in body. I wonder who would have a bunch of bodies and lenses around and a resolution test setup …. Hmmmm … ;)

LensRentals Employee

Roger Cicala said:

Troy,

I probably didn’t write clearly, but my point was we DON’T know exactly what is corrected and what isn’t. It’s easy to see in jpgs, of course, but there are some claims correction is applied in RAW in certain cases, before it even gets to the raw converter. I don’t know if that’s true, some people that understand raw files better than I say it happens, others say it doesn’t.

Applying distortion correction in raw files is something I’m against unless it’s given as an option. Since distortion correction always, without exception, reduces resolution to some degree, I’d like the option. I don’t want the decision made for me.

Roger

Leave a Reply