Equipment

Wide-Angle Micro 4/3 Imatest Results

Published May 9, 2012

Note: The results for standard range m4/3 lenses and zooms at 25mm are in the next article.

A lot of Sony users liked seeing some Imatest results for NEX lenses, and some micro 4/3 users asked us to do the same thing. That’s a more complicated undertaking for several reasons. First, there is a much wider selection of native lenses for micro 4/3. Second because there are a lot of different micro 4/3 cameras. But we wanted to tackle the project because, quite honestly, we’d never done any lab testing on m4/3 lenses. We wanted to know, too.

Making the project something we could do in reasonable time means cutting some corners. First we can’t test every camera. We chose the Olympus OM-D EM-5 for several reasons. We wanted a camera with 16-megapixel resolution, which eliminated most m4/3 cameras. Also, the physical layout of the Olympus lets us test more quickly in our lab than we can with any of the 16-mPix Panasonic cameras. There just wasn’t enough time in one day (all we could set aside for this right now) to do two different cameras.

Second, we split the test into two sets of lenses because there were so many to test. In this part we did lenses that were 25mm or wider in focal length. We’ll do the longer lenses later. Zooms with a long range were tested near their wide end in this test, and will be tested again at longer focal lengths when we do the longer lenses.

Disclaimer

The usual “don’t read this stuff and go insane” cautions apply: this is a test of lens and sensor resolution for MTF 50 (detail resolution) done at a distance of 10 to 30 feet, depending on focal length. The results might be different at 4 feet or 400 feet. The results might be difference if we tested at MTF 10 for best acutance. Focus is done manually and bracketed so it’s not a test of a camera’s autofocus ability. The tests use a controlled-lighting test-target with a hardware mounted camera at ISO 400, so photos taken in the dark of night or hand-held for 12 second exposures will give slightly different results. It doesn’t measure autofocus speed, lens size, manual focus feel, bokeh, color rendition or anything like that. It’s not a lens review; it’s one simple test of resolution.

More importantly, remember that while we are testing RAW files, both Olympus and Panasonic seem to do some in-camera corrections on their own lenses, but not the other brands. We don’t know what effects, if any, that might have on this test. Possibly Panasonic lenses might do a little better on Panasonic cameras and Olympus lenses a little worse. We’ll figure that out later, but I doubt any such corrections will affect the MTF 50 at all. If it did the effects should be minor and limited to the edges and corners. But we don’t know for sure. Yet.

Addendum 4/13/12: I had hoped that the m4/3 community might not be quite as Fanboy riddled as Canon and Nikon. Obviously I was wrong about that. Because there was such an uproar about the results of the Panasonic / Leica 25mm I’ve repeated the tests on 6 copies (all that we have) and also on Panasonic cameras, because Fanboys have been on suicide watch and full-attack-mode since they’ve taken them so out-of-context. The average of 6 lenses on both Olympus and Panasonic 16 Mpix cameras are now reported. The results are basically unchanged.

The Lenses

We had a nice selection of wide-angle m4/3 lenses in stock on the day we conducted these tests:

Imatest Results

The values are for MTF-50 (which correlates with fine detail resolution) measured in line pairs / image height. The first (higher) number is the MTF 50 measured at the center of the lens only. The second (lower) number is the average of the MTF-50 measured at 13 points including the center, 4 corners, 4 sides, and 4 mid points. I keep a point total for every time someone posts the question “what do the two numbers mean” as an indicator of the reading comprehension for each group of camera users. I have high expectations for m4/3 users, so please don’t let me down.

I’ll also mention that numbers sometimes make minor differences seem really large. A difference less than 50 lp/ih is probably not noticeable in real-world photography. So if you print a reasonably sized image from a lens resolving 860/750 lp/ih and compared it to a shot with another resolving 810/700 lp/ih the difference would probably not be noticeable. You would almost certainly notice a difference of 100 lp/ih.

At Widest Aperture

The first graph shows the numbers for each of our lenses (zooms measured at 30mm) with the lens aperture wide open. This test isn’t a level playing field, since some lenses are f/4 wide open while the primes are at anything from f/1.4 to f/3.5. But since some people tend to shoot every lens wide open, it may be useful to know how well (or not) the lenses do at their widest aperture. I’ve listed them from widest to longest focal length. The zooms were all shot at 14mm so they might be a bit better or worse at other focal lengths.

MTF 50 at Widest Aperture

Lens Max Avg
Olympus 12mm f/2.0 860 730
Panasonic 14mm f/2.5 950 785
Panasonic 7-14 f/4 965 765
Olympus 9-18 f/5 805 680
Olympus 12-50 f/3.5 825 680
Olympus 14-42 II f/3.5 720 600
Olympus 17mm f/2.8 720 590
Voigtlander 17.5mm f/0.95 565 475
Panasonic 20mm f/1.7 870 735
Voigtlander 25mm mm f/0.95 530 435
Panasonic/Leica 25mm f/1.4* 690 590

*Panasonic-Leica results on this test are the average of 6 copies. The average numbers are slightly better at f/1.4 than  the first version of this article, which reported only one copy, but the difference is not really significant.

There’s not a lot to see from this set of numbers. As usual, good primes are as sharp, or sharper, at wide apertures than consumer-grade zooms are at lower apertures. The Panasonic pancake lenses pleasantly surprised me. Pancakes are usually not the greatest lenses, but he Panasonic 14mm and 20mm are really quite good.

Stopped Down to f/2.8 or Smaller

We can level the playing field a little bit by shooting the wide primes at f/2.8, as shown in the graph below. It’s somewhat more meaningful: the zooms are still at smaller apertures, but everything else is shot at f/2.8 here.

MTF 50 at f/2.8

Lens Max Avg
Olympus 12mm f/2.8 1000 845
Panasonic 14mm f/2.8 1010 850
Panasonic 7-14 f/4 965 765
Olympus 9-18 f/5 805 680
Olympus 12-50 f/3.5 825 680
Olympus 14-42 II  f/3.5 720 600
Olympus 17mm f/2.8 720 590
Voigtlander 17.5mm f/2.8 965 810
Panasonic 20mm f/2.8 1050 875
Voigtlander 25mm mm f/2.8 995 855
Panasonic/Leica 25mm f/2.8* 960 820

*Panasonic-Leica results on this test are the average of 6 copies. The average numbers at f/2.8 are slightly better than  the first version of this article, which reported only one copy, but the difference is not really significant.

By f/2.8 the superiority of most of the primes over most of the zooms becomes pretty apparent. The Olympus 17mm f/2.8 is easily the weakest of the prime lenses, which isn’t really surprising. It’s an older design and a consumer-grade pancake lens. The two Panasonic pancakes being so good surprised me, though. They’re not much more expensive than the Olympus 17mm. I had expected a bit more out of the Panasonic Leica 25mm given it’s cost and build quality. It has other good qualities, this is simply the results for MTF 50. The Voigtlander lenses, despite being built for ultra-wide aperture (which usually results in a rather soft lens) did quite well. But as easily the most expensive lenses in the group, that seems appropriate.

Personally my biggest disappointment was in the Panasonic 7-14 f/4 zoom, which is probably my favorite m4/3 lens. But we were testing at 14mm, and that lens is actually a bit better at the wide end. And really, it’s probably a bit much to ask a lens to be this incredibly wide, and also incredibly sharp. The Nikon 14-24 pulls it off for about $2,000, but that’s the only one I can think of.

MTF 50 at f/4

We can level things out even further by shooting all of the lenses at f/4. Most of these lenses would get a bit sharper still at f/5.6, especially in the average (as opposed to center) resolution.

MTF 50 at f/4

Lens Max Avg
Olympus 12mm f/4 1040 870
Panasonic 14mm f/4 1025 860
Panasonic 7-14 f/4 965 765
Olympus 9-18 f/5 805 680
Olympus 12-50 f/4 830 685
Olympus 14-42 II f/4 730 605
Olympus 17mm f/4 735 610
Voigtlander 17.5mm f/4 1070 835
Panasonic 20mm f/4 1075 880
Voigtlander 25mm mm f/4 1030 925
Panasonic/Leica 25mm f/4* 980 850

*Panasonic-Leica results on this test are the average of 6 copies. The average numbers at f/4 are slightly lower than  the first version of this article, which reported only one copy, but the difference is not really significant.

There’s really not much new information here that we didn’t already see at f/2.8. The Olympus 9-18 and 12-50 zooms are clearly better than the little 14-42 zoom. The Panasonic 7-14, despite working at such wide angles, is better than either of the others. Overall another “you get what you pay for” situation – slapping  II on the name doesn’t make it a great lens. (BTW – although I expect m4/3 shooters to go a little less Fanboy than the Canon and Nikon guys, let’s get this out of the way: some Fanboy always says “the rental guy wants you to rent the most expensive lens”.  Actually the rental guy could care less: the profit margin is the same on a really expensive lens and a cheap one.)

Of the prime lenses, only the Oly 17 pancake isn’t very good; the Panasonic pancakes were clearly better. The Panasonic/Leica 25mm disappointed a bit: it was good but I had expected it to be THE best of the bunch and it’s not. The Olympus 12mm was just excellent, though, and the Voigtlanders provide an amazing wide aperture lens.

Part 1 Summary

Compared to the NEX system lenses I just tested, m4/3 users have a host of wide-angle, native-mount choices. There are good pancakes, standard primes, interesting (and expensive) ultra-wide aperture primes, all at reasonable prices. There’s a good f/4 ultrawide zoom and a couple of reasonably sharp zoom lenses. The big hole, as I see it, is that there’s no wide-aperture zoom, period. For my shooting style, the m4/3 begs for a wide aperture.

Remember, we’re just talking about resolution here. When I’m choosing a lens, it’s my starting point: I require good resolution. But that’s all it is, a starting point. If you’re seriously considering some of these lenses, go to a good review site where you can get more in-depth information about distortion, handling, focusing accuracy, etc.

In this part we’ve looked at most, but not all, of the m4/3 lens choices for shooting at 25mm or wider. We’ve only tested on the Olympus OM-D. We’ll do some testing on at least one of the Panasonic 16mm cameras just to make certain there’s no significant difference soon (I doubt there will be, but it’s possible). We’ll also do a second post on the longer lenses in the next couple of weeks.

Roger Cicala

Lensrentals.com

May 2012

Author: Roger Cicala

I’m Roger and I am the founder of Lensrentals.com. Hailed as one of the optic nerds here, I enjoy shooting collimated light through 30X microscope objectives in my spare time. When I do take real pictures I like using something different: a Medium format, or Pentax K1, or a Sony RX1R.

Posted in Equipment
  • Lester

    Love your tests! I was very pleased with my Panny 7-14, I thought it might do a little better than it did in your tests. The PL 25 should certainly have done better, awaiting your confirmation of a poor copy. And my copy of the Panny 20 was poor and so never showed what you found, rats… My Oly 12-50 does exactly what you saw, as does my Panny version of the 14-42. I dismissed and sold my Panny 14-45 too quickly, I now regret it, look forward to you testing it.

  • Roger Cicala

    LJC,

    Boy, do I wish it were free. They’re working on the IE 8 fixes, but since all the other browsers (correct me if I’m wrong, I don’t do this kind of stuff, so I’m repeating what I’m told) use certain protocols that IE 8 doesn’t they developed it in the other formats and then are doing the IE 8 fixes. I’m sure IE 8 is the single biggest, but the others together are much larger. I’m told IE 9, though, works fine.

    The fixes are largely done on the website, but the blog, understandably, had back burner until the website was fixed.

    Roger

  • Roger Cicala

    John,

    All lenses were, of course, measured in exactly the same way with best results from focus bracketing reported for each. We had a second copy to confirm results with all except the Pan/Leica, as I said.

    Roger

  • Lee

    Wow, lot of ill tempers today.

  • john

    Roger,

    Yes, I did read the article. Indeed, I noticed your randomly-spread-throughout-the-text guesses trying to explain why the pannyleica performed “poorly”. But if one of your lenses is not measured in the same conditions as the others, you should EXPLICIT say that in every table this “little” issue, for instance using an asterisk on the lens name in every table and its correspondent legend.

    You just had one copy of the pannyleica. How many copies of the panny 20mm f/1.7 you used? And how many times you repeated the test for each of the above lenses?

  • ginsbu

    Thanks for doing this test. One thing that would make the numbers more meaningful to me would be to report a minimum measurement (or an average of just the corners) in addition to the maximum and average.

  • David Bateman

    Excellent work and great test. I will look forward to follow up test with GH2 (hopefully). Also I would like to see the >25mm lenses when you post it. Thinks that are way too much work for you, but I would like to see are the zooms at multiple points rather than just 14mm. To see how they fair at min, max and middle. Also hopefully you can add results with Sigma 30mm, that was the best in your Nex test and I would like to see the numbers on the EM5 for comparison. I mean are they the same (1040/840 @f4) to then add possibility to Compare across systems.

    I did like that you didn’t add a paragraph as you did in the NEX7 test to compare across D800, NEX7 ete. People will do it anyways.

  • LJC

    Yes I’m on IE8. It’s the most popular browser version currently in use (both worldwide and in USA according to pingdom and statcounter) so tbh you should have “words” with your software house. (

    imo) New websites, especially those used to generate income, should be developed on IE8 in the first place then check it’s compatibility with other browsers and versions so that you best reduce the risk of irritating customers. But I seem to be in the minority there..

    Unless this was a freebie, in which case you get what you pay for ;p

    Signed,
    A. Developer

  • Interesting test, thanks.

  • PDC

    A thorough and very helpful review. I have the Panasonic 20mm, 7-14mm and both of the Noktons (17.5 and 25). My personal assessment of these puts them in a similar rank, thought the 7-14 should be considered a special purpose architectural lens (and it does a very good job). The Noktons are amazing, although I recently tested the 17.5 for chromatic abberation and purple fringing is definitely an issue wider than f2.
    See:
    http://www.drivehq.com/file/df.aspx/publish/pdcsylva/JPEGsCV17TestingPDC
    or http://www.drivehq.com/folder/p9818891.aspx
    for a series that tests Voigtlander Nokton 17.mm axial chromatic abberation (purple fringing artefact) in weak daytime backlight conditions.
    #23, #32 F0.95
    #24, #33 F1.4
    #25, #34 F2.0
    #26, #35 F2.8
    #27, #36 F4.0
    #28, #37 F5.6
    #29, #38 F8
    #30, #39 F11
    #31, #40 F16
    All processed through Lightroom 4.1 from RAW (Panasonic GH2)
    For -2 series Lightroom Manual Lens Defringe AllEdges applied
    For -3 series Defringe plus “Direct Positive” tone preset applied
    Conclusions: Apertures wider than F2 will produce significant purple fringing in fairly weak backlighting. Lightroom Defringe marginally effective in removal.
    Recommendation: Avoid backlit scenes if shooting wide open. Stopped down to F2.8 or smaller will avoid purple fringing.

  • wow.

    Well done mate! thanx so much for another good source of info on the net, i was taken by the 25mm relative dissapointing performance, it was said to be as sharp as 20mm,

    I hope that the extra $$ in 25mm would bring a better rendering and bokeh, but i have yet saw a good comparison between the two.

  • Can you test the new pz 14-42 pancake lens?

    Given Panasonics claim that it is better than the 14-45 and several user tests that comfirm this, I would find this very interesting.

  • jim thomson

    Great set of tests.
    Biggest surprise was the Panasonic 14mm. This lens usually gets a negative comparison to the 20mm. Looks like it held its own in this set of test.
    I’m curious to see how the Sigma 19mm stacks up given how well the 30mm did on your NEX tests.
    It would also be nice to see how the Panasonic 14-42 kit compares to the Olympus.

  • Richard B

    Thanks for the very interesting test. Do you plan to test the 45mm f1.8? The few people I have spoken with who use it seem to like it, indeed they seem to think the 12mm/25mm/45mm combo covers a great deal of their needs.

  • Nilesh

    Thanks for the tests Roger…helps me decide which system to for mirrorless. I wish there was something like Panasonic 20 f1.7 at at that price fpr NEX.

    Hate to say this but don’t care much for the new blog layout! font too small(I know I can change, but why do just for your site?), comments and content crowded etc.

  • Roger Cicala

    John,
    Do you mean other than the explanation in the text about only having one copy to test and being concerned it may be a bad copy, planning on retesting as soon as other copies get back, asking you to take the results with a grain of salt until then, and mentioning that it might test differently on a different camera? Do you want other explanations than the ones I’ve given? Well, I guess testing on 16 megapixel cameras instead of 12 might be it, or perhaps testin on an Olympus body. But I think the explanations I already gave in the article are more likely. Oh, wait: you didn’t read the article did you? Just teed up and went Fanboy?

  • Esa Tuunanen

    Roger, how about testing Zuiko 12-60mm for comparison before someone thinks m4/3 has native lenses boasting with optical quality?
    Makes supposedly premium prime of 12mm look not so high quality.
    http://pekkapotka.squarespace.com/journal/2011/12/30/olympus-mzuiko-12-50mm-f35-63-in-comparison.html

    Tom, if you think you’ve got lots of light in desert try fully snowy landscape, that’s why E-M5’s base ISO of 200 was slight disappointment for me.
    But f/8 is good compromise between small aperture and avoiding bigger diffraction softening.
    With pixel size of 4/3 sensors (and 20+MP APS-C) Airy disc starts to grow larger than pixel little before that but impact starts to really grow after f/8.
    Photozone has easy to read reviews where effect can be seen:
    http://www.photozone.de/Reviews/overview#fourthirds
    (lots of reviews for most systems)

  • john

    Regarding panny 20mm f/1.7 and pannyleica 25mm f/1.4, care to explain why you got such results totally in contrast to those of slrgear (and many others tests)? There one can see the 25mm f/1.4 clearly shows much less blur units than the 20mm f/1.7 at ALL apertures.

    You’re in fact the only one who says the 20mm f/1.7 is sharper than the leica 25mm!?

  • Sylvain

    Roger, thanks for the nice effort.
    Just a little down, I think your result would gain much being put in a simple chart.

    greetings,
    S.

  • Roger Cicala

    Good questions Dave. We don’t know if the IS has any real effect, but shutter speeds are fast given the good lighting and we haven’t seen any effect on or off with other setups.

    We use ISO 400 just because it seems a practical test point where things are pretty even. No camera is at it’s base, but none are struggling with noise, either. The decision was arbitrary but once we started there, we keep going: it’s where our database is shot at and it’s a big database.

    We focus bracket every shot picking the best shot. Probably AF on this camera would be just as good, but it wouldn’t be better. It just eliminates a variable.

    Roger

  • Tom

    Hi Roger,
    Great test, thank you. I’m feeling very chuffed, as I just added the 14/2.5 to the 20/1.7 and 45/1.8. I researched this lens for ages (I’m “careful” with my money), and felt that it was at least equal to the 12/2 for resolution. Add in the size and cost, it was a no-brainer. Now you’ve made me feel very clever….
    When shooting landscapes with the 20/1.7, I’ve typically stopped down to f8. I use an E-PL1, so often need to go that small to get the shutter speed low enough anyway (I live in the Middle East desert & New Zealand mountains). Do you think I’d get better results a little wider with an ND filter?

  • Roger Cicala

    Not overlooked the Sigmas, just didn’t have any to test that day. We’ve got lots more to test, but each lens is an hour or two of work, and the equipment’s primary use is testing returning lenses so we have to do these when workload allows 🙂

    I plan on getting to all of the lenses. First priority was to repeat a few on a Panasonic body to make sure there weren’t any major differences between cameras, then hopefully I’ll get another batch done next week.

    Roger

  • Sandeep

    You have overlooked the new Sigma lenses, the 19mm f2.8 and the 30mm f2.8. Looking at the MTF of the Sigma 19mm it looks like it has better edge performance than the Panasonic 20mm.

  • Richard

    This test shows pretty much exactly what I saw with a few lenses. The Voigtlander is lame and overpriced and arguably, you can produce a better underexposed and luminance raised image with the 20mm than it. Meaning, the Voiglander’s speed is only good for defocusing the background. The 14mm also bears out what I experienced, even though I had read tests that said it wasn’t that good. Maybe there are variations in its quality?

  • Dave

    Very informative, and as you say, a good starting point. I’ m curious about why you use Iso 400 and not the camera’s base iso, not that it would change the relative results any. Also curious if image stabilization would have an effect (my understanding is you can’t turn it off on the em-5, though the usual recommendations with other systems is to turn IS off when on tripod). This might make a difference depending on shutter speed, but again wouldn’t affect the relative ratings as you have controlled for exposure. Just curious.
    Lastly, with a cdaf sensor, why not use Af instead of manual for the tests? Doesn’t this remove a variable (user technique for MF)?

  • Roger Cicala

    LJC – are you in IE 8? I know we’ve changed formatting and it’s causing all kinds of IE 8 problems, which they’re working on now. Like most new site rollouts, we’re doing about 100 bug fixes a day, but finding about 150. Thanks for the comments, it helps know what kind of problems people are seeing.

  • Roger Cicala

    hi Rich,

    I absolutely will — the only reason it isn’t on this first set is it was out of stock. Which makes sense since it probably is the best mid-range zoom.

    Roger

  • Hi there,
    Fantastic test – thank you so much for the effort.

    The only lens missing I think would be really interesting to include is the Panasonic 14-45mm – Its currently the sharpest mid-range/standard zoom available (until hopefully the 12-35 X lens will beat!)

    If you get time to add the 14-45mm I (and many others) would really appreciate it.

  • LJC

    Amendment to previous message: It’s stripping out blank lines or duplicate line feeds which is causing the wall of text problem (URLs same colour as background kinda unforgiveable though ;p)

  • LJC

    Like the new site; hate the new blog! (Not the content of course, but the layout)

    It looks like you’ve lost a lot of your formatting that aided readability (Is that a word?). Or perhaps its just interpretating line feeds differently than the last version resulting in a wall-o-text.

    On the plus side, it looks as though you’ve addressed the wrapping issue that was preventing me from reading the blog earlier in the week.

    Also, currently on my screen below this text box I have a big blue banner containing URL links on the left hand side, however the URLs are the same colour as the background so all I can see is the headers (In order from top to bottom; Copyright notice, address, Navigation, 8 invisible links that I can see by using select-all, Navigation (again), 3 invisible lines, Support, invisible, Follow Us then 2 invisible URLs).

    I would assume that this is version 2.0 and a nice shiny 2.1 is on the cards after you’ve paid some lackey to perform some useability testing, but I’m way too cynical ;p

Follow on Feedly