Equipment

MTF Tests for the Sigma 14-24mm f2.8 Art Series Lens

Most of you know I’m a big fan of Sigma’s Art prime lenses. They give superior optics at excellent prices just about every damn time. I’ve been more mixed in my opinions of Sigma’s Art zoom lenses. I thought the Sigma 24-105mm f/4 Art was the best 24-105mm I’ve ever tested (and I hate testing 24-105s). The Sigma 24-70mm f/2.8 Art, on the other hand, I thought was adequate.

Sigma 14-24mm Art Series

I wasn’t sure what to expect from the Sigma 14-24mm f/2.8 DG HSM Art, so I was eager to test it and grabbed copies as soon as we had them in stock. Why then, you may logically ask, am I just now writing it up? Well, partly because I have jobs that like, you know, pay me to do stuff. And somewhat because this became an exciting learning experience that I’ll tell you about at the end of the post.

About the Lens

Years ago, Nikon came out with their Nikon 14-24mm f2.8G ED lens. It was a revolution; wider and sharper than any other wide zoom made at the time. It remained the gold standard of wide zooms for years. Sigma isn’t filling a void here; Nikon already has an excellent 14-24mm, and Tamron has the not-quite-as-wide but really close 15-30mm f/2.8 Di VC USD.

The Sigma is less expensive than the Nikon by $400, but $200 more expensive than the Tamron 15-30mm f/2.8 Di VC USD. The Sigma, at 1150 grams is a bit heavier than the Nikon (1000g) and about the same as the Tamron (1100 gm).

So I thought that the new Sigma should be as good as the Nikon 14-24mm and better than the Tamron 15-30. That’s a tough task. Yes, I’m aware someone is going to make noise about other lenses. First, f/4 is not f/2.8. Second, 16mm is not 14mm, they are very different beasts (not to mention, most 16-35mm zooms are actually 17mm at the wide end).

One Disclaimer

Of course, the usual ‘this is not a lens review, it’s an optical test’ stuff applies to this post. One other thing needs to be mentioned when we test full-frame f/2.8 lenses wider than 16mm; we are either right at, or possibly past, the limits of our machine. It’s complex, it’s arguable, but it deserves mention that you should probably consider the graphs in this post ‘close, but might not be exact’ at the outer edges.

Also, I’ve violated my usual testing protocol for the first time, and I think you should know that. The test I’m showing you today is my second set of 10 copies of the lens, not the first 10. If you’re interested in why you can read about it in the addendum.

Optical Testing

As is our practice with 2x zooms, these lenses were tested at both ends (we do middle focal length testing on 3x and longer zooms). To start with, let’s mention the actual focal length of the lens is 14.3mm to 23mm. I mention this because that’s less ‘fudging’ than the label usually has; +/-5% is pretty standard for a zoom. Distortion is also reasonable: 0.8% at 14mm and 1.1% at 24mm. At both ends the distortion is mildly mustache shaped – a bit like the Zeiss 21mm f/2.8.

MTF of the Sigma 14-24mm f/2.8 DG HSM Art

At 14mm

This is an excellent resolution. About halfway to the edge, we start to see some astigmatism-like separation of the sagittal and tangential lines (it could also be the lateral chromatic aberration).

As I mentioned above, 14mm full-frame lenses are right at the limits of our machines automation. The drop in tangential MTF you probably note at 4mm away from the axis is perhaps a testing artifact; I won’t bore you with details about distortion being mapped at insufficient points for a very wide-angle lens. It could also be caused by an aspheric element. I can’t say for sure. 

Olaf Optical Testing, 2018

At 24mm

The resolution is even better at 24mm, just excellent across the field, and with no testing artifacts.

Olaf Optical Testing, 2018

Stop-Down MTF

The above graphs are the average of all ten copies tested at the widest aperture. We took one typical copy and stopped it down for retesting. There is a marked increase in resolution at f/4 compared to f/2.8. There was very little change between f/4 and f5.6 so I’m not going to clog things up with those graphs.

14mm

Olaf Optical Testing, 2018

 

24mm

You may notice that the MTF graph looks a little worse at the edges at f/4. This is a complicated thing that involves distortion, the lens’ field curvature, and a change in focus point of the machine, not a real thing.

 

Olaf Optical Testing, 2018

MTF Comparisons

Nikon 14-24mm f/2.8G ED

We’ll start with the comparison to the long-term gold standard, the Nikon 14-24 f/2.8G ED. The Nikon is a much older design, of course, but still a superb lens.

At 14mm

Here is an excellent example of two great lenses that are different, not one-is-better. Both are excellent in the center. In the outer 1/3, the Nikon has less sagittal-tangential separation; the Sigma has higher resolution. Will that make a difference in your image? With careful comparison, you might notice the Sigma has better corner resolution, but perhaps busier bokeh. You might not notice any difference at all.

Olaf Optical Testing, 2018

At 24mm

At 24mm the Sigma is strutting its stuff. It’s sharper everywhere. This difference should be noticeable, for sure.

Olaf Optical Testing, 2018

Tamron 15-30mm f/2.8 Vi DC USD

At 14 vs. 15mm

The Tamron is an excellent lens at the wide end, very comparable to the Nikon. As with the Sigma-Nikon comparison, I won’t try to call better (especially given that we’re testing at the limits of our bench). The Sigma may be a bit better in the center; the Tamron maybe a bit better at the edges. Again, though, we’re straining the limits of our testing capabilities here, so I can’t say there’s any noticeable difference.

Olaf Optical Testing, 2018

AT 24mm vs. 30mm 

Again, it’s not close at the longer end; the Sigma is clearly better here.

Olaf Optical Testing, 2018

If you want a really good example of just how hard it is to do 14mm, here’s a comparison of the Sigma 14-24mm f/2.8 with the Canon 14mm f/2.8 prime. Depending on what you emphasize, the Sigma comes close to violating Roger’s Prime Directive: No zoom is as good as a prime. In this case, the zoom may be better.

Olaf Optical Testing, 2018

The Canon is not the best 14mm prime; I chose it because f/2.8 to f/2.8 makes a nice comparison. The Sigma 14mm f1.8 Art is the best 14mm prime. At f2.8 is better than any of the others. So Roger’s Prime Directive holds true.

Olaf Optical Testing, 2018.

 Field Curvature

Let’s leave the fanboys behind to argue about the comparison MTFs and get into something useful for photographers who actually use the lens; the field curvature.

14mm

The sagittal field is a gentle M curve (lower left) while the tangential field (lower right) is a strong W. This gives the overall field (upper left) a bit of a U shape, and more importantly, means there’s going to be some astigmatism at the edges. It’s a 14mm lens, after all.

Olaf Optical Testing, 2018

The strong field curvature affects the standard MTF curve (it is focused in the center, so the edge MTF is low partly because it’s out of focus). We can pull the data above to give Best Individual Point MTF (BIF). This is a theoretic measurement, showing how much resolution you can get if you focus on that specific point rather than the center.

Below is the center focus MTF on the left, the BIF MTF on the right. As you can see, if you focus off axis, you can get near-center sharpness just past halfway to the edge. Beyond that, you can get good sharpness, but you will have some astigmatism.

Olaf Optical Testing, 2018

As I mentioned above, if you focus off axis, the center of the image will get softer, because of field curvature. Since 14mm is used for landscapes at times, it can be useful to find the best focus point – the place that if you focus there, you get the best overall sharpness from corner-to-corner. Below is a graph showing where we calculate that point should be, and what the MTF would look like if you focused there.

Olaf Optical Testing, 2016

So, if you want to get the best edge-to-edge sharpness on your landscape shot, focus a bit less than halfway to the edge of the image; that will give you the best average sharpness across the frame.

24mm

At 24mm the tangential field is still slightly W shaped, while the sagittal is a shallow U shape. The important thing is they are more similar to each other, and the overall field is a gentle U with less astigmatism than at 14mm. This is a really nice performance for any lens (even a prime) at 24mm.

Olaf Optical Testing, 2018

When we compare the center focus MTF (below left) with the Best Individually Focused MTF (below right) we see that at 24mm you can get excellent sharpness throughout most of the field; even at the edges, things are very good.

Olaf Optical Testing, 2018

If we find the focusing point that gives our best overall focus, once again it’s at 8mm from the center, just about halfway to the edge of the image. That’s convenient; the Best Average Focus point is at the same place at either end of the zoom. Less to remember.

Olaf Optical Testing, 2016

Conclusion

I said at the beginning that I expected this lens to be at least as good as the Nikon 14-24mm f/2.8 and better than the Tamron 15-30mm f/2.8. I think it’s pretty apparent that while they are all within hair splitting distance of each other at 14mm, the Sigma is the best at 24mm. Optically, this one is worth the money and the best overall 14-24mm zoom. There are lots of other things that go into choosing a lens, but at these very wide angles, things like autofocus speed are not big points.

That being said, if you’re buying a 14mm lens to be a 14mm prime (and some of you are), then all three of these do very well. Honestly, though, for a little more money you should consider the Sigma 14mm f/1.8 Art if that’s the case. If you’re thinking of matching a 14-24mm f/2.8 with a 24-70mm f/2.8 to collect the whole focal length set, then the Sigma is the best optical choice.

Two things I’ll mention. First, for the 72,534 of you that want to compare it to 16-35s, go right ahead. But it’s not a fair comparison, and I’m not going to do it. 14mm is very different than 16-35; much more difficult to design and manufacture. If a 16-35mm will meet your needs, and hold your filters without additional contraptions, then get a 16-35mm.

Roger Cicala and Aaron Closz

Lensrentals.com

July, 2018

Addendum: Sigma’s Response

Our protocol is always the same: the intake people inspect and test lenses on test charts, fail anything that’s obviously bad and pass the rest on for testing. I test the first ten lenses they send me to write the blog post.

This time a weird thing happened. On testing day 1, I tested the first eight copies of the Sigma. At 24mm five of them were identical and excellent. Three were not OK and was a big gap between the two groups. This is NOT normal. Generally, we see a cluster of lenses that are good, a lens that’s not quite that good, another not quite as good as that second lens, etc. In other words, the variance is always a steady trend away from best.

This is not something I’ve ever seen before in testing hundreds of lenses. I have a good working relationship with Sigma, so I emailed them as a courtesy, as I do with any manufacturer that corresponds with me. (They generally ask me to retest and want to argue about why my results aren’t valid. I generally retest and find my results are valid.)

Sigma didn’t do this. Instead, they asked for the serial numbers of the lenses. A few hours later they told me they had pulled the tests of those lenses at the end of the assembly, that the three lenses in question had tested much better than this at the factory, and asked me to send them to Japan for repeat testing.

Most of you don’t realize the enormous shock this caused me. I’ve been testing lenses for a decade and working with manufacturers for nearly that long. Leica has this kind of testing for every lens. Zeiss has it for certain lenses in their lineup. Nobody else does. Leica lenses. Some Zeiss lenses. Sigma Art lenses. Yes, I know all about other claims of ‘we test every lens’; that means it was mounted to a camera to program the AF firmware. Nobody else (with the possible exception of Canon with some of their newest lenses, and I’m not certain of that) is testing every copy optically and keeping the results on file.

Ten days later (shipping to Japan takes a while), Sigma emailed to tell me the lenses I sent back tested much worse than their original end-of assembly tests. Their new tests agreed with my findings and were, in their words ‘not acceptable for this lens.’ I’ve heard ‘this lens is within spec’ from manufacturers on lenses I knew were bad so often, for so long, that I reread the email just to be sure. To paraphrase a bit: “We have retested the lenses and agree with your findings. They aren’t good now. They were quite good when they left here. We’re investigating to find out what happened.”

Then they had their engineers disassemble the lenses and found some minor damage to rollers consistent with an impact that had caused the problem. I’ll be blunt, usually, I’d expect a ‘so you guys must have dropped them’ next, because these came from 3 different suppliers. And that is certainly possible. The lenses passed through several hands here on their way to me, and while very rare, things get dropped (and that people dropping don’t always admit they’ve dropped).

Instead, Sigma said they would track the shipment and see if the lenses were in the same crate or pallet during transit. Then they told me something that made me nearly weep with gratitude, showing that in a few companies engineering, not marketing, still runs the show:

Our engineers have also taken a number of new lenses from the assembly line, dropped them in various ways, and then disassembling them to look for damage. We did this until we had reproduced this type of damage. We will now see if there are changes we can make in the lenses or packaging to prevent this from happening in the future.

This is, in my opinion, what superb engineering and quality assurance are all about; look for any weaknesses and strengthen them. I’ve been working in this industry for many years, and this is the attitude I wish every company had, but few do.

Also, to be as certain as I can that the tests I’ve presented here are representative, I’ve tested a total of 25 copies now; the three damaged lenses were the only ones with any issues.

 

OFFTOPIC – And a bit of fun, for both of you that read this far.

I LOVE http://splasho.com/upgoer5/ where you have to write using only the 1,000 words most commonly used in American English. So, I rewrote a couple of paragraphs until they passed the test. If you want to try your hand at it, the two original paragraphs are under “14mm field curvature”. Post your results and shame me with how poorly I did.

The way the field moves changes how close to perfect the MTF (which is focused in the middle) is. We can pull the numbers above to give Best-at-each-point MTF. This is not a real world thing, because if you focus on that point, rather than the middle, the middle gets soft.

Here is the middle focus MTF on the left, and the Best-at-each-point MTF on the right. As you can see, if you focus away from the middle, you can get as good as you would see in the middle, until you go past half of the way to the edge. If you go further you can get good pictures, but the lines going away from the middle won’t be the same as the lines going across those lines going away from the middle.

Author: Roger Cicala

I’m Roger and I am the founder of Lensrentals.com. Hailed as one of the optic nerds here, I enjoy shooting collimated light through 30X microscope objectives in my spare time. When I do take real pictures I like using something different: a Medium format, or Pentax K1, or a Sony RX1R.

Posted in Equipment
  • Arlington Brian

    No reason to make people do that, they can just borrow a cat. They love knocking expensive stuff off of tables.

  • suk it!

    lets hope sigma can get any issues with their af under control. so there is no doubt. I know the vast majority dont have issues with them. but many oem fanboys go online like to bh/adorama and write negative reviews when they arent verified buyers has been inflated. on the internet anyone can be anyone and can say what they want and no one can verify it.

  • As I mentioned, it is possible they were dropped here. Also ‘local distributor’ for us generally means from New York or California, so a couple of warehouses, several trucks, and a journey of a 1,000 miles or more.

  • Ruy Penalva

    But that lens come from local distributor, what kind of shock could they have ben taken? I did not doubt that drops could impair a huge amount of lens. But seems to me a little fantasy to presume a shock in the original package. Do they have the same percentage of impairment? In other words, the three lenses behave uniformly under the curve? Sorry to not see in details your graphics that are becoming every day more hard to swallow and digest. My bet is this is not a kind of marketing from Sigma and Lens Rental?

  • Ruy, not sure why you think that. We find drops are the most common cause of all lenses in our fleet becoming optically impaired.

  • Thr “Or” is perfect. I’m stealing that.

  • At two years old, my Sigma 20mm ƒ/1.8 had the iris implode, locking the aperture lever, and breaking the aperture arm in my D2x. Sigma fixed the iris. It broke again. They fixed it again. It came back to me much softer–something was out of alignment. They said to send it back with some pictures showing the problem. I sent it back with some big prints of shots before and after. They recombobled it and brought it back to its former glory. No charge other than the first time shipping it to them.

    That was ten years ago. Still works great!

    I only wish Nikon USA service was so good!

  • I really enjoy my little Sigma 60mm F2.8, and I’ve been saying for awhile that if Sigma releases a m43 60mm F1.4 I’ll pre-order it. Engineering and support like this are exactly what I want to support. Dear sigma: release an m43 foveon body and I’ll pre-order that too.

    I’m happy to support companies like this with my wallet.

    Roger, thank you for sharing the story.

  • Ruy Penalva

    This smells a little bit fantasy from Sigma. What king of shock could made this in this lens? Fabrication defect.

  • bdbender4

    When is a line not a line? When it is not straight. When is a not straight line not a not straight line? When it is a whole family of not straight lines. (When a whole family of not straight lines are happy in the middle, they are not so happy at the side.) If you want a happy family of not straight lines, you can say they are happy at another place you pull out of your ear. You can look and see how happy the family of not straight lines are at the out of your ear place. But don’t believe it. It is not real.

    Down on the left is the very happiest family of not straight lines from the middle. The families of happy not straight lines from the out of your ear place are on the right. As you can see, if you play around enough you can make families of very happiest not straight lines away from the middle just past half way to the side. If you go further toward the side, the families of not straight lines get sad. Not so happy.

    (So the deal is: you can have a few not straight lines in the middle that are really, really happy. But then the other not straight lines are
    kind of sad. Or, you can make a lot of the other not straight lines kind of happy. Not really happy, but kind of happy. But then the not straight lines in the middle get sadder.

    Or: “You can make some of the not straight lines happy some of the time, but you can not make all of the not straight lines happy all of the time.”)

  • Maxim Podtopelny

    I found myself crying at the end of addendum. I read this blog for many years and this was definitely the cutest thing I’ve ever read!

  • Adam Palmer

    More love for Sigma. I was sold after I got the 50mm 1.4 way back when. I know have sigmas from 20mm to 400mm. They sure know how to make a great lens at a reasonable price.

  • Love it.

  • Ilya Zakharevich

    Look whether the picture goes along a straight line, or goes away from the (straight!) part which reads and stores the picture. The worse this is, the worse is the “how-clear-at-each-point” drawing (this drawing shows how clear is the picture at different points). Really, in this case, if you want to make your picture clear in the middle, it stops to be clear at the edge.

    On the other hand, we can work over the numbers above, and give you the drawing we name “most-clear-at-each-point”. This may be not very important to you; this drawing shows how clear your picture may be at every point if you do not care what happens at other points.

    And now you can look at once at the “how-clear-at-each-point” drawing on the left, and at the “most-clear-at-each-point” drawing on the right. As you can see, to get your picture as clear as it is possible in the middle, you can move your most-important-to-be-clear point as far as half way to the edge. If you move it more, then it would be way easier to see that the lines going away from the middle are much more clear than the lines going across.

  • Marco

    There are a lot of guys testing lenses and equipment out there but there is nobody like Roger

  • On another note, I must say I’m still hesitant about Sigma Art’s long-term durability, compared to Canon L and Nikon flagship glass, let alone Zeiss. I’m a wedding photographer for my day job, (pun intended?) and I’ve been co-workers with a pretty huge team of photographers that has had nothing but grief from their Sigma Art glass, from AF woes to serious durability / longevity issues. I’ve seen multiple copies of the 24, 35, and 50 Arts that have literally rattled to pieces, coming apart at the mount or somewhere around the focus ring part of the barrel, …and the 24 Art is just horribly prone to getting terrible edges, even if all seems kosher mechanically.

    For this reason, I’ve continued being only partly a fan of Sigma Art, and still partly a fan of Rokinon when necessary. (Such as at 24mm) However, since Roki-Bow-Yang’s lens “repair” department is almost nonexistent, …this news of actual factory spec testing has me very very pleased. I doubt they’re as discerning and thorough as a complete OLAF test, but at least it’s a start.

    Probably what I’ll end up doing is buying a ~14mm lens directly from LR, either used or through the “I’ve rented but decided I want to own” program. (I forget the real name)

  • It’s a tough job, but somebody’s gotta do it. I’ve got an iron stomach, (I love driving on windey roads!) …so I’d be happy to drop a couple lenses in the name of R&D.

    Partly because I’ve already dropped lenses enough myself to fully understand the frustration of gear this just isn’t designed right…

  • (Insert heart-eyed emoji here)

  • These are done at infinity

  • bdbender4

    Holy cow! Three cheers for Sigma! Integrity is loose in the world, in a week when integrity in the political world… let’s not go there.

    Here is my wish: that for their next act, Sigma will design some really good SMALLER primes. Another approach besides the huge-heavy-distagon-packed-with-glass.

    And Roger, really, you love the splasho thingy? What is writing without wit, without soul, without eloquence, vivacity, sophistication, clarity, fine degrees of articulateness? Where else can you use 25 cent words? Tsk, tsk.

Follow on Feedly