MTF Tests for the Sigma 14-24mm f2.8 Art Series Lens

Most of you know I’m a big fan of Sigma’s Art prime lenses. They give superior optics at excellent prices just about every damn time. I’ve been more mixed in my opinions of Sigma’s Art zoom lenses. I thought the Sigma 24-105mm f/4 Art was the best 24-105mm I’ve ever tested (and I hate testing 24-105s). The Sigma 24-70mm f/2.8 Art, on the other hand, I thought was adequate.

I wasn’t sure what to expect from the Sigma 14-24mm f/2.8 DG HSM Art, so I was eager to test it and grabbed copies as soon as we had them in stock. Why then, you may logically ask, am I just now writing it up? Well, partly because I have jobs that like, you know, pay me to do stuff. And somewhat because this became an exciting learning experience that I’ll tell you about at the end of the post.

About the Lens

Years ago, Nikon came out with their Nikon 14-24mm f2.8G ED lens. It was a revolution; wider and sharper than any other wide zoom made at the time. It remained the gold standard of wide zooms for years. Sigma isn’t filling a void here; Nikon already has an excellent 14-24mm, and Tamron has the not-quite-as-wide but really close 15-30mm f/2.8 Di VC USD.

The Sigma is less expensive than the Nikon by $400, but $200 more expensive than the Tamron 15-30mm f/2.8 Di VC USD. The Sigma, at 1150 grams is a bit heavier than the Nikon (1000g) and about the same as the Tamron (1100 gm).

So I thought that the new Sigma should be as good as the Nikon 14-24mm and better than the Tamron 15-30. That’s a tough task. Yes, I’m aware someone is going to make noise about other lenses. First, f/4 is not f/2.8. Second, 16mm is not 14mm, they are very different beasts (not to mention, most 16-35mm zooms are actually 17mm at the wide end).

One Disclaimer

Of course, the usual ‘this is not a lens review, it’s an optical test’ stuff applies to this post. One other thing needs to be mentioned when we test full-frame f/2.8 lenses wider than 16mm; we are either right at, or possibly past, the limits of our machine. It’s complex, it’s arguable, but it deserves mention that you should probably consider the graphs in this post ‘close, but might not be exact’ at the outer edges.

Also, I’ve violated my usual testing protocol for the first time, and I think you should know that. The test I’m showing you today is my second set of 10 copies of the lens, not the first 10. If you’re interested in why you can read about it in the addendum.

Optical Testing

As is our practice with 2x zooms, these lenses were tested at both ends (we do middle focal length testing on 3x and longer zooms). To start with, let’s mention the actual focal length of the lens is 14.3mm to 23mm. I mention this because that’s less ‘fudging’ than the label usually has; +/-5% is pretty standard for a zoom. Distortion is also reasonable: 0.8% at 14mm and 1.1% at 24mm. At both ends the distortion is mildly mustache shaped – a bit like the Zeiss 21mm f/2.8.

MTF of the Sigma 14-24mm f/2.8 DG HSM Art

At 14mm

This is an excellent resolution. About halfway to the edge, we start to see some astigmatism-like separation of the sagittal and tangential lines (it could also be the lateral chromatic aberration).

As I mentioned above, 14mm full-frame lenses are right at the limits of our machines automation. The drop in tangential MTF you probably note at 4mm away from the axis is perhaps a testing artifact; I won’t bore you with details about distortion being mapped at insufficient points for a very wide-angle lens. It could also be caused by an aspheric element. I can’t say for sure. 

Olaf Optical Testing, 2018

At 24mm

The resolution is even better at 24mm, just excellent across the field, and with no testing artifacts.

Olaf Optical Testing, 2018

Stop-Down MTF

The above graphs are the average of all ten copies tested at the widest aperture. We took one typical copy and stopped it down for retesting. There is a marked increase in resolution at f/4 compared to f/2.8. There was very little change between f/4 and f5.6 so I’m not going to clog things up with those graphs.

14mm

Olaf Optical Testing, 2018

24mm

You may notice that the MTF graph looks a little worse at the edges at f/4. This is a complicated thing that involves distortion, the lens’ field curvature, and a change in focus point of the machine, not a real thing.

Olaf Optical Testing, 2018

MTF Comparisons

Nikon 14-24mm f/2.8G ED

We’ll start with the comparison to the long-term gold standard, the Nikon 14-24 f/2.8G ED. The Nikon is a much older design, of course, but still a superb lens.

At 14mm

Here is an excellent example of two great lenses that are different, not one-is-better. Both are excellent in the center. In the outer 1/3, the Nikon has less sagittal-tangential separation; the Sigma has higher resolution. Will that make a difference in your image? With careful comparison, you might notice the Sigma has better corner resolution, but perhaps busier bokeh. You might not notice any difference at all.

Olaf Optical Testing, 2018

At 24mm

At 24mm the Sigma is strutting its stuff. It’s sharper everywhere. This difference should be noticeable, for sure.

Olaf Optical Testing, 2018

Tamron 15-30mm f/2.8 Vi DC USD

At 14 vs. 15mm

The Tamron is an excellent lens at the wide end, very comparable to the Nikon. As with the Sigma-Nikon comparison, I won’t try to call better (especially given that we’re testing at the limits of our bench). The Sigma may be a bit better in the center; the Tamron maybe a bit better at the edges. Again, though, we’re straining the limits of our testing capabilities here, so I can’t say there’s any noticeable difference.

Olaf Optical Testing, 2018

AT 24mm vs. 30mm 

Again, it’s not close at the longer end; the Sigma is clearly better here.

Olaf Optical Testing, 2018

If you want a really good example of just how hard it is to do 14mm, here’s a comparison of the Sigma 14-24mm f/2.8 with the Canon 14mm f/2.8 prime. Depending on what you emphasize, the Sigma comes close to violating Roger’s Prime Directive: No zoom is as good as a prime. In this case, the zoom may be better.

Olaf Optical Testing, 2018

The Canon is not the best 14mm prime; I chose it because f/2.8 to f/2.8 makes a nice comparison. The Sigma 14mm f1.8 Art is the best 14mm prime. At f2.8 is better than any of the others. So Roger’s Prime Directive holds true.

Olaf Optical Testing, 2018.

 Field Curvature

Let’s leave the fanboys behind to argue about the comparison MTFs and get into something useful for photographers who actually use the lens; the field curvature.

14mm

The sagittal field is a gentle M curve (lower left) while the tangential field (lower right) is a strong W. This gives the overall field (upper left) a bit of a U shape, and more importantly, means there’s going to be some astigmatism at the edges. It’s a 14mm lens, after all.

Olaf Optical Testing, 2018

The strong field curvature affects the standard MTF curve (it is focused in the center, so the edge MTF is low partly because it’s out of focus). We can pull the data above to give Best Individual Point MTF (BIF). This is a theoretic measurement, showing how much resolution you can get if you focus on that specific point rather than the center.

Below is the center focus MTF on the left, the BIF MTF on the right. As you can see, if you focus off axis, you can get near-center sharpness just past halfway to the edge. Beyond that, you can get good sharpness, but you will have some astigmatism.

Olaf Optical Testing, 2018

As I mentioned above, if you focus off axis, the center of the image will get softer, because of field curvature. Since 14mm is used for landscapes at times, it can be useful to find the best focus point – the place that if you focus there, you get the best overall sharpness from corner-to-corner. Below is a graph showing where we calculate that point should be, and what the MTF would look like if you focused there.

Olaf Optical Testing, 2016

So, if you want to get the best edge-to-edge sharpness on your landscape shot, focus a bit less than halfway to the edge of the image; that will give you the best average sharpness across the frame.

24mm

At 24mm the tangential field is still slightly W shaped, while the sagittal is a shallow U shape. The important thing is they are more similar to each other, and the overall field is a gentle U with less astigmatism than at 14mm. This is a really nice performance for any lens (even a prime) at 24mm.

Olaf Optical Testing, 2018

When we compare the center focus MTF (below left) with the Best Individually Focused MTF (below right) we see that at 24mm you can get excellent sharpness throughout most of the field; even at the edges, things are very good.

Olaf Optical Testing, 2018

If we find the focusing point that gives our best overall focus, once again it’s at 8mm from the center, just about halfway to the edge of the image. That’s convenient; the Best Average Focus point is at the same place at either end of the zoom. Less to remember.

Olaf Optical Testing, 2016

Conclusion

I said at the beginning that I expected this lens to be at least as good as the Nikon 14-24mm f/2.8 and better than the Tamron 15-30mm f/2.8. I think it’s pretty apparent that while they are all within hair splitting distance of each other at 14mm, the Sigma is the best at 24mm. Optically, this one is worth the money and the best overall 14-24mm zoom. There are lots of other things that go into choosing a lens, but at these very wide angles, things like autofocus speed are not big points.

That being said, if you’re buying a 14mm lens to be a 14mm prime (and some of you are), then all three of these do very well. Honestly, though, for a little more money you should consider the Sigma 14mm f/1.8 Art if that’s the case. If you’re thinking of matching a 14-24mm f/2.8 with a 24-70mm f/2.8 to collect the whole focal length set, then the Sigma is the best optical choice.

Two things I’ll mention. First, for the 72,534 of you that want to compare it to 16-35s, go right ahead. But it’s not a fair comparison, and I’m not going to do it. 14mm is very different than 16-35; much more difficult to design and manufacture. If a 16-35mm will meet your needs, and hold your filters without additional contraptions, then get a 16-35mm.

Roger Cicala and Aaron Closz

Lensrentals.com

July, 2018

Addendum: Sigma’s Response

Our protocol is always the same: the intake people inspect and test lenses on test charts, fail anything that’s obviously bad and pass the rest on for testing. I test the first ten lenses they send me to write the blog post.

This time a weird thing happened. On testing day 1, I tested the first eight copies of the Sigma. At 24mm five of them were identical and excellent. Three were not OK and was a big gap between the two groups. This is NOT normal. Generally, we see a cluster of lenses that are good, a lens that’s not quite that good, another not quite as good as that second lens, etc. In other words, the variance is always a steady trend away from best.

This is not something I’ve ever seen before in testing hundreds of lenses. I have a good working relationship with Sigma, so I emailed them as a courtesy, as I do with any manufacturer that corresponds with me. (They generally ask me to retest and want to argue about why my results aren’t valid. I generally retest and find my results are valid.)

Sigma didn’t do this. Instead, they asked for the serial numbers of the lenses. A few hours later they told me they had pulled the tests of those lenses at the end of the assembly, that the three lenses in question had tested much better than this at the factory, and asked me to send them to Japan for repeat testing.

Most of you don’t realize the enormous shock this caused me. I’ve been testing lenses for a decade and working with manufacturers for nearly that long. Leica has this kind of testing for every lens. Zeiss has it for certain lenses in their lineup. Nobody else does. Leica lenses. Some Zeiss lenses. Sigma Art lenses. Yes, I know all about other claims of ‘we test every lens’; that means it was mounted to a camera to program the AF firmware. Nobody else (with the possible exception of Canon with some of their newest lenses, and I’m not certain of that) is testing every copy optically and keeping the results on file.

Ten days later (shipping to Japan takes a while), Sigma emailed to tell me the lenses I sent back tested much worse than their original end-of assembly tests. Their new tests agreed with my findings and were, in their words ‘not acceptable for this lens.’ I’ve heard ‘this lens is within spec’ from manufacturers on lenses I knew were bad so often, for so long, that I reread the email just to be sure. To paraphrase a bit: “We have retested the lenses and agree with your findings. They aren’t good now. They were quite good when they left here. We’re investigating to find out what happened.”

Then they had their engineers disassemble the lenses and found some minor damage to rollers consistent with an impact that had caused the problem. I’ll be blunt, usually, I’d expect a ‘so you guys must have dropped them’ next, because these came from 3 different suppliers. And that is certainly possible. The lenses passed through several hands here on their way to me, and while very rare, things get dropped (and that people dropping don’t always admit they’ve dropped).

Instead, Sigma said they would track the shipment and see if the lenses were in the same crate or pallet during transit. Then they told me something that made me nearly weep with gratitude, showing that in a few companies engineering, not marketing, still runs the show:

Our engineers have also taken a number of new lenses from the assembly line, dropped them in various ways, and then disassembling them to look for damage. We did this until we had reproduced this type of damage. We will now see if there are changes we can make in the lenses or packaging to prevent this from happening in the future.

This is, in my opinion, what superb engineering and quality assurance are all about; look for any weaknesses and strengthen them. I’ve been working in this industry for many years, and this is the attitude I wish every company had, but few do.

Also, to be as certain as I can that the tests I’ve presented here are representative, I’ve tested a total of 25 copies now; the three damaged lenses were the only ones with any issues.

OFFTOPIC – And a bit of fun, for both of you that read this far.

I LOVE http://splasho.com/upgoer5/ where you have to write using only the 1,000 words most commonly used in American English. So, I rewrote a couple of paragraphs until they passed the test. If you want to try your hand at it, the two original paragraphs are under “14mm field curvature”. Post your results and shame me with how poorly I did.

The way the field moves changes how close to perfect the MTF (which is focused in the middle) is. We can pull the numbers above to give Best-at-each-point MTF. This is not a real world thing, because if you focus on that point, rather than the middle, the middle gets soft.

Author: Roger Cicala

I’m Roger and I am the founder of Lensrentals.com. Hailed as one of the optic nerds here, I enjoy shooting collimated light through 30X microscope objectives in my spare time. When I do take real pictures I like using something different: a Medium format, or Pentax K1, or a Sony RX1R.