Sigma 24-35mm f/2 DG HSM Art MTF Curves and Sample Variation
For years now, I’ve been telling people to always take the results of tests done on one copy of a lens with a serious grain of salt. Not matter how carefully it was tested, it’s still just one copy. Others will be worse. Others will be better.
Because I’m so aware of copy variation, I (in theory at least) test at least 10 copies of a lens and average those results before I present them. I do that because it gives a much better idea of what the average copy will be like.
Sometimes, though, it seems the true purpose of my life is to serve as a warning to others. I just can’t pass up an opportunity, don’t listen to myself, give in to temptation, and test just one copy of a lens. I justify it to myself because sometimes one copy is all I’m going to get, at least for a good while.
So today I’m going to use myself as an example of why testing one copy of a lens is just a bad idea. While doing so I’ll eat some crow and also apologize to Sigma. Because a few weeks ago I published the MTF charts on one copy of the Sigma 24-35 f/2 DG HSM Art zoom that I was loaned for a day and published the results. After that Lensrentals got multiple copies in stock and I did my usual testing of 10 copies. It is very apparent that that first copy was the worst of the bunch. The other copies, each and every one, were better.
Sigma 24-35 f/2.0 MTF
I’m going to put our original results from one single copy we tested earlier on the left, the average of 10 other copies on the right, so that you can see how different the average results are compared to the original single test lens.
Again, I want to emphasize that the MTF charts on the right are our official average results from testing 10 copies of the lens. I just put the original single test lens results on the left up so you can see the difference.
This gives an excellent example of why single-lens test results can be deceiving. If you notice, the single lens had excellent center resolution, a bit better than average. But off-axis (away from center) the test lens had a LOT of astigmatism, much worse than the average lens. In fact, it was worse than any of the 10 copies we ran for this test.
The average MTF curves for the Sigma zoom actually do (as Sigma claims) compare very well with prime lenses. Below are comparisons between the Sigma 24-35 f/2 and the Canon 24mm f/2.8 and 35mm f/2 IS lenses. The zoom is certainly right there with the two Canon primes.
Copy-to-Copy Variation
Now that we have our 10-copy testing done we can generate consistency numbers and variation graphs for the Sigma lens. These are really quite good, comparing very favorably to what we saw from Canon and Nikon 24-70 f/2.8 zooms.
Conclusion
Looking at the results of the Sigma zoom now that we have tested a number of copies, it’s very obvious Sigma has done it again. They’ve broken new ground and made a zoom that is a full stop wider than other zooms, yet is optically on a par with similar aperture prime lenses. And they’ve maintained excellent copy-to-copy consistency while doing it. Superbly done, yet again, Sigma.
Roger Cicala, Brandon Dube and Aaron Closz
Lensrentals.com
September, 2015
Addendum: I know many of you want to know if I have an explanation as to why the first lens was the worst lens. I can’t say for certain, it was loaned to us for just one day, but I would guess that it may have gotten knocked around during use or shipping. The problem it had was subtle. Center resolution was still excellent and there were no clear signs of decentering (we checked carefully on some pretty expensive equipment as well as the usual test targets). It’s a good example of why it’s so difficult to decide, using just one copy, if a lens is as good as it can be or not. Even out of sorts, the Sigma was still excellent, about as good as we’d expect an excellent zoom to be. It just wasn’t as good as the other copies, which are better than we expect an excellent zoom to be.
22 Comments
Andre Y ·
Thanks Roger and Brandon for following up on the original report. If you had put the bad sample in with the rest, how much would the consistency scores have changed?
Roger Cicala ·
Andre, they would have definitely been worse, but I didn’t plug it into the calculations.
Chris Knight ·
I am curious if the loaner was an early production unit (low serial number). If you happen to have pics of the lens from when you had the loaner, check the serial number and check it against your batch of 10.
Roger Cicala ·
Chris, I can’t say. It was a while ago and we didn’t even write the serial number down, just Copy1 on the test files. I would guess that it probably had more to do with that particular lens having made the rounds from place to place before it got to us. We see our own lenses get knocked out of sorts by shipping every once in a while, for sure.
Joshua ·
I’m curious what would have become of the the first, poor-testing loaner lens if it had been one of the lenses procured for normal service. Would it have been included in the rental pool?
Roger Cicala ·
Joshua, by our protocols any lens off from the average by that much would have been removed from service.
Curby ·
For years now, I’ve been begging you guys to do multiple copy reviews so I don’t have to take results of tests with a serious grain of salt. Thanks again!
Randy ·
Thanks, Roger but I wouldn’t be too quick to disregard your first impressions. I know Sigma is supposed to have turned over a new leaf with these Art lenses but as someone who used to sell Sigma/Tamron/Tokina/Vivitar lenses, Sigma variation was pretty extreme. Lens design and optical quality is one thing; consistency is something else. Since most of us can’t easily compare multiple lenses it’s nice to know a brand is really on top of quality control.
brandon sundt ·
dear randy,
I think you skipped the part where Roger says that the charts on the right are an average of ten lenses, and how the lens copy variation numbers are good.
CarVac ·
Wow, what a lens. Fascinating that the first copy happened to be exceptionally bad. If it was loaned to you, then clearly somebody else used it (for a review?) too and might have the wrong impression of the lens.
One question as an aside: would you be able to link to relevant blog posts from the individual lens pages?
Roger Cicala ·
Car Vac, that’s a good idea. I’m forwarding that.
Andrew ·
Do you have plans to sell pre-tested lenses with MTF results for that particular copy included in the package? A lot of people would be interested, I’m sure.
Another service could include sending to LensAutority client’s own camera body to find best matching lenses, or vice versa.
Like “OK, we tested your 5Dsr with 20 samples of 24mm f/1.4, would you prefer better center peak performance or better sharpness across the frame? Do you plan to use it mostly for interiors, landscape or astrophotography? Results for 5 best copies attached to the email, please pick which one will be yours.”
Roger Cicala ·
Andrew, we’ve considered much of that. With some high-end lenses we do include an MTF chart with the lens at LA, but MTF testing is expensive and for most lenses it hasn’t made any difference in sales (when we’ve listed lenses both with and without MTF tests it hasn’t mattered much in how quickly they sell – people are much more worried about if there’s a scratch on the front element or how much dust is inside 🙂
EL ·
Roger,
Do you buy lenses directly from manufacturers or from commercial retailers? I’m somewhat curious, given how important your data and reviews are to influencers within the camera community, whether lens manufacturers might start “binning” lenses, i.e., selecting the best copies to ship to you, in order to get better reviews from you. That would be no bad thing for your business or your customers, but it might skew the data more favorably.
Roger Cicala ·
EL we buy from a number of retailers, but basically off the shelf stock. We don’t buy lenses directly from manufacturers, although we do buy other things directly.
Anyway, while a manufacturer might cherry pick a few lenses to go to reviewers, I’m pretty comfortable that none has the capabilities of cherry picking 100 copies to go to a lens rental house.
Frank Kolwicz ·
The difference between that first example and the 10 brand new(?) ones may speak more to the durability of the lenses than anything else. I’ll be watching for reports on how they hold up to a year’s worth of rental abuse.
Brandon ·
Frank, the astigmatism of the early sample isn’t caused by a tilt or decenter so to speak, as it was pretty highly symmetrical. A spacing error front-to-back can hugely affect the astigmatism, which is what I suspect happened to that sample. Perhaps that will be a consistent issue with this model, but I doubt it given the track record of the other “Art” series lenses.
It probably was just an extreme outlier.
Brandon
CarVac ·
These still haven’t been put up on The-Digital-Picture yet for comparison.
Roger Cicala ·
CarVac, they have them, but I think they’re just backed up a bit.
Carl ·
Perhaps you can update your original post with a link to this one? It’s a shame as many, including myself, shied away from purchasing the Sigma because of the atypical astigmatism.
Mike Aubrey ·
I don’t know if this will be seen since it’s an old post, but it would be useful if you added a link to this post in the original single copy analysis from July 2015–just in case people are searching and end up at the old post rather than this corrected one.
iKonOkLasT ·
Some of these plots need to be revised in light of later reviews. Variance plots on a scale from 0 to 2? The results of that single lens should have created much greater variance bands than plotted here.