Sigma 24-70mm f/2.8 DG OS HSM Art Sharpness Tests
Sigma has been on an incredible run these last 5 years, releasing one amazingly sharp lens after another. They’ve made lenses no one has ever tried before and not only succeeded, they made them amazingly good on the first try. Their quality control has become as good as anyone’s, better than most. And their repair service has become one of the best out there.
Like many of you, we’ve waited for the Sigma 24-70 f/2.8 Art lens for quite a while. It would have image stabilization, it would be less expensive than the brand name alternatives, and it would be sharp as heck, because it was a Sigma Art.
I’ll save those of you who hate to read the trouble of reading. Even Babe Ruth hit singles sometimes. It had to happen. Sigma has made lens after lens that exceeded everyone’s wildest expectations. Sooner or later they were going to make one that didn’t. This isn’t a bad lens, but we’ve come to expect amazing things from Sigma Art lenses and this lens is not amazing.
As always, these are the results of 10 tested copies; each tested at four rotations. For those who don’t speak MTF, the easy version is higher is better, and dotted and solid lines of the same color close together are better. And as always this is an MTF test, not a review. I’m still not sure I can pronounce bokeh, much less describe it to you.
MTF Results
We’ll look at the results at 3 focal lengths; 24mm, 50mm, and 70mm. We expect most 24-70mm zooms to perform best at 24mm and be weakest at 70mm. The Sigma is actually a bit different, having its best performance at 50mm.
24mm
One thing to note at 24mm is the bulge of astigmatism-like separation in the middle of the field, from 4mm to 12mm or so off-axis. I’m not sure what this will look like in photographs, but it might be, well, different. Or maybe not noticeable. I’ll be interested to see.
50mm
Things sharpen up nicely and the curves become much smoother and regular. I expect 50mm is not only the sharpest zoom range, but probably has the best out-of-focus appearance, too.
70mm
Resolution drops off at 70mm, but the curves stay smooth good away from center.
Copy-to-Copy Variation
I can’t say this was great, honestly. At 24mm we have a nice, tight range but things get a bit random at both 50mm and 70mm. Overall I’d call this better than average at 24mm and a little below average at 50mm and 70mm.
24mm
50mm
70mm
Field of Focus Curvature
Please don’t mistake this for distortion measurements, which someone did a couple of weeks ago.
24mm
There’s a gentle curve at 24mm in the sagittal field, with the tangential field curving more severely.
50mm
At 50mm the sagittal field is perfectly flat with the tangential field reversing curvature into a mild mustache pattern.
70mm
At 70mm the sagittal field remains flat. The tangential field, well, we had the expectations setting a little high on our bench and the curve really didn’t resolve well enough for us to clearly tell about the tangential field. Maybe a mustache. Maybe who cares.
Comparisons
Well, the charts are nice and all, but it’s always good to have comparisons. I’ve carefully selected the ones I think are appropriate and avoided the ones you wanted to see. It’s not that I’m purposely cruel, wait, yes it is.
Sigma 24-70mm f/2.8 Art vs Canon 24-70mm f/2.8 Mk II
The Canon 24-70 f/2.8 L Mk II is about as good as it gets for zooms in this range.
24mm
The Canon is at its best at 24mm and the Sigma gets pretty beaten up here.
50mm
At 50mm the story is a little different. The Sigma is at its best at 50mm and the Canon has dropped off a bit. In the center things are completely even. The Canon is just a little bit better in the middle of the field. So if you want to compare your new Sigma Art to your buddies Canon, try to do it at 50mm.
70mm
Both lenses have fallen off a bit at 70mm. The Canon is a little better here but not as dramatically better as it was at 24mm.
Sigma 24-70mm f/2.8 Art vs Tamron 24-70mm Di VC
This is probably a more reasonable comparison; the two image-stabilized third-party zooms. The Tamron G2 version will be out soon and is expected to be better, but we don’t have MTF tests on it. Because soon is not the same as now.
24mm
At 24mm, the Tamron is clearly a bit better.
50mm
The Sigma again shows it is at its best at 50mm, and particularly away from center, it is a little better than the Tamron.
70mm
At 70mm the Sigma is better than the Tamron, which is clearly weakest at 70mm.
Sigma 24-70mm f/2.8 Art vs Nikon AF-S 24-70mm f/2.8 ED VR
Nikon has a different emphasis in their 24-70, giving up some center sharpness in exchange for good sharpness across the entire field.
24mm
The pattern is familiar, at 24mm the Sigma just isn’t as good.
50mm
AT 50mm, though, the Sigma is clearly sharper in most of the frame. This is the weak focal length for the Nikon and the strongest range for the Sigma. In the outer 1/3, though, the Nikon is a little sharper.
70mm
At 70mm the Sigma has better sharpness at the higher frequencies, the Nikon is a smoother away from center.
Conclusion
I’ll admit I’ve been a bit of a Sigma Fanboy lately. The only thing better than aggressively trying new things is aggressively trying new things and making them awesome and that’s what Sigma has been doing. But I’m not a big fan of this lens. This is an adequate lens, but nothing more than that.
I’d probably feel better about it if it didn’t have ‘Art’ on the label. I’ve come to recognize Sigma Art to mean ‘as good or better than any other lens in that focal length, even when the others cost way more.’ This lens I would describe as adequate overall. It’s weak at 24mm and good (but not awesome) at the longer parts of the zoom range.
If it didn’t say Art on the side and cost a few hundred dollars less, I’d probably be less disappointed. If I was being snarky, I’d say they left the “F” off of Art on this one. But I’m trying to be less snarky these days so I won’t say that. Or at least won’t say it again.
The Sigma 24-70mm f/2.8 Art Series‘ better performance at the long end may appeal to people that already have 24mm covered with a good wide-angle lens. If you use your 24-70 f/2.8 mostly at 50 and 70mm then the weakness at 24mm may not bother you much.
I think most people considering this lens are going to wait to evaluate the Tamron G2. If the Sigma price falls significantly it may be a more attractive option, but right now I can’t see a strong reason to make it your 24-70 choice. It’s not a bad lens, just not an Art lens, really.
Roger Cicala and Aaron Closz, with the invaluable assistance of hard-working intern Anthony Young
Lensrentals.com
July, 2017
Addendum: As requested, comparison to the Tokina 24-70 f/2.8
24mm
50mm
70mm
113 Comments
Yair ·
Thank you very much Roger !
You confirmed what i saw on test images from many places, the Lens is average ,certainly a disappointment after the 85,135,14 which are superb.
Please advice are you going to make Sharpness Tests to the Nikon 28 1.4 E and the Milvus 35/1.4 soon?
It is very interesting how they will perform.
Thanks again for the great Job.
Roger Cicala ·
The Nikon is on the list, but not at the top of things; we’ve got a long pipeline at the moment. The Milvus will be ahead of it, whenever we get enough to test.
Yair ·
Thank you!
?ukasz Moszczy?ski ·
Canon 24-70 2.8 II ($1749) and Sony 24-70 G Master ($2198) are still unbittable.
Even weight of over 1 kg did not help Sigma 😉
Roger Cicala ·
The Canon has been seen for somewhat less than that, but the Sony, well, as we say “They sure are proud of that one”.
Yair ·
I was hoping Nikon users will have a good option for stabilized 24-70 2.8.
It didn’t happen.
ToastyFlake ·
Hopefully the Tamron won’t let us down.
Yair ·
I don’t see a reason why the Tamron will be any different than G1
The optical structure is completely the same
ToastyFlake ·
Better glass. I’m expecting performance to more closely match the 70-200 g2.
Yair ·
I saw more information here that I am wrong and optical performance is improved.
So we have hope!
ToastyFlake ·
I’ll just stick with a couple of primes and my feet at this focal range, if this one doesn’t turn out good. Too much money for mediocre performance from the all the options right now.
Michael Clark ·
Even if the glass is the same, if the supporting architecture and the manufacturing process that puts it all together allows for more consistency in keeping each element properly oriented with regard to the others then the average copy of the lens can be better.
Łukasz Moszczyński ·
Canon 24-70 2.8 II ($1749) and Sony 24-70 G Master ($2198) are still unbittable.
Even weight of over 1 kg did not help Sigma ;)
decisivemoment ·
Tamron G2 appears to have the exact same optical design as the predecessor. So the only way waiting on that’s going to be worthwhile is if the G2 quality control matches the apparent improvements in AF performance and VC.
dyna ·
As a regional Tamron representative, I have been assured by Japan that this is not actually the case. Glass formula and coatings in particular have changed, as well. They are by nature of field of coverage and certain correction characteristics similar but improvements have been made throughout the entire piece, inside and out. It is considered an overhaul.
Yair ·
Thanks you so much for the information!!
This is very important information mainly for Nikon users that do not have any better option for 24 – 70 standard zoom
Micah ·
“Do not have a better option”? Despite the bad press, the latest Nikon is very, very good. It ain’t cheap, but you get your money’s worth. I’m not a fan of Nikon USA’s service, but I’d still pick it as a superior product. At least in this space. Primes…Sigma is just nailing it, no question.
Xrqp ·
The MTF graph that Tamron shows is identical to previous lens. Is it possible the coating affect more the transmission and not so much the MTF?
Reggie ·
Yeah, they literally just have the same images for the optical construction and mtf charts on their marketing site. It’s copied and pasted.
Maxim Podtopelny ·
Looks like Sigma messed up 24-70 and 100-400 letter signatures. By the way, are you going to write something about 100-400 C?
Roger Cicala ·
100-400s are difficult to test, actually in theory out of the range of our bench. There are workarounds, but they’re time consuming. So it’s on the someday stack. It would take most of a week to test, compared to less than a day for a prime and usually 2 days for a standard zoom.
Sergiu Mosoia ·
Hi Roger, thank you for the test, this is disappointing for me, really.
Here is why: I had big expectations for this lens, hoping that the optical quality will match my Canon 24-70 II – I frequently use 1/320 @ 70 mm (f/2.8) with this lens on the 5D Mk IV…which is short, very short (on the 5D Mk III I used 1/160 with the same lens, but I also had to correct now for the increase in resolution… and the bigger issue is that my hands are no longer that steady…).
Again, thank you.
Franck Mée ·
Hi Roger,
“bokeh” is actually very easy to pronounce. The “bo” part is anywhere between the close “bo” of “boat” and the open “bo” of “bot” — but always a short vowel, not a looong one.
The “keh” part sounds very much like the “quet” of “bouquet” if you pronounce it correctly (I’m French by the way): it ends like “May”, minus the “y”.
So bo-ke, [boke] in phonetic writing, two short vowels, no diphthongs. Now, even Japanese people are used to English-speakers saying like “bo-kay”, so that should be okay though not really correct.
Just don’t say “bow-kee”, as they added the “h” (normal roman transcription would be “boke”) especially to prevent your guys from doing this mistake, which makes it sound like “boki”, which means bookkeeping. 😉
Have a nice day!
Max Manzan ·
Thank you very much Roger for your efforts. You provide, as always, very valuable information on lens performance. Cheers.
Lorenz Ammon ·
Thanks Roger for not being a real fanboy and not holding back the results when the lens couldn’t meet your (high) expectations 😉 You mentioned the not so great copy-to-copy variation. I’m wondering if there is variation in the sample variation of a lens’ different production runs, i.e. if maybe the first production runs show more sample variation than the later ones? Could your vast data collections on lenses (lens manufacturers) answer this question?
Roger Cicala ·
Lorenz, if there is we’ve never detected it. We’ve done batches from lenses years apart in production and never seen a difference.
Lorenz Ammon ·
Thanks Roger. Good (and comforting) to know that the manufacturers seem to be consistent in this regard (and that already right from the start).
Ruy Penalva ·
Really not worth to buy unless to use in E mount with mc-11 adapter. Lets wait for Tamron tests.
Chik Sum ·
Why say so? in E mount the GM lens is the best current available 24-70 lens!
Ruy Penalva ·
For Canon users
TinusVerdino ·
Al these tests are at F2.8 I assume. Stopped down all 24-70’s will perform well.
Roger Cicala ·
That is correct. All at f/2.8
Adam ·
Can’t wait for the tear-down! If you do end up doing one, might you be appending to the 2013 Canon, Nikon, Tamron comparison?
Roger Cicala ·
Adam, I don’t think we’re going to tear this one down any time soon. We have a bunch of new Cine lenses that are ahead of it on the tear down list.
Adam ·
Also fun reading!
Benjamin S. Kim ·
It would be better to test in real life with test chart. I see a lot of different results from this site and other sites. Such as Sony 24-70mm F2.8 GM which has a poor IQ compare to other 24-70 especially at the edge.
Chik Sum ·
Really? I saw the GM photos seems quite good actually
Alpha Omega. ·
It is very clear that 24-70GM has poor edge sharpness and have lower score on Dxo.
Reggie ·
This site has demonstrated why you need to test many copies of a lens, especially zooms. There is just too much copy variation.
TK ·
Thank you for your review. I like your honest conclusion.
Can you make a comparison chart over 24-105 f/4 art? I have this(24-105) lens and want to see the difference.
Roger Cicala ·
I don’t have 24-105 Art data, I’m afraid.
Alan Yuen ·
I will wait and see Tamron
sickheadache ·
All Shot at 2.8…nothing but 2.8. What would be the conclusion at different settings? Hummm The world may never know! lol.
Roger Cicala ·
Well, it will never know from me. Because we don’t have that kind of time to give for more free testing. However, if you’d like to fund 2 days of testing on the optical bench, we’ll be happy to test at the aperture of your choice.
sickheadache ·
Roger…you know what…I am a long time renter from Lens Rental. Maybe in the kindness of Lens Rental…You can send me a D810 plus Grip, and the Sigma 24-70mm Art, for my new testing chores. thanks a heap….Thomas Douglas.
dyna ·
It would, in many ways, be an exercise in redundancy. Lines will tighten up and general performance will improve overall until diffraction kicks in, which will be earlier and earlier on the higher resolution cameras with greater pixel density. It seems rudimentary to note as such and, really, because this rule is nearly always in stone, it also seems a waste of time (more of a vanity project, really) to test at the minimum f-ratios.
Xrqp ·
You greatly (excessively?) generalize. Not helpful compared to OLAF. You never know at what f stop the MTF could jump.
ToastyFlake ·
Oh, snap!
Gabriel ·
Who cares…most people by a 2.8 to shoot at 2.8 not to freakin stop down!
PWPhoto ·
Not necessarily. I own a Zeiss Milvus 35mm f2 and I hardly ever shoot it at f2. In fact I shoot it at f8 90% of the time. I didn’t buy it for the speed. I bought it for the clarity, build quality, micro contrast and color rendition. I bought a Zeiss and for all the reasons you buy a Zeiss. It isn’t always about speed.
Michelle Bernard ·
I believe you’ve done tests on the Tokina 24-70 f/2.8 – what is the comparison? Thanks!
Roger Cicala ·
I’ll add those, but want to put out the caveat the the Tokina has a really big sample variation, so the averages we post may really not reflect your copy. Some are a lot better, some a lot worse.
Roger Cicala ·
They’re up now
John from PA ·
When you test a Sigma lens that can be tweaked using the dock, is that done prior to your tests or do you test right out of the box?
Roger Cicala ·
John, that has nothing to do with optical testing. The “Tweak” is to improve autofocus accuracy – it doesn’t change the optics at all. No testing is ever done using autofocus.
Joshua Efron ·
I’d been waiting a couple years to see what Sigma would come up with, so I was really hoping Sigma would at least hit a triple on this one. Guess now it’s time to wait for Tamron’s next version and decide where to go from there.
Meanwhile, my Tamron 28-75 2.8 it is.
Micah ·
I was frowning at this comment until you said 28-75mm. It’s such cheap plastic crap, available with several different brands printed on it…yet it’s far better than it has any right to be! That’s one of those magic lenses, that a good sample is sharp, and it’s really quite impossible to beat for the price. I do fear that camera resolutions are finally starting to outpace it’s capabilities though…
Cassander ·
I'd been waiting a couple years to see what Sigma would come up with, so I was really hoping Sigma would at least hit a triple on this one. Guess now it's time to wait for Tamron's next version and decide where to go from there.
Meanwhile, my Tamron 28-75 2.8 it is.
Athanasius Kirchner ·
Wow Roger, you’ve really become harsh in your judgment recently. That lens looks fine to me, and in line with the Nikkor’s cost/performance ratio (less dollars for less performance), but I guess you expected an Otus killer ^_^
Thanks for the test!
Roger Cicala ·
You’re right. My expectations were too high and it bled through a bit. I really have gotten to the point that
Art meant Incredible Lens. I think it’s a good, although slightly overpriced, lens. If it have been $999 and didn’t have Art on the side I’d have thought it was great.
Nathan Connolly ·
Hey Roger, apologies if this question is one of those recurring things you always answer, but how does the presence of Image Stabilisation in a lens affect the ability to test it on an off-camera rig like yours (or when using it adapted with a “dumb” mount)? Is the stabilising element positively locked down when IS is switched off or the lens is un-powered, or does it just rely on spring tension to hold it in a relatively neutral central area?
Brandon Dube ·
What happens depends on the lens. On our usual Canon lens fodder, the IS is disabled on a powered camera and the unit locks in place. This is not always the case, and on lenses which leave the IS unit loose when unpowered, you will typically see above-average variance due to the allowed decenter of one of the groups.
CheshireCat ·
And this time “Art” stands for Artchoke. Thanks for confirming. Well done test as usual.
David Bateman ·
Very interesting test.
I would say if you have a Canon camera, the Canon version is clearly the best choice.
However, if you have a Nikon camera, things switch here and the Sigma is best. The sigma only really falls short at 24mm. But is better than all other Nikon options at 50 and 70mm.
Its like a large 50mm lens with little room to zoom. I would guess that, 35, 50 and 70mm are the strong points of this lens and 24mm would be only used if you need it. So if loched in to Nikon may be the best choice.
Unrest ·
Not so sure. Have you considered focus speed, accuracy, etc.? I’d be shocked if the Sigma or Tamron can focus better than the Nikon. Also consider the original 24-70 2.8 G from Nikon that some prefer for better center sharpness compared to the new VR version.
Xrqp ·
The OLAF data shows the Nikon lens is better than the Sigma overall. Maybe not enough to justify $1100 more.
I guess you could say Sigma is better if you severely discount the outer area of the image. But the outer 1/3 by lineal distance, is 55% of the image area. The outer 1/4 is 44% of the area.
JarnoP ·
For me the pattern here is that while Sigma’s Art primes are mostly superb, Sigma’s Art zooms are just OK, but not challenging the Canon/Nikon pro-zooms. I bought the Art 24-105/4 OS and it was just a heavier version than the Canon equivalent with slower auto-focus. It was not bad and some focal lengths I was telling myself it was bit better, but in general it was on only par and I sold it away.
Carleton Foxx ·
“Because soon is not the same as now.”
Another enigmatic pronouncement from the Confucius of Cordova. What wisdom is the ancient one trying to impart?
dyna ·
Looking forward to your Tamron SP 24-70 G2 test, Roger, for obvious reasons but also for personal ones as, for me, once the SP 35mm and SP 85mm lenses came out, the original 24-70 VC went on the shelf. I would really love to see you test the SP 85mm, as a matter of fact: it’s just a phenomenal lens and I think it would pleasantly surprise even your admitted Sigma fanboy self 🙂 I’ll be getting my SP 24-70 G2 sample in about two weeks, is my understanding. I’m looking forward to it.
Roger Cicala ·
I’ve got the Tamron 85 tested long ago, just never got around to writing it up. I’ll see if I can pull the graphs today.
Roger Cicala ·
https://uploads.disquscdn.com/images/98e891933a2787799dc9ced2a440b8cc29597f5b2187944e50b5366b8cb05861.png
dyna ·
Thank you, sir! Most intriguing.
LAB 2.35:1 ·
Hey Roger, thanks for the test, as always. Though I’m sure I’m in the minority here, it would be nice to see you add Sony’s glass to these comparisons in the future. Many folks adapt 3rd part lenses to the “universal” e-mount, as you know, and since nowadays Sony’s lineup of primes and zooms is quite competitive to the usual suspects I think it’s relevant to get them included.
For instance, I’m considering buying the GM 24-70 (rented one from you last year for 3 weeks) but with this new Sigma and upcoming Tamron I’m wondering if I should give them a serious consideration especially since they have the VC and Sony doesn’t. I bought the Sony FS5 recently which came with a free Metabones IV so I’m itching to try 3rd party glass on my e-mount bodies, since I have the adapter, and 24-70 is high on my list.
DONOTUNPLUG ·
Hey Roger, thanks for the test, as always. Though I'm sure I'm in the minority here, it would be nice to see you add Sony's glass to these comparisons in the future. Many folks adapt 3rd part lenses to the "universal" e-mount, as you know, and since nowadays Sony's lineup of primes and zooms is quite competitive to the usual suspects I think it's relevant to get them included.
For instance, I'm considering buying the GM 24-70 (rented one from you last year for 3 weeks) but with this new Sigma and upcoming Tamron I'm wondering if I should give them a serious consideration especially since they have the VC and Sony doesn't. I bought the Sony FS5 recently which came with a free Metabones IV so I'm itching to try 3rd party glass on my e-mount bodies, since I have the adapter, and 24-70 is high on my list.
JCT ·
Roger,
Thanks for the efforts. A couple of questions if I can. I too, like may others, have been spoiled by the ART 1.4 primes.. just great lens’. That makes this review, while I’m sure accurate, difficult to understand, as Sigma has really tried to up the game on all counts. The 150-600 Sport is just great, as least my copy (a beast, but great).
So Sigma originally planned a June release of these as I understand it and we’re near the end of July.. Mine is on Pre-Order. What I’m wondering, if you’re allowed to comment, is what production level were the 10 you had? Were they ones from “around the block” a few times with different reviewers, etc., or were these ones ready to ship. I’m wondering if the delay has anything to do with Sigma tightening up as much as they can before full mass production. The RAW images I’ve downloaded from various sites show the lens to perform very well at a lot of different focal lengths, f/stops, etc.
Again, appreciate the efforts.
Chuck Seybert ·
Hey All
What I don’t understand is why a company as big as Sigma would not do the same tests as Lens rental and try to improve on things that Roger and Co have found.
JCT ·
Chuck.. I’m sure they do, before, during, and after the final build, which is way some of what Roger did is troublesome given the “Art” lens history. Zoom lens’ very rarely keep pace with their Prime lens counterparts.. There is always some sacrifice made for the convenience of a zoom. Again, downloading the RAW files and processing in LR would indicate a very high performing lens (someone on another forum suggested using the Canon 24-70 MK II profile in LR6 – it works very well).
Chuck Seybert ·
I just returned this lens to Lens Rental after having it for seven days, Really enjoyed it, I found it performed better then my old 24-70 canon v1 and my just sold tamron 24-70 2.8 vc Waiting for the new tamron before deciding which one the purchase.
JCT ·
Chuck.. I am curious why none of the major sellers, B&H, Adorama, etc. have the lens and if some of these others were earlier production runs and not final?? I saw Amazon say they would start shipments on July 29th.
Chuck Seybert ·
Someone over at DPR, in the third party lens forum just received his lens a couple days ago. Said it was fantastic.
JCT ·
Chuck,
Just posted a link to a series of RAW files on new Sigma vs. my original Canon 24 -70 2.8 (not the MK II) .. So far I see no reason to return it, but hopefully posted enough images for people to make their own decisions. I’m not “seeing” what the tests would imply. Again, using the LR profile for the new Sigma. Have viewed the two sets both before and after NR/Sharperning preset for my 5D MK III..
Brandon Dube ·
Currently, neither Sigma nor any other manufacture does MTF testing with the level of rigor Olaf does. It is expensive and requires expertise they don’t have.
JCT ·
Brandon.. To be fair, OLAF though is not the only higher end testing mechanism. There is also K8, ImageMaster, LensCheck, and Wells Research. I’m not suggesting you need to use them all to validate OLAF, just that different systems can (and likely will) produce different results.. How much different, I don’t know, but that would also be an interesting comparison.
I’m also not ready to “diss” the designs at Canon and Nikon, and now how Sigma and some others are up’ing their game. Should a lens live up to design expectations (with accounting for some variation copy to copy), yes, it should, but again we’re also talking how light on a bench vs. “lighting” in a real world shoot behaves. The two don’t necessarily equate, and to my mind, both only simulate an expectation. Whether design or bench, neither test is more than “here’s how a lens performs given a certain lighting situation (or set of situations)” .. Neither emulate field lighting which I think is in line to your other reply talking to 135L and 14L II.
For me, the real performance tests will come with varying shots I can create and side by side lens comparisons with the Canon 24 – 70 (V1) I have first against the Sigma, then if necessary, a comparison against (most likely) either a rental Canon 24-70 2.8 L II or a refurbished one from Canon.
My only “argument” in MTF test results is that opinions are formed (and in some cases decisions made) with never having the lens in your hand… There are other characteristics that also contribute to a lens choice that MTF never addresses.. IS/OS, Build Quality (including weather seals), etc.. and then measure the whole ball of wax against price.
Brandon Dube ·
Olaf the instrument does not measure MTF. It could with some modifications, but those would be detrimental to its use as an alignment station, which is its true purpose. The location of the pinhole(s) in Olaf is also undesirable for lenses with short focal lengths, fast apertures, and good image quality.
Olaf is also a company, and all we do is optical testing and alignment.
I have used every commercially available MTF bench on the market, and a number of custom-built ones. Olaf has an ImageMaster with all the bells and whistles, and also a Wells Research bench that now sits on a shelf with all of our other eclipsed equipment (imatest, projection setups, etc). All lens MTF measurement techniques should yield the same results, given the same conditions. In this case, the conditions are spectrum of light, and object distance. The purpose of these quarter million dollar, traceable instruments is that measurement bias and errors are removed.
The same is also true of slanted-edge or other system MTF test setups, but slanted-edge is much more algorithmically intense than the pinhole or slit techniques used in lens-only MTF benches and there is a lot more room for debate in what is right and wrong.
I think MTF correlates exactly with “field work,” if you understand what it means. A scene ultimately has some spectrum of spatial frequency content and some spectrum of color content. MTF is exactly the degree to which the contrast any any given spatial frequency is supressed, and if the lighting in the MTF test and the scene are sufficiently similar in color then you could even use the MTF tests to simulate with great accuracy what an image taken of the scene would look like.
JCT ·
I think we’re going to have to agree to disagree Brandon. My customers don’t buy images because I tell them the MTF data said the lens was performing up to spec. They buy images based on every single thing that I, as a photographer, create with the equipment on hand. A lens is only one component, an important one (and I’ve conceded it should perform up to specs), but just as no two photographers are the same (nor their vision), no two scenes are likely ever the same (unless we’re talking somewhat static studio lighting – but even there things like skin tone, etc. can vary greatly).
For me, the value of this, or any lens, doesn’t come from a data chart. It comes from what I see in the results, how I feel when using it, how it holds up when I absolutely HAVE to have it perform, etc. The MTF charts are only one piece, and again for me, static bench lighting cannot predict “totally” how a lens will perform in a given user’s hands in and varying light they either create or are given by Mother Nature.
Again, I think your efforts are important, but not a decision maker – which unfortunately some use these charts as.
Brandon Dube ·
Ah, but your claim is false. Bench testing can predict, “totally,” how a lens will perform in a given user’s hands in and varying light they either create or are given by Mother Nature. This is covered in a number of ISO standards, 9334, 9335, 9358, and 11421 for MTF and stray light.
To believe otherwise is to believe that the optics and opto-electronics industry/industries are filled with people incapable of modeling and understanding imaging which is simply untrue.
If your principal concern for a lens is its contrast and resolution at full aperture (or other apertures we show data for), then these charts can be a decision maker. If you care about stray light, color, or other imaging properties it would be ridiculous to expect a resolution chart to tell you about those things.
JCT ·
Brandon.. I’m going to just have to believe my thoughts just as I’m sure you believe yours. NO modeling can ever account for EVERY given situation or combination of equipment, lighting, user knowledge/choices (or error). It can “predict” what it will do, but only within the parameters set forth by the testing equipment and environment of the tests. Any scientific study is always bound to that construct. Again, my opinion.. I appreciate and respect yours.
Brandon Dube ·
Ok… but all situations can be characterized and parameterized in the same way as a test or simulation. We did not get to the moon because we threw up our hands and said we couldn’t model things.
JCT ·
Nor did Apollo 13 almost cost the lives of three astronauts or the Hubble Space telescope become flawed when all the specs said they shouldn’t. Stuff happens..
Brandon Dube ·
I do not see the connection between accidents and the ability of our race to model things.
whereisaki ·
Granted, some lenses that are not great in measurable terms may have unique and useful qualities. But how is LensRentals supposed to measure the suitability of a lens for what you create, or how you will feel when using it?
Obsessing over numbers may be foolish, but at least they are usable for comparison purposes (provided there is no escape clause such as “pre-production,” hand-delivered sample, prototype, etc.) Otherwise, we are stuck with “Leica glow”, “dimensionality” and “sparkle” which are great if they exist but not something that can be measured.
Brandon Dube ·
Most don’t want to spend the money on metrology. It costs a lot and does not drive profits, which does not make sense to most business people.
Roger Cicala ·
Sigma does extensive testing, has developed a target based analysis based on their own equipment and at least owns MTF benches. (I have no knowledge on how they use them.) They also have started putting out some real-world MTF tests, as well as the computer generated ones. They’re more aggressively testing than probably anyone else.
Zeiss does actual MTF testing on their own benches, but the K8-K9 are a bit long in the tooth and have some limitations. When I last discussed it with them they only did an single cut across the lens, so they assume rotational symmetry.
Canon has probably the most advanced testing on-axis during assembly, but they don’t test off-axis very enthusiastically and tend to believe the easy (but very inaccurate) theory that if it’s good on-axis it’s good.
But mostly companies don’t want to spend money on metrology unless forced to. It’s expensive. And if it finds flaws, and that’s expensive. They do it when they absolutely have to if the company is run by accountants, or sometimes because they want to, if the company is run by engineers. That’s a generalization, obviously, but a pretty accurate one.
Chuck Seybert ·
Thanks Doc,
Keep up the good work, Looking forward to the Tamron 24-70 2.8 g2 tests.
Brandon Dube ·
JCT,
All the tests we do are for production copies of lenses. We should always be conscious of of expectations. Even at 24mm where the new 24-70A is weakest, on-axis it is very similar in performance to the 135L, a highly respected lens, and the corners are pretty comparable to e.g. the 14L II.
JCT ·
Brandon.. I wasn’t so much challenging that they were “production” level.. but perhaps “what” production level. We’ve often seen “early” production levels differ from “mass final go to market” production levels differ based (perhaps) on final tweaks coming from testers (like yourselves) and others. Clearly the lens was “later” than Sigma originally announced.. Could that have been a tweak to some coating process, perhaps. At this late stage, it would preclude a redesign, but in Sigma’s case could also mean some tweak in software as well.
Brandon Dube ·
We’ve tested over 4,000 lenses, ranging from the very first ones off the production line to ones built 15 years after a lens was released. We have never detected an optical difference between production runs. Slipping a month or two on release schedule is not very long, I doubt they made any significant changes to the design, it is far more likely they were waiting for shop floor time while a different lens was being produced. Most lenses are not produced continuously, and I would imagine the 14/1.8 and new cine lenses were having stock built up.
JCT ·
I appreciate that effort, but then that doesn’t account for how bad copies (and we know that there are those) within, at times, certain serial number ranges aren’t as good as with earlier or later versions of the same lens. I have no where near the experience level or detailed knowledge you have, but I also believe, given the chance, that “tweaks” can and are likely made between production runs.. just an opinion.. Your own methodology of testing augments that. You average the results of 10 lens’.. If there were no optical difference, just test one..
Brandon Dube ·
Those 4000 are just the ones we’ve tested rigorously. Lensrentals (i.e. not Olaf, with test charts not an MTF bench) has probably crossed 100,000 lenses doing that by now.
There’s nothing to tweak in optical manufacturing once you begin volume production. The traditional way carries too much cost and risk to change, and CNC is always best effort per-piece and there are no batches.
The mechanics could change, but the ones relevant to the optics never have. Just PCBs moving around or being replaced or upgraded, different seals, etc.
“bad runs” and serial number ranges either do not exist, or are so uncommon they are not worth discussing. We would have seen it in our results by now, and internally we track far more than we show on the blog.
Roger Cicala ·
JCT – these were copies bought from camera stores. New in box, readily obtainable, nothing special.
whereisaki ·
If only all lens testers got their lenses the way we consumers do. And none of this “pre-production” nonsense.
David Alexander ·
The 24/28-70/2.8 seems to be Sigma’s achilles heel. None of them (and there have been, what? At least a half-dozen over the years?) have been best-of-breed. It’s curious.
24x36 ·
Since they screwed up the ergonomics (on this and the last version), making the zoom ring turn in the opposite direction compared with all of my other Nikon mount zooms (ironically all Sigmas), I’m actually kind of glad it’s a dud. I wouldn’t have bought it anyway, and now I don’t feel like I’m missing anything. ;-D
So in my own twisted logic kind of way, thanks for the good news, Roger! ;-D
JCT ·
For anyone interested. A series of personal samples comparing my Canon 24 -70 2.8 (MK 1) to the new Sigma. About 60 RAW Files across varying subject and a willing spouse for some. 🙂 Legend (so to speak) are sets of Canon first at 24mm then 70mm at 2.8, 5.6, 11 then the Sigma of the same image (as close as possible) with the same sequence of shots. These were not meant to be portrait or landscape masterpieces, but a varied series of shots to see if I noted any of the issues pointed out here based on MTF testing. Personally I don’t think I did but leave any thoughts up to folks and their individual views. Again, no rulers to measure distances, no charts, just shooting the same things with both lenses. YMMV.. 🙂 The last three are just “Macro” shots with the Sigma..
https://my.pcloud.com/publink/show?code=kZOACcZOzBaf65fR2mF9meTnaobgJjtu537
Troy Tully ·
Did you notice that the Canon Photos had a much more pleasant background bokeh? Great job!
JCT ·
Troy.. First, thanks. I did notice the difference. Some shots more noticeable than in others.. Actually some where Sigma Bokeh was smoother, but haven’t made the association as to why yet. It isn’t f/stop. I don’t think I would term it “much” though. Example would be those bee shots.. nothing wrong with the Bokeh there. 🙂 Again we’re looking at it at 1:1 to truly notice the difference. If I’m looking for REALLY smooth bokeh, then I’ll use the 1.4 Art lenses. 🙂 A 24 – 70 isn’t a portrait lens.. It is an event photographer’s lens. Can it be used for portraits, of course, but nothing beats a prime lens if that is what you’re looking to do.. Again, thanks for looking..
ShitMyNameWontF ·
“sharpness tests” hahahh. more like “let’s look at charts and guess”!
Kevin Barré ·
Now that you have the Tamron SP 24-70mm F2.8 Di VC USD G2, can you give us an update on how it compares?
Roger Cicala ·
Kevin, I won’t get to test it for at least another week. I’m badly behind.
Lance Blackburn ·
Just goes to show how difficult it is to design a optically excellent across the rage 24-70 lens.
Bruno Vieira ·
Great post!! Any chance to add the comparison with the non-VR Nikon??
Someone ·
Would like to see the comparison with the Nikon 24-70mm E VR.
🙂
Steven Hyatt ·
Roger, I’m still shooting the older G version of Nikon’s 24-70 and it’s nearing retirement age. How does this Sigma compare to that one (the E Nikon is just too big for my taste)?