Lenses and Optics

Canon 24-70 Mk II Variation

Published November 4, 2012

A while back we ran Imatest resolution numbers on a few copies of the Canon 24-70 Mk II. That was right at release time and the resolution was most impressive, but we only had 5 copies available to test. Over the next couple of months a number of reviews came out. Most seemed to consider the resolution stellar, but a few found it average and several people seemed to have bad copies.

Of course all of this made me interested in checking the amount of copy-to-copy variation in this lens. We finally got caught up enough to run a reasonable sample size through Imatest evaluation, so I thought I’d share the amount of variation we’re seeing. Since most of the complaints I’ve seen online were at the 70mm end of the zoom we tested there first.

 

 

A word before we begin: if you aren’t familiar with the degree of resolution variation in zoom lenses don’t read any further. It will make you go insane and make a fool of yourself online. If you want some background, I would impartially and modestly suggest reading this first.

Resolution Variation

If you aren’t familiar with our Imatest results, the graph measures MTF 50 at the center (horizontal axis) and the average over the entire lens (vertical axis) reported in line pairs / image height. Each dot represents the best image of a given copy of the lens from a bracketed set.

The first graph shows the results of 70 copies of the 24-70 f/2.8 Mk II at 70mm (blue dots) and a set of 125 copies of the Canon 24-70 original version (red dots) we tested a couple of months ago. (Note: the axes do not go to zero to give a more expanded view of the points.)

 

 

A look at the graph shows a couple of things. Overall, the range of variation of the Mk II lens is about the same as the Mk I version. (And these are recent Mk Is after we learned how to keep the 70mm variation to a minimum. It used to be much greater.)

You can also see that three of the tested Mk II lenses seem to be out-of-sorts.  Even with 70 copies, though, it’s hard to be absolutely certain where the ‘acceptable’ cutoff should be by just looking at the graph.

For that reason we use SQF numbers to help determine where the acceptable cut-off should be. (If you don’t want to read the entire link, SQF basically tells us how big of a difference in resolution would be apparent in an 8 X 10 print.) Basically we read the SQF on the highest few lenses, and then put a cutoff where the lenses become 5 SQF units below the worst lenses. An SQF difference of 5 should be barely noticeable in an 8 X 10, high quality print.

SQF suggests that the lowest 3 copies of the 24-70 Mk II are noticeably less sharp than the best. In addition, the two lenses above those lowest three are right on the SQF 5 difference.

The bottom line is that 3 lenses out of the 70 copies were not up to expectations. That is a bit higher than the 2% unacceptable rate we usually see ‘out of the box’.

One other thing demonstrates the difficulty we have with one of the less-than-expected copies; they’re still better than the best of the 24-70 Mk I versions. They just aren’t as much better as they should be.

Comparing to the 70-200 f/2.8 IS II at 70

When I tested that first small batch of 24-70 Mk II lenses I said they were a bit sharper at 70mm than even the legendary Canon 70-200 f/2.8 IS II lens was. The 24-70 resolved 950 Line pairs / Image height at the center, and 810 overall, compared to the 70-200’s 885 / 765.

Of course, I wanted to repeat that comparison now that we have a reasonably large sample size (the original group was just 5 lenses). The average (mean) resolution of the70 samples of the 24-70 II is a bit lower at 940 / 800.

Since then, I can’t tell you the number of people who’ve commented, “but Bill tested a 24-70 against a 70-200 and the 70-200 was sharper.”  Yeah, I’m not surprised. Showing you the actual data makes it pretty obvious why I’m not surprised.

In the graph below the same 24-70 data points are compared to a run of 85 copies of the 70-200 f/2.8 IS II (red crosses) shot at 70mm.

 

MTF 50 data for Canon 24-70 Mk II and 70-200 IS II lenses, both at 70mm

 

If you pick one of our 24-70s and one of our 70-200s at random, there’s almost a 40% chance the 70-200 will have the same, or better, resolution.

I think most people realize there is sample variation. But this provides a nice illustration showing just how random a test report comparing just one copy of each lens can be. Don’t get me wrong; those reports are totally worthwhile.

But we have to be really careful splitting hairs with camera lenses. With 70 copies tested, I’m comfortable the average (mean) resolution of the 24-70 Mk II is slightly higher than the 70-200 f/2.8 IS II at 70mm. But that difference is much less than the sample variation

Addendum

I think it might help to demonstrate how big number differences above are fairly small image differences in the real world. Below are 100% crops of one of the highest rated ten copies of 24-70 IIs and one of the lowest 10 (not counting the bottom 3 that I called not OK). These are 100% crops of an ISO 12233 chart, something a pixel peeper could shoot at home. If you look really carefully I think you can barely tell the best lens. But it’s close.

Center of ISO12233 chart

 

Top right near corner of ISO12233 chart

 

For the record, there is a 110 point center and 80 point average difference in the two lenses, so they’re near the extremes of difference. You can see it if you pixel peep like this at 100% in side to side comparisons. But I think you’d agree in real photographs it will probably be impossible to see the difference, as SQF suggests.

BTW – if you feel the need to ask which was the highest and which was the lowest, then I’ve made my point.

 

Roger Cicala

Lensrentals.com

November 2012

Author: Roger Cicala

I’m Roger and I am the founder of Lensrentals.com. Hailed as one of the optic nerds here, I enjoy shooting collimated light through 30X microscope objectives in my spare time. When I do take real pictures I like using something different: a Medium format, or Pentax K1, or a Sony RX1R.

Posted in Lenses and Optics
  • pieter kers

    Roger, thank you for responding;
    Yes that is what i meant.
    – If the lens that is best at 70mm also is the best ( or almost) at the other focal lengths…
    It would be interesting to combine the results of 4 ( 24-35-50 and 70) focal lengths and see the variation..
    (4x 70 tests!) but would be interesting…

  • Roger Cicala

    Pieter,

    I’m not sure I understand the question. If you’re asking is the best lens at 70 also the best at 24, then the answer is no, it is not.

    Roger

  • pieter kers

    Roger,
    many thanks for testing 70 (?!) copies

    My question is ( hope it is new) does the best 70mm result combines with the best 50 and 24mm?
    or …

  • Mark

    Roger (I almost said “Dude” cause I love you lol) you do yeoman’s work. I so look forward to your reviews. Keep up the fantastic work. Oh, btw, I sold my MKI for the new version. I like the lighter weight. Offsets the extra weight of my new 1Dx.
    Mark

  • I will probably get the Tamron for now until I can afford the new lens. VC will be nice, but I don’t really need it as much as I would like the IQ of the canon.

  • Roger Cicala

    Glenn, it’s right between the two Canon 24-70s.

  • I would love to see where the Tamron 24-70 VC fits in here

  • Roberto

    Roger, if I can get the spreadsheet I can try to play a little with it. No promise I’ll be quick though.

  • Mus

    Hi Roger,

    Excellent website and very useful information for photographers. Thank you.

  • If the same excel sheet would have a column with the year of manufacture that would be a great thing to check for as well.

  • This is fantastic – the data methodology that is.

    If you could upload an excel file for both lenses at 70 with 70ish observations each – that would be fantastic.

    Thanks

    Asad

  • Roger Cicala

    Pertinent point, Roberto. I’m not sure of the appropriate distribution tests, but I’d be happy to furnish the data spreadsheet if you want to refine the math. I’d like to know myself.

    Roger

  • Roger Cicala

    Hi Steve,

    At my testing differences it was definitely sharper in the corners and sides at f/4, maybe a bit more in the center. I saw a tiny bit more improvement in the corners at f/5.6. That’s usually the max sharpness level, I didn’t test further.

    Close up it might be a bit better, but I agree with you: for what you shoot (and what I shoot) I may well prefer the f/4.

  • Steve

    Roger,

    Love your analysis of multiple copies – it sure explains why for some lenses one review website can reach a vastly different opinion than another equally reputable website. Having that cloud of data points really tells the story. My question is – it’s not explicitly stated in your discussion, but am I correct in assuming that all of these comparisons are only measured with the lenses at f/2.8 aperture? My two main photographic interests tend to be close ups, ie flowers, and landscapes. Other than cases where I trying to get good bokeh for an isolated flower shot, I usually stop down to improve the depth of field to get an entire flower in focus, or for landscape, again stop down for better depth of field. I was wondering if you could give your opinion as to, on average, how much sharper, if any, the 24-70 II would be stopped down to say f/8.0 vs the Mark I @ f/8.0, or for that matter vs. the 24-105 f/4.0L, the lens I currently have, which came as the kit lens with my 5D MkIII. I have been waffling on whether to upgrade to the new 24-70 f2.8II somewhat based on the mixed early reviews, but based on your cloud of data, looks like it would be a safe bet to take the plunge. The news that Canon is reportedly going to have a December release of a new 24-70 f4.0L with 4 stop Hybrid IS and close to macro MFD, has further muddied the waters for me. I’ll probably hold off any upgrade decisions until you and others have weighed in on that lens. Maybe I’ll just have to rent both the 2.8 and the new 4.0 from you guys early next year to help me make my decision. Thanks again for all your very useful blogs – keep up the great work!

  • Roberto

    Great test. But I don ‘t necessarily agree with the statement “Overall, the range of variation of the Mk II lens is about the same as the Mk I version”. The absolute variation is the same, but relatively to the average the Mk II has a 25% of variation while the Mk I has a 33% variation. One should really look at the distribution tough (is it gaussian, what is the RMS?), maybe the Mk I is larger in that plot just because the sample is bigger.

  • It is fantastic that you (a) have a zillion lenses to test and (b) you have enough background in the experimental sciences to present and analyze the data correctly!

    Keep up the good work!

  • Ian Anderson

    The ISO 12233 charts at the end are priceless! Thanks for putting this whole discussion in context. I think it also goes without saying that these differences only matter if you’re using perfect technique – i.e. solid tripod, mirror lock up, cable release, etc…

  • Roger Cicala

    Hi John,

    There’s very little correlation between the two ends, except when a lens is obviously awful at one end (I’m talking defective awful, not just lower end of variation). There’s a good chance (over 50%) that it’s bad at the other end. What shocked me at first where the number of lenses that would be totally unusable at one end after a drop or failure of some sort, but still fine at the other end.

    But having high resolution one end of the zoom range doesn’t correlate much with high resolution at the other.

    Roger

  • John

    Roger,
    Do zoom lenses that have higher SQF scores at one end of the zoom range than others in the batch tested tend to have similar higher SQF scores at the other end of the zoom range? In other words, if you picked one of the highest scoring 24-70 IIs based on the test at 70mm would you have any confidence it would also be above average at 24mm and 50mm?

  • Samuel

    Thank you for the insightful advise and points Roger. It is indeed true that the proof of the pudding is in the eating. If I am constantly doing near infinity landscape scenes, it would be unfair to judge a lens based on near-distance tests and Imatest results. I am reminded of your article on testing macro lenses at normal shooting distances, where you noted that at macro distances, resolution figures improved dramatically.

    Still, it is interesting to know the numbers. What is nice to see and know is the ‘spread’ of all the data points. It translates into a useful piece of information – whether the TS-E is more consistently built than the 24-70 II, implying one is more likely to get a good copy. The results from 70mm at the moment don’t really inspire confidence!

  • Roger Cicala

    Sam,

    The trouble is I’d be selling you the best lens for my test camera shot at 18 feet. If I repeat the run on a different camera the graph looks the same, but the individual points all shift around – the highest and lowest may be more in the center, some of the center will be higher and lower.

    If I change setups to a different size target and shoot it at 30 feet or 8 feet, the results would be slightly different, too, and Imatest can’t test at infinity. I could test the lenses at infinity on an optical bench, but we’d still have the camera-to-camera variation thing.

    It’s like the Heisenberg principle: there’s a limit to how accurately we can describe a lens.

    It’s a funny thing for an optical geek like me to say, but putting the lens on your camera and shooting what you usually shoot is the ultimate test. My numbers are capable of finding bad lenses and eliminating them from the herd, but not which of the herd is slightly better on your camera.

  • Roger Cicala

    Samuel, I don’t in large numbers yet. It takes a couple of days to do a run of that size. I will soon.
    Roger

  • Samuel

    Roger,
    Do you have the data of how the 24-70 II @ 24mm compares to the 24mm TS-E II?

  • Sam

    Hi Roger!

    Brilliant work as usual!

    You see those blue dots that score exceedingly well? Ever thought of re-selling those copies at a premium over the $2299 retail price of this lens? You’ve done the testing and take the stress out of the question as to whether one has purchased a “good copy” or not. You could sell it for say $2499 with a “LensRentals.com Tested Sharp” certificate. Maybe the premium price also guarantees the buyer up to two adjustments/cleanings/repairs by LensRentals.com? At the very least, the certificate would permit the buyer to charge a premium during re-sale.

    Just a thought for additional revenue for an exceptional website and small business.

  • @Kai,

    No — these results show the spread of lens sharpness across multiple lenses (70 of them), not multiple autofocus trials on the same lens.

    I am pretty sure that Roger is reporting the best results for each of the 70 lenses tested, i.e., autofocus accuracy has been eliminated from the comparison.

  • Steffen

    “If you pick one of our 24-70s and one of our 70-200s at random, there’s almost a 40% chance the 70-200 will have the same, or better, resolution.”

    The above means actually:

    “If you pick one of our 24-70s and one of our 70-200s at random, there’s almost a 60% chance the 24-70 will have better resolution.”

  • Dr Croubie

    Ah, sample size variation.
    100% of the people I surveyed in the 3 minutes before making this post have no idea what the hell it even means (even after i explained it to her).

  • Kai

    Roger,

    How about the wide end?

    So this would indicate that the MkII doesn’t have the rotation sensor to improve the focus accuracy, since the variation is similar to the MkI and the 70-200 MkII, correct?

    Thanks for putting the info up here.

  • Arun

    I hope this helps people understand things like political opinion polls as well. (sorry, but it is that season).

    I really appreciate the effort you have put into these tests, this is a great service to the photographers’ community.

  • simon

    so if you were very lucky when buying a mk1 and now want to upgrade because the mk2 should be so much better but this time your completely out of luck you pay $2299 for very little improvement. I wonder whether for this price one could expect better quality control (I know probably not cheap but it is a very expensive lens)

    and thank you for providing us with such large sample sizes. as you write it is obvious that there will be variation but it is very nice to actually see it.

Follow on Feedly