fbpx
Geek Articles

Is Your Camera Really The Best Optical Test?

Published September 13, 2016

“But you must not eat from the tree of the knowledge of good and evil, for when you eat from it you will certainly die.” Genesis 2:17.

I have enjoyed, for some years now, the process of learning about lenses and optics. This blog shares a lot of what I’m in the process of learning. A lot of you like reading about what we’re doing and finding out what we’re finding out about what is. A lot of others don’t like it because it upsets their preconceived notions of what should be.

This post is going to be more polarizing than most. I’m going to talk about the very real phenomenon that people are describing on the internet; that they’re seeing more lenses with optical problems. If you aren’t interested in that, or are comfortable just saying they’re all bad photographers, this will be a long, boring article you want to skip.

85testcenter 2

 

One of the more common responses on various internet forums seems to be “this manufacturer or that needs to up their quality control.” There’s a bit of truth to that. A slightly less common response has been “your manufacturer, but not mine, can’t make their lenses without a lot of variation.” There’s a bit of truth there sometimes, but not very much. The broader truth is we’re observing an industry-wide issue. It’s not happening because lenses are worse. Lenses are actually better than they’ve ever been overall. There are some exceptions, of course.

For those of who don’t want to read further, here’s the summary:

  • The optics of new lenses are designed to take advantage of high resolution sensors. They are sharper than they were a decade ago.
  • An optically misaligned lens is just as soft in the bad area no matter how good the design. But the difference between the good and bad areas may be greater.
  • Opto-mechanical design and sample variation isn’t really much different than it was when the best cameras were 16 megapixels. (Some manufacturers have made improvements, but not most.)
  • Cameras resolve far more than they did a few years ago. They can demonstrate lens flaws you might not have seen with your last camera.
  • Metrology (optical testing and measurement of lenses) is the same as it was decades ago (There are some exceptions, particularly Sigma). In many cases you may see defects in your pictures that the service center can’t see with their testing.

In a nutshell, you can now take a picture with a high-resolution camera and see defects in the lens that might not have been apparent on your last camera. The manufacturer or service center often still use crude optical tests that don’t show the defect.

Why Higher Resolution Makes a Difference

Yes, I know this is obvious, sort of. But still, it’s worth discussing.

First, let me say that optical physics guys can do all kinds of math and show that, particularly if you involve vague terms like ‘depth of field’ and ‘resolution’ any statement about resolution is either right or wrong. I’m going to stay out of that and try to limit this discussion to words, pictures, and common sense. We’ll start with a simple, common-sense analogy. Things are way more complex than this, but it gives an idea of the overall picture.

Looking at Points

We all understand that a lens resolves a point in the world around you into a point on the sensor. If we look at a 5 micron point projected through a good lens mounted on our Olaf Bench, the point looks like a point, at least in the center of the image.

point

5 micron point source seen through the center of a really good lens.

 

Out towards the edges, where aberrations are warping the light rays around, that point of light looks somewhat different.

edge

The same point source, seen near the edge through an excellent lens.

 

Now remember, that’s a magnified view of that point source I’m showing you; we’re basically looking at the image the lens makes magnified by a microscope. Let’s pretend that the white box in the image below represents a 6.25 micron pixel from a Canon 5D Mk III (22.3 megapixel full-frame sensor). Despite the fact that the dots look different on the microscope of my optical bench, they may look like perfect dots to your camera .  .  .  . or nearly perfect. (And again, I’m leaving out details about AA filters, Bayer arrays, etc. but the analogy holds if we made this a 4 or 16 pixel array instead of a single pixel.)

6mics

 

This little analogy should explain why I often say, “Yes, there’s a difference in the optical bench tests, but I doubt you’ll see it on your camera.” Those dots are clearly different and the MTF taken at those two points will be different. But basically either one was going to fill up 1 pixel on my camera, with maybe just a tiny bit of bleed over from that lateral point on the right. The optical bench might see the difference, but your camera sensor probably wouldn’t.

But now many of us have a  40- or 50-megapixel cameras. Which means our pixel pitch is now 4 microns instead of 6.25. What the lens rendered as a 1 pixel dot now is bigger than a pixel. Still, even that edge pixel looks pretty good, with just a little spill into adjacent pixels (or array of pixels, whatever your pleasure).

4mics

 

But that was a really great edge pixel from a $5,000 prime lens. What about one from a more standard looking lens? Well, this edge pixel from this zoom lens is smearier and has a more noticeable chromatic aberration. And you should get the idea that it might be more apparent on a camera with smaller pixel pitch, as represented on the left below.

badpixcomp

 

My analogy might be a little clearer if I average each of my pretend pixels from above, since my allegorical camera would only be reporting one value for each pixel. (Yes, I’ve invented color sensing pixels and eliminated the Bayer filter for this allegory, but if you prefer, consider each box to be a 4-pixel array complete with Bayer filter).  But the point is simple: higher resolution means you’re going to get better detail about how smeary that dot is. In this case, either camera would show you this lens has lateral chromatic aberration, but the higher density pixels (on the left) also show the complex coma-type aberrations a bit more too.

badpixcompavg

Yes, I have mad Photoshop skills

 

Depth of Field

Depth of Field is one of those interesting phenomenon that’s both very scientific and very arbitrary. It’s determined by a complex mathematical formula, so it’s seems uber scientific. The key definition in determining depth of field (and some other concepts) is the Circle of Confusion (CoC). The definition of the circle of confusion (in words) is the width where the point blurs enough that it is larger than some maximum allowable diameter.

Depending upon your purpose, ‘maximum allowable diameter’ can have different definitions, and therefore depth of field calculations that come out of the formula can be very different. For an SQF calculation you might want to make it larger than some point size on a final print of a certain size. When we’re working with camera sensors, reasonable definitions might be the width of a pixel, or a 4 pixel square, or a 16 pixel square. Whatever formula you want to use, though, a smaller pixel means a smaller circle of confusion, which means a narrower depth of field.

Now, you may ask, why does that matter? Well, one thing people are seeing more frequently is a tilt in their lens. (In forums people call every lens defect “decentering”, but optical decentering often isn’t involved. Tilt and spacing errors are at least as common as optical decentering.)

Let me show you the effect of depth of field in two field-curvature graphs of the same lens, which has a small tilt. With a larger depth of field (on the right) there’s no way you’d notice it on any type of test chart testing; focus on the center and the left and right sides are going to be equally sharp. With a narrower depth of field (on the left), you might start to notice one edge is softer than the other if you were testing on a chart, brick wall, or other flat surface.

tilts

For this purpose, imagine the black line is your test chart, carefully focused in the center of your lens. The colors (and numbers) show the actual MTF measurement you would see at all focus points even if you had focused past or in front of the actual chart.

 

The drawing is a bit exaggerated since I basically doubled the depth of field by stopping the lens down (which is why the MTF is higher on the left). With the wider depth of field, though, the left edge MTF is 0.82 and the right 0.77. Those numbers aren’t very different and most of us would agree that was normal. With the narrowed depth of field, however, the MTF on the left edge is about 0.84 but on the right edge it’s 0.57. If I give you those numbers for your lens you’ll understandably scream that it’s awful and decentered. (Of course, it’s actually tilted, but people on the forums always scream ‘decentered’. It’s like playing Lens Bingo.)

This was probably pretty obvious without the illustrations, but the bottom line is if you move from 20-ish megapixels to 40- or 50-ish megapixels, you’re more likely to see flaws in your lens if you look for them. This has happened to some excellent photographers I know. They have noticed a lens here or there that was fine on their 5D Mk III or A7 that has a weak side or corner on their 5Dsr or A7rII.

So Why Doesn’t the Service Center See This?

This section is not going to name names. I’m under nondisclosure agreements with any company we help with testing so I’m ethically and legally unable to share names. You can take it on faith that I’m sharing facts or feel free to think I made it all up. I would point out, though, that for 10 years I’ve only made stuff up on April 1 and this is September.

The summary is almost all of you greatly overestimate the type and amount of optical testing that lenses get, whether it’s at the factory after assembly or in the repair center when it has a problem.

A lot of repair locations literally do resolution testing on an 8 X 11 or 13 X 19 ink jet printed chart. One used pictures of a bookshelf across the office to do optical adjustments on very expensive lenses (I know because they left their memory card in the camera they insisted we send in with the lens). AF 1951 charts are still commonly used (and remember, the ‘1951’ is the year the chart was developed, which means it was designed for film).  Some use large, high-resolution ISO 12233 charts, but not many.

Factory Service Centers and Factory Authorized repair facilities generally use a factory-specified graph, again often printed at ink-jet resolution. It’s shot with a camera hooked up to adjustment software that gives pass-fail readings. This isn’t necessarily bad, although it’s not great. And depending on the factory, that chart may actually not be as good as an AF1951 or ISO12233 chart.

Some service centers use Lens Test Projectors and a center-only collimator to do testing and adjustments. If you ask a manufacturer’s engineer what the gold standard of testing is, with few exceptions that’s what they’ll say: a center-only collimator and a lens test projector. (Trust me on this one, I’ve had several say just that until we showed them why it wasn’t). If you ask them how long has that been the standard, though, they’ll tell you since the 1960s. That means since film.

Cooke Lens Test Projector, courtesy Cook Optics

Cooke Lens Test Projector, courtesy Cook Optics

 

Don’t get me wrong, a lens test projector and centering collimator is really good testing; generally better than test charts. We’ve used them in the past. They’re still the gold standard for Cine lenses everywhere. But Cine lenses are resolving 4K (just under 9 megapixels) or perhaps 6k (about 19 megapixels), not the 40- or 50-megapixels of a high resolution SLR. (BTW — I’ve got a couple of lens test projectors and centering collimators sitting on shelves in the back gathering dust if anybody wants to buy one.)

Things may actually be a little worse than what I’ve described so far. Remember, the ‘spec’ of ‘in spec’ is whatever the manufacturer says it is. A wide-angle lenses has a field of view of over 50 degrees to each side, for example, but I know of two manufacturers that don’t test wider than 30 degrees. They feel that anything significant will show up by that angle. That’s not my experience, but that’s what they say. Another does all of their testing at f/4 because that’s the standard for their automated testing software. And yes, a whole lot of problems that show up at f/1.4 disappear at f/4.

For example, scroll back up and look at the introductory picture, the crop from the center of a test chart. That lens was sent back from a factory service center twice, the last time with a note that said it met manufacturer’s specifications, no they couldn’t share those specifications, and there was nothing else they could do. It’s fixed now, but not by them.

My point here is not to get you all gathered together with pitchforks and torches to go storm the castle. This wasn’t a plot, in my opinion, it’s simply a lot of inertia. Metrology isn’t sexy and doesn’t make money. Until recently complaints about lenses weren’t any more frequent than they were back in 1990, so there was no real motivation for change. And I do want to repeat, the reason I know some of this stuff is because a number of manufacturers and service centers are suddenly and rapidly trying to improve their testing because they realize it now is a problem.

Better Metrology is the Answer, Right?

Well, yes, of course, but there’s another problem with better metrology: it’s too much better. I’m going to pull back the curtain a bit and show you things you don’t want to see. For years now, you’ve been seeing my summary MTFs, the average of 10 lenses, each tested at 4 rotations. So the MTF you see shows either the average or the range of 40 measurements at each point. It’s sometimes more accurate than the computer generated MTFs you see listed with a lot of lenses, but it’s still a summary.

Olaf Optical Testing, 2016

Olaf Optical Testing, 2016

 

We do 10 or more copies and come out with an ‘expected range’ for a given lens. Then we can plot a particular lens (the lines in the graph below) against that range and find out if it’s acceptable. The one below, for example, shows as about average or maybe a little better.

Olaf Optical Testing, 2016

Olaf Optical Testing, 2016

 

That’s good, but remember, even the line above is the average of the 4 rotations and two sides of the tested lens lumped together. What I don’t show  you very often is what a single lens looks like at 4 rotations because you’ll get upset.

Let’s look at the raw measurements for that went into making the lines for the lens above: this is the full MTF at each of 4 rotations; “0 degrees” is side-to-side as mounted on a camera, “90 degrees” is top-to-bottom, and the other two are diagonals.

_MTF_Rotation

 

That’s not quite as smooth and beautiful, is it? That’s reality, my friends, for basically ever lens we test (and while I’m not identifying the lens, this is a really good prime lens). I expect you’re now wondering, after looking at the 90 and 135 degree rotations above if one corner of the lens is significantly softer than the rest. Obviously it is on the MTF bench. But will you see it on your camera?

Just to give you an idea below are 4 more copies of that lens, hand selected to be among the best from 20 copies or so. Even with these hand-selected  best copies of prime lenses there is, if you look, always one corner that’s a little softer or more astigmatic than the others. But none are quite as weak as the one above. But most of the 20 looked about like the one above?

 

allmtfs

 

Unlike chart testing or even lens test projectors, optical bench testing shows us MORE resolution than your camera can see. At this degree of examination, no lens we’ve ever tested is perfectly identical in all 4 rotations. That’s many thousands of lenses tested. And none that were perfect. Ever. The fact is, those 4 lenses in the graph above are as close to perfect as we ever see.

This is why, when people suggest I cherry pick them a perfect copy, I tell them I’ve never seen one. With better lenses, mainly primes, we expect ‘so good that it looks perfect on your camera’. For most lenses we are looking for ‘within the expected range we see for this lens after eliminating bad copies’.

Let’s go back to the examples above. That lens on top, you may have noticed, is indeed a bit worse than the 4 ‘best possible’ copies out of 20 I picked out for the second image. The worst copy, the one on top, was an ‘average’ copy of this lens. Was it acceptable? Actually it was. It rented 20 times and no one ever had the slightest complaints about it. After every rental it was tested on a high-resolution, oversized chart and it looked fine.

Even when we made a point of looking for the weakest area on test charts or in photos, it looked fine on a 5D III. If we put it on a 5Dsr, though, we did agree we could see a little weakness on test charts. We thought.

My point here is that when we have a ‘too good’ test, so within that test we have to decide ‘where is the cutoff.’ In other words, we still have to decide ‘what is in spec.’

I’ll pause now for 476 of you to say “well, for that kind of money they should all be perfect.” Because I know you want to. Even those of you who know it’s not possible. And then I’ll say like I always do, “that was just a $1,5oo prime.” Because you can’t get perfect with $15,000 primes. The goal is not perfect. The goal is close enough so you can’t see any different in the picture you take.

So What’s the Bottom Line?

It’s pretty simple, actually. New cameras are showing defects in lenses that manufacturer’s weren’t quite ready to deal with. They (and we) are regrouping to put more adequate testing in place. We had a bit of a head start because we have been doing higher level testing for a while. We’re still, for example, the only place on the planet (as best I know) that can test electromagnetic focus lenses on an optical bench, for example, although at least one manufacturer will be doing it in a few months.

But we still don’t know exactly where the cut-off should be. We test lenses that go into satellites and stuff. Those aren’t perfect either, but in those cases the engineers who designed the cameras know exactly what resolution the camera can resolve and tell us so we can say a given lens is OK or not OK. The manufacturers, if they want to, can do that same thing. And at least in some cases they are starting to. But we’re all going to have to live through a transition period while those changes are made.

And I want to make it clear that we’re struggling with this, too. Months ago I tested a lens for Fred Miranda. I correspond with Fred regularly and I know he is an exceptionally careful photographer and will notice any (and I do mean any) flaw in a lens. I sent him a lens that passed with flying colors. Sure it had a slight abberation in one corner, but it was well within ‘my spec’. Fred sent it back after one day’s shooting for a soft corner — and absolutely identified that same corner I had passed as ‘good enough, you’ll never notice it on a camera’. My ‘in spec’ standard for that lens was developed when a 5D III was high resolution. He had put the lens on a much higher resolution camera and was noticing something that wouldn’t be noticeable on a lesser camera. (To be honest, it probably wouldn’t be noticed by most photographers, either. But Fred has bionic MTF eyes, I think.)

When high-resolution SLRs came out all of us, myself included, were wondering which lenses would let us maximize all that resolution. Few of us; not me, not you, and not the manufacturer, worried that that high resolution would let us see the weaknesses in a lens. Crowd-sourced complaints, legitimate complaints, have gotten their attention (and mine).

Things will get better in a couple of years, because changes are being made, but it’s not better right now. I know you want instantaneous changes, but they aren’t possible. For example, we took the decision a few months ago to improve our standard testing – the optical bench is excellent, but it takes 10 or 15 minutes to test a lens. We can’t run 800 lenses a day through it. But even our custom-made, high-definition test charts are barely adequate for testing the best lenses on the best cameras. We’re having to develop, in conjunction with a metrology company, new equipment that can be as accurate as an MTF bench but fast. That equipment isn’t cheap (think small house or maybe a Bentley) but more importantly you can’t just order one from Amazon. They’re built to order and it will be months before it’s up and running.

And remember, we’re a small company that responds to change quickly. Big companies have 12 layers of management approval and budgets made out a year or two in advance.  The testing department is managed by an optical engineer who is human, and humans are often a bit resistant to change. There’s accountants who are going to want to know why everything is costing so much, especially because this wasn’t in next years budget. The companies are aware of the problem, they’re addressing it, but it’s going to take them some time to make changes.

And, as best I know, everyone is still figuring out the best ways to test lenses to make sure we catch every defect that might be visible on a photograph without all the ‘noise’, if you will, of false-positive results. Remember, we’ve never seen a perfect MTF curve on any lens, but we’ve seen lots of lenses that are photographically perfect on the best cameras. Some experimenting is needed to figure out exactly which things are significant and which aren’t. (If you’ve read this far and still think ‘they should all be perfect’, I don’t know what else to say except reality sucks, doesn’t it?)

For example, we’ve just started doing field-of-focus testing on lenses. It may well be that this is going to correlate better with the defects you can see in a photograph than actual MTF tests do. We’re still deciding what frequency of MTF test is most appropriate for newer cameras. Historically, 20 line pairs / mm correlated most closely to what you see when you take a picture. In higher pixel-density cameras we know 30 lp/mm is better and we’ve moved to that. But we aren’t sure if 40- or 50 lp/mm might be necessary. Those have a lot more false positive results, though, which makes things more difficult.

In the meantime, those of you out there who have been saying, “just go take some pictures and see if they’re OK.” Well, right at the moment, you have the correct answer, at least for people shooting the highest resolution cameras for a photo. A lot of photos of different subjects looked at carefully are probably going to identify things we’d be arguing about in the lab.

And this is coming from a guy who spends all day, every day, testing things optically in a lab. Right at the moment, ‘take more pictures’ is correct and I am, well, less right. But give me a few months to experiment and few hundred thousand dollars worth of new testing equipment and I’ll be righter.

 

Roger Cicala

Lensrentals.com

September, 2016

Author: Roger Cicala

I’m Roger and I am the founder of Lensrentals.com. Hailed as one of the optic nerds here, I enjoy shooting collimated light through 30X microscope objectives in my spare time. When I do take real pictures I like using something different: a Medium format, or Pentax K1, or a Sony RX1R.

Posted in Geek Articles
  • Joe, I wish I did know more (and Aaron probably does but he’s out of town). But we don’t work on them very often, and as you’d expect, they are different inside just like they’re a different beast to master.

  • Joe Schmidt

    My service case is an example for Roger’s key message: problems increase if high-resolution SLR meets traditional lens manufacturing and quality control.

    I sent in an AF DC-Nikkor 135f/2D. Two years old, in production since 1991 (with minor modifications) and still the only Nikon 135mm AF prime. If the photographer is able to master the challenges of the DC-machanism and the characteristics of the screwdriver AF this lens is very good on a D750 and also acceptable on a D810. This is probably the reason why Nikon still dares to sell this old-style lens for $1,200
    nowadays.

    I’m sure that the weak upper right edge of my lens would never be apparent in analog era. But with current cameras you clearly see the softening of a pupil if you focus & recompose the picture about 20% towards the critical edge. It would be very interesting if the manufacturing specifications and tolerances have changed when this lens was placed in the digital era. I know nothing about the optical adjustment possibilities of this lens and if the service center statement “no centering possible” is really true. Perhaps Roger knows more?

  • So if pixel size on a 5DS is so small that it can show previously unknown lens flaws, what happens when you put the same EF lens on something like a 20MP m4/3 camera?
    And how hard is it to design lenses for even smaller sensors (at least if they’re big enough that you still have to use glass lenses)?
    I imagine something like a speed booster would help because it makes the points “more pointy”, but surely it still has to be a bigger problem than anything that FF sensors have to face?

  • Sell it and buy a new one. :-/

    My backpack is pretty much a continuous revolving door of lenses for this reason. Trying to get stuff calibrated by manufacturers is a crapshoot, and it costs more than selling/re-buying the same lens. I buy all my stuff used, so I don’t take much of a hit when I sell it.

  • What manufacturer was this? That’s bad enough that they deserved to be named.

    I sent a 16-28 to Tokina no fewer than five times for a soft corner. After the fifth time I just sold it and bought a Nikon 14-24 (which I adapt to Canon, which is a bit kludgy, but such is my level of desperation), which has proven far sharper and uniform.

    Most of what I shoot is star fields with wide lenses and large apertures. Any decentering/tilt will turn up in those shots. I’m pretty much the worst-case user for manufacturers…

  • Carleton Foxx

    I’m skeptical that a customer-hating company like Nikon would ever do anything to help improve their repair experience. But I’m usually wrong about everything, so hopefully this is one of those times.

  • Brandon Dube

    Right. Wavefront testing gives you a lot of information. You can look at the zernike fit of the wavefront and perform pass/fail based on any one aberration. On-axis things are stable; coma and spherical will vary a bit,but that’s it. Off-axis, whether a lens has dominating coma or astigmatism or oblique spherical can change dramatically between samples of the same model.

    If you want to use wavefront data for an overall metric, you can do RMS wavefront error or PV wavefront error. RMS WTF is analogous to strehl ratio, which is essentially the integration of a diffraction limited MTF curve, divided by the integration of the mesaured MTF curve.

    “the point” of MTF is that it sums all image-degrading effects into a single function. If you take an image of anything more complicated than a point or maybe a line, you can’t distinguish aberrations from each other without being a real expert at interpreting images. To that end, we don’t care what softened the image. MTF tells us the image was softened, not by what; we don’t care by what, and it is (very dramatically) slower to do wavefront testing. It’s also more expensive, and you can only test one field point at a time for most equipment. The test operator must also be more skilled to produce an equal quality of measurement compared to an MTF bench.

    MTF testing gives you the same “single shot pass/fail” with no iterative testing at higher speed and lower cost. It could be useful for adjustment, but the OLAF bench that lives next to the MTF bench is very good for that, and a hell of a lot faster than a wavefront sensor.

  • Jim A.

    Defects can be visualized with a wavefront analyzer, and different defects produce characteristic outputs. The wavefront sensors we utilize do not analyze a single point, they analyze an area. They could be scaled to analyze a larger area and produce results for the full image circle of a typical camera lens. If that lens differs from an ideal example, it will show up. If you’re trying to do quality control, you’re looking for deviation from some standard “acceptable” result. You don’t have to measure MTF, although I’m not arguing it is an improper test. All I’m doing is adding another possibility to the discussion. If lens A is the lens that produces my reference wavefront, and I put lens B in place, there will be some difference between A and B. If that difference exceeds my tolerance, then lens B gets attention to fix the defect. If not, lens B goes on the shelf. All I’m suggesting is wavefront analysis might eliminate the iterative testing of each test subject and make it a single shot pass/fail detector. Thus improving the utilization of scarce and expensive skilled labor by having those people work on lenses with known defects.

  • Brandon Dube

    There is no need for a wavefront sensor; the image quality is what is important. MTF informs to the image quality; a wavefront sensor tells you how the image quality came to be that way, which is unimportant.
    They also only function for one field point, which is not very helpful.

  • Jim A.

    I wonder if a large aperture wavefront analyzer would enable quick assessment of lens quality. We use them to assess and measure small lenses and human eyes. It should be possible to scale one up, at probably great cost and effort, but it would give you a pretty accurate snapshot. You can see higher order abberations pretty quickly. Well, the computer and analysis software can tell you about it, to be technically correct.That might fall into the same trap though, it might be too sensitive. But if it was, you could define “acceptable” in software and let the computer reject the ones needing attention. We only use a single wavelength laser, so chromatic aberration assessment would need a more complex lighting scheme. But, you’re probably not going to correct that very much anyways, right? No doubt smarter guys than I are working on it your metrology challenges. Thanks for the great, well written and informative article.

  • Joshua Efron

    Thank you. This blog continues to be one of the most interesting things on the internet, and the work you do is the reason I will choose to purchase a used lens from Lens Rentals over a new lens from the manufacturer.

  • Jon, email me and let me know what you’re interested in and I’ll send you some pics.

  • Jon Gilchrist

    How much for the old test gear? I do a lot with old film lenses.

  • Joe, I agree. I don’t know first-hand, I’ve never worked with them, but I was told for many years Nikon had the only testing system that actually used multiple collimator sources to test optically on-and-off axis when they made adjustments. I tend to believe this was true because I’ve talked to many people who knew of it, and because Nikon optical adjustments were so routinely excellent. I think eventually everyone will go to such a system. It’s expensive (several hundred thousand USD) but there’s no choice.

  • Joe Schmidt

    Roger, thank you very much for your great article!

    In fact, it’s disconcerting what factory service centers are actually doing and – in some
    cases – communicating to the customer. Last month I sent a $1,200 prime with an obviously weak upper right edge to factory service center.

    Answer: “The lens is in technically flawless condition and its exterior was cleaned. The imaging characteristics were tested with practical trials and on optical test benches. Everything is within manufacturer specifications. See attached printout. Further centering
    is not possible for this lens. As a gesture of goodwill: free of charge.”

    If you expect a “printout” of one of the “optical test benches” you will be
    disappointed. It’s a color laser print of a test chart photo taken with the
    lens. Considering the 133lpi print-raster the provided picture has 1.3 megapixel.
    Probably, that’s not a good basis convincing the customer that “everything is
    within manufacturer specifications”.

    My concern: the staff of the factory service center isn’t able to use even the
    state-of-the-art testing equipment properly. Furthermore, lens adjustment in
    general seems very unpopular work for service centers. Your lens teardowns are
    telling us why…

  • April 1. I promise. 🙂

  • Athanasius Kirchner

    What a disappointment. Couldn’t you make some great story up, involving spies, the CIA, kidnappings and the like? 😛

  • Adam Fo

    Reminds me of when I used to work at a cine rental facility in the UK in the early 1990’s.(Joe Dunton then Panavision) Before sending zooms out you would put them on the collimator, make sure it holds focus zooming and focuses to scale. Occasionally you would project a test chart onto the wall.
    On a feature film the crew would come and test including using harps to test acceptable depth of field etc.
    What normally happened on the first day of shooting ? The DOP asks for the lenses to be netted up on the back or decides to shoot the whole film with a light diffusion filter on the front. Because of motion blur and the fact you seldom filmed flat objects you didn’t need particularly sharp lenses.
    Did it ruin some of the great films of the period, of course not….

  • bdbender4

    Roger, amazing (uh-MAAAY-zing!) once again. I wonder if/how the camera makers are twigging to this? Nikon with it’s new lens design paradigm? Canon with it’s new attention to manufacturability? Making 23,200 lenses per day to accommodate this level of scrutiny is a pretty significant challenge. I wonder if/when the flatness of “full frame” and medium format sensors enters into this type of analysis as an uncontrolled variable? I wonder what this might have to do with Zeiss’s microcontrast? (I also wonder why the mainstream media spend most of their time like a flock of chickens, running from yesterday’s “dumpster fire” to the most recent one. But I digress.)

  • Andrewh

    The testing and quality control may not have a significant impact on the price but surely lenses that resolve more and also have fast apertures are going to be much more expensive than the older designs, many of which are still sold today.

  • yaley, I don’t think it’s going to be a significant cost, but I’m not in a position to say. But I haven’t seen anything to suggest it.

  • yaley

    Is this upgraded quality control leading to higher priced lenses? I guess it does so..
    But you said some manufacturers are already trying to do this. Are their lenses getting more expensive?

  • kimH

    Thanks for – yet another – great article! I have a friend who moved to 5Ds and had to upgrade a lens or 2 (soft corners or decenterred if you will. 😉 ).

    It continues to amaze me how what I would call “spRAY of Light” onto a sensor actually becomes a picture you can store and print. Taking an analogue blast of Light/rays through glass onto a sensor and – it works beautifully, light never comes organized as 5um dots and yet…

    I was looking at a large poster 3*5 meters in an airport – recently. beautiful picture of a landscape – crisp with good light and walked up close – 20 DPI…. Max. Still, it worked!

    The comfort of knowing your bag is full of the best you can get is important (to me), but getting out there and actually using what is in the bag must be the goal (at least for me).

    Please don’t stop making these articles, they are read by many with joy!

  • Rob Stanton

    Fascinating. Thanks for that. I wasn’t crazy after all.

  • Michael Clark

    What is the best way when dealing with one of those unnamed manufacturer’s Authorized Service Center to convince them that your 70-200 f/2.8 that use to be the most amazing zoom lens you have ever used is now demonstrating a 20-25º tilt in every picture you take with it? Because it is still sharp at the center (or either edge, just not all at the same time) they say there’s nothing wrong with it.

  • John, that depends a lot, but in many cases they are lenses you could buy off the shelf, slightly modified for fixed focus and to work in vacuum.

  • John Dillworth

    “We test lenses that go into satellites and stuff.” I’m intrigued. can you share details? How different are satellite lenses vs. the best primes I can buy on earth?

  • david

    There are lenses (some of the Loxias, for instance) that have been tested to be less than amazing wide open but super sharp stopped down for landscape use. I guess there’s no way to extrapolate.

  • David, I can’t say it always is, but I can say without question that wide-open reveals the most defects. There are lots of lenses we find to have problems at f/1.4 that look fine at f/2.8.

    But I guess there could be some specific issue that is more apparent stopped down – probably that most of the lens sharpens but one area lags behind. But I could also see a scenario where a given copy doesn’t sharpen up quite as much as others. At least I think I can 🙂

  • david

    Great article! An expert, well-written explanation of the state of the art. Many thanks.

    This may be a question for the future, given how much work you’re doing already, but here goes: Is wide-open test data a reliable indication of lens quality at other apertures? I read all your tests, but I shoot stopped a bit down most of the time….

Follow on Feedly