Pinhole images from new, but optically decentered lens. Olaf Optical Testing, 2016

After we published a number of posts about copy-to-copy variation, people were quick to say that this company or that needs to ‘just’ improve quality control. I totally agree but realize most people don’t have a clue what ‘improving quality control’ would really look like. I think they have some vague idea of hiring a guy named Joe to sit at the end of the assembly line, check all the lenses, and reject all the bad ones. Optical quality control for lenses is way, way more complicated than that.

Before we start, let’s get a couple of things out of the way.

  1.  If you expect that the lens you buy has the exact same MTF that the manufacturer published, then don’t read this. You’ve got too much reeducation to do.
  2.  If you ever say, “for that kind of money I expect perfect” please don’t read further. Perfect doesn’t exist. For $20,000 and up you can get pretty close. For $2,000 you should be able to get reasonably close.

On the other hand, a reasonably expensive lens should have reasonable quality control and minimal copy-to-copy variation.  We know it can be done; some manufacturers are already doing it. Others totally suck. But when they suck, sticking Joe at the end of the assembly line won’t really fix things. If you want to read this pretty long article, I’ll explain why, and also explain some of what can be done.

A Few Basics First

To Some Degree, Consistency Costs Money.

Here’s a quick, relatively cheap example. I need to buy a fairly large, optically flat piece of glass to calibrate our MTF bench. But how accurate of an optical flat do I need? A flat accurate to 1/4 wavelength of visible light should be flat enough, right? That costs $450. But 1/20 wavelength is more accurate. That one costs $825. Is it worth the difference? Well, it depends on what you’re doing with it. For my purposes, 1/4 wavelength is plenty good.

Please don’t respond with, ‘well there will be discounts for quantity’. Sure there are, but better still costs more. Want a million 1mm screws toleranced to be between 0.96 and 1.04mm? They’re really cheap. Want them to be between 0.99 and 1.01mm? The price is going up. It’s cheaper per screw to buy a million, but the tighter tolerance is still more expensive.

Optical Testing May Be Cheap. Or Not.

People love to say testing won’t add much cost to a lens.  ‘Not much’ is a pretty variable term, though. A state-of-the-art MTF bench costs between $250,000 and $500,000. And it takes the time to test a lens. A single test run across the lens takes a minute or two. We need at least two angles (4 would be better) for each lens; plus, mounting, unmounting, and other things take time. So we can test a lens in 5 minutes; 12 lenses an hour. Let’s work 12 hour days and we can test 144 lenses a day. Unless it’s a zoom lens. Then you must test each lens at multiple focal lengths (3 or 4) so things down to maybe 3 or 4 lenses an hour.

If your assembly line is churning out 5,000 lenses a week you will need a lot of benches. Any way you slice it, the equipment and the operators to run it are going to add something to the price of the lens. I think a company being really efficient could do it for an additional $20 per prime lens, $60 per zoom. (I couldn’t break even at that cost, but I’m assuming they have economies of scale that I don’t have.)

Of course, if you want the best, state of the art testing, like interferometry, things could get really pricey. Interferometry requires different reference elements for each type of lens, and reference elements alone cost $25,000 to $50,000 each.

You can set up an Imatest-type program or even high-quality optical targets on a rail system to keep things properly square and get good results, especially if the images are computer analyzed.  That’s still very good and certainly less expensive. So there are lots of good ways to check optical performance to a high level, but they are going to cost a bit of money.

Or, you can make some ink-jet printed, thick-bar test charts, hook a camera to a computer, and let the computer say ‘pass/fail’ and poorly test a lot of lenses for almost no cost at all. This is actually what many manufacturers do. It’s quick, cheap, and has the added advantage of setting the failure bar really low so there aren’t many rejections. They have to do something like this anyway, in order to set autofocus and back focus for most lenses.

Some manufacturers do have high-level test equipment and test every lens. More often, they just do this type of testing on a percentage of lenses. (BTW – this sure doesn’t mean that type of equipment is available in your country’s repair center; it may only exist at the corporate manufacturing facility.) Many manufacturers do have some type of testing other than the ‘computer takes images of a few thick, ink-jet bars’. Then again, many do not.

But the Most Expensive Thing is Failing

But if you’re thinking, “OK, so they all need to have better testing and that doesn’t cost too much,” well, no there’s a lot more to it that that. The worst thing that can happen from the manufacturer’s standpoint is for a lens to be completely assembled and then fail quality control at the end of the assembly line. Let’s say that lens you pay $1,000 for costs $500 to manufacture and assemble and then fails a quality check.

What do you do? Throw it in the trash and take a loss? That’s actually fairly likely if it’s a consumer-grade lens. Disassembling and testing each component to salvage the good parts may be more expensive than throwing it away. You could just disassemble it, reuse all the parts without testing them, and hope it was just an assembly error and not a bad part. (This is pretty common among small manufacturers, and actually works pretty well.) But it still means the assembly cost of the lens is more than doubled. Now we assembled, disassembled, and reassembled and still have just one lens to sell.

Whatever you do, you will have to account for the cost of failed lenses. If you have to manufacture 1,100 lenses to sell 1,000, then each of those 1,000 lenses has to include the cost of those 100 failures. If you had to disassemble the 100 but were able to reuse the parts that still costs money and the cost has to be accounted for. One way or another, failure is always expensive. So good quality control needs to be done before and during lens assembly, not just afterwards.

And, just in case you haven’t thought about it everyone, from the owner of the company to the manager of the assembly line, at some point has to think, “well, just how bad did those 100 lenses fail? Are we maybe being too picky?”

Optical Quality Assurance is a Series of Feedback Loops

Design Feedback Loops

You may not have thought of it, but the biggest part of quality assurance probably occurs during design and prototyping.

Optical Design

The lens designer is given a set of goals and constraints; price, size, necessary features, release date, etc. The designer usually tries to avoid really expensive glass types and extreme lens shapes; these can cost 10X to 30X more than common glass.

Prototype elements for a future lens. Image courtesy C-4 Optics.

After the initial design is completed there are a series of redesigns to refine the lens. A lot of this refinement has to do with ‘tolerancing’. Each element in the lens is evaluated given the known accuracy to which it can be made, to see what effect that variation will have on actual performance of the assembled lens.  The lens may have to be redesigned several times so that inevitable variation in glass element production doesn’t degrade performance too severely.

There are tradeoffs for every change. We may get better consistency by changing this, but it might cause a slight aberration or increase the cost significantly. Other people are sitting in these meetings along with the optical designers. The marketing guys may want the best MTF curves possible, even if that may mean the lens is going to have more variance. The financial people may veto a more expensive glass. Different companies may give different emphasis to different things. The same company may give different emphasis with different lenses.

Mechanical Design

The mechanical design of the lens is usually far more complex than the optical design, and the same steps have to be made during this design. How accurately can each element be placed in the lens barrel? How much variation will occur during focusing or zooming? The optical designer will have identified if there are critical elements that may need to be adjusted in the lens so that the mechanical designer can add shims, eccentric collars, centering rings, or other forms of adjustment to his design.

The complexity here has to be seen to be believed. I’m working with another company designing some simple, manual-focus prime lenses. The designs for the mechanical parts run well over a hundred printed pages. That’s just mechanical parts; no autofocus, no zoom mechanisms, no image stabilization, and no optics. Every single part has to be sourced to certain specifications.

Image Courtesy C-4 Optics

Just like the optics, the mechanical design has to be toleranced. This piece may need to be accurate within a few microns, this other piece may be ok if it’s within 0.1mm. Even things like thermal expansion of parts need to be considered. The same type of compromises has to be made here. This part may be more durable made out of this substance, but that will have worse thermal characteristics and cost more. It’s endless.

Not to mention, the mechanical designer has to let the optical designer know these things. For example, if the mechanical design can only place an element within X microns, but the optical formula will be altered if the element is moved by 1/2X microns, there’s a problem and we have to go back to square one. The optical designer might have to change his formula again, perhaps giving up a little sharpness or adding a slight aberration to obtain more consistency.

Assembly Design

The designers and the assemblers also need to give feedback to each other. Can the manufacturing department put all the pieces together accurately and quickly? What type of automated machinery does the assembly line have and how accurate is it? Using existing machinery is always going to be preferred over buying new types of equipment just to assemble one specific lens out of the dozens of types that must be assembled each year.

It isn’t surprising that when you view our teardowns that you see different brands of lenses tend to have a similar look inside. The mechanical designer would much rather reuse or simply resize parts from drawings and mechanisms that already exist instead of redesigning hundreds of pieces from scratch. The assembly line will be more efficient if the process is similar for many lenses, not different for each specific type of lens.

Assembly variations must be considered in the optical design, too. A lot of lenses contain groups of cemented elements, for example. How is the cementing done? What type of cement is used? How accurately will the group be cemented?

There are going to be QA checks within the manufacturing and assembly process.  We also assume, at least for better lenses, the individual glass elements are tested on an interferometer prior to coating because that’s fairly standard. There is, probably, a centering check done for lens groups after they are cemented together, or bonded within a plastic sleeve, both because that is simple and inexpensive, and because it’s a place where an error is likely to occur. The particular optical design may need certain elements to be shimmed or adjusted during assembly and techniques for that have to be set up.

How Things Go Wrong

All of this feedback, in theory, should result in a good lens that can be accurately manufactured with good copy-to-copy consistency. However, there are some pretty obvious problems that not only can occur, they probably do occur to some degree in every lens.

Feedback between the designers may be limited or not exist at all.

While this probably isn’t a big problem in large, dedicated lens manufacturers, it’s actually fairly common when things are outsourced. Sometimes the lens is optically designed by one firm, mechanically designed by another, assembled by a third, all for a fourth company who brands and markets the lens. Even when it’s all done in-house, there are deadlines for both time and money to be met; the designers might want 6 more months to tweak things but prototyping is going to start Thursday. In any case, there are certainly times where the optical designer ‘assumed’ a lens or group would be placed to a certain specification, while the mechanical designer or assemblers ‘assumed’ the optical designer knew it couldn’t, etc.

Feedback between manufacturing and design isn’t accurate.

This is probably more common than any of you imagine. I’ve learned the hard way manufacturers don’t always meet their promises, and we always have to remember that outsourced parts are usually provided by the lowest bidder. To a greater or lesser degree, nearly every manufacturer uses some outsourced parts. A few companies make almost everything in-house; others make almost nothing in-house.

Even when the part is made in-house, there are cost and time constraints. Is that tool getting a little worn and the manufactured tolerance increasing? Is a subcontractor having financial problems and using equipment past the time it should have been replaced? Quite commonly someone expected someone else would deliver to a specification they promised, but didn’t meet. I have watched this happen myself on a small scale, and been told of first-hand experiences on a large scale.

Workers are Workers

Joe on the assembly line might not be quite as worried about eventual quality control as he is about meeting his quota for the day. Or he may be hung over, not feeling well, or really pissed at his boss. No matter how good the design and tolerancing are, some lenses are going to be assembled poorly. Consider that dozens of people contribute to manufacturing and assembling every lens. It only takes one goofing off to mess up the lens or camera you just bought.

One guy goofing off is all it takes to ruin things for everyone.

Testing is Inadequate

We only know of two SLR lens manufacturers (Sigma and Zeiss) that state they optically bench test every fully-assembled lens (for at least some lines) after manufacturing. Sigma has a proprietary optical testing machine and Zeiss uses their own MTF benches. But even in those cases, we don’t know if that means they test every single lens they make or only certain lenses. We assume other manufacturers test to some degree after complete assembly, because autofocus and image stabilization, at least, have to be calibrated into the lens chip.

The key point, though, is that what type of testing is generally quiet and what standards they test to are pretty vague. Many manufacturers use very simple testing;  basically, some slanted lines printed by a laser printer or pojected on a screen, photographed with the lens mounted to a computer-tethered camera. For many lenses, this is all the optical testing that is done.

There are reasons for this (note I didn’t say good reasons). Testing isn’t done by an optical engineer, it’s done by a low-wage employee. Simple test charts can be shot with a camera tethered to a computer that gives a green (pass) or red (fail) output.

That is sufficient for programming autofocus, but it’s not nearly sufficient to determine if a lens is optically acceptable. This type of testing arguably was adequate back in film days but is not even close to adequate for a lens being used on a high-resolution digital sensor.

I know there are many among you screaming you’re absolutely certain this isn’t true, but I can assure you it is. I’ve had more than a few manufacturers tell me their factory tests showed no variance at all in their lenses. When we found out what their factory testing consisted of, it wasn’t even as sensitive as what a lot of you do at home. As I told one factory representative, the reason they call it an AF1951 chart is because it was developed in 1951.

Don’t get me wrong. Doing a 4-quadrant, 21-point MTF on every lens after assembly isn’t necessary for quality control. It can see things that just aren’t going to be seen in the real world. But something more than deciding if the bars on an ink-jet print are sharp enough to make the computer beep green probably is necessary. I’m 100% for automating testing because automated things are less expensive and more accurate. But it needs to be a meaningful test.

Compromises on Quality MUST be Made

All of the feedback loops I talked about exist between the people with different agendas. Perhaps the optical design and mechanical design departments both agree the lens should be a little less sharp in order to be assembled consistently. The marketing people may scream bloody murder because they want the best possible MTF charts to show in their brochures. The financial people have put a ceiling on what the costs of lens parts are going to be. In every case, with every lens, the final manufactured product is going to be a compromise. Even if it’s a $5,000 lens. Even if it’s a $20,000 lens. Just because you think no compromises should be made, doesn’t mean it can be so.

In some cases, lowering QA standards by just a bit makes a lens that, at least on paper, meets everyone’s needs. Here’s a simple example. What if a fairly large number of lenses are just barely failing the company’s current QA standards? Some are awful, of course, and must never leave the factory. But what about the ones that are just barely not OK? If we sold those 1,000 barely not OK lenses, probably 950 of them would stay sold and then we just write off those 50 that come back. That might be the ‘best’ solution. Costs restraints are still met, the MTF curves are still great, the designers get their bonuses.

This last illustration probably raised your blood pressure a bit. But it’s a reality. No group of lenses is perfectly identical, so the decision isn’t black-and-white. It’s shades of gray. And someone, somewhere, in the company has to decide what shade of gray is OK and what isn’t OK. And for many companies, at this stage, it’s all theoretical. The lens and mechanical designers have plugged their variation numbers back in their software and told the company what to expect. Even if it’s worse than expected, the company may, or may not, be doing testing that would detect it.

Don’t think the company doesn’t realize that only a few of their customers are pixel-peepers – the majority won’t notice one corner is slightly soft or that their copy isn’t quite as sharp as Joe’s copy. Also, they realize that internet noise is real. There are going to be complaints about the best lenses, there are going to be people who love the worst lenses. If the lens in question is just a little worse, the internet conversation is not going to seem much different than any of the others. 600 people saying it’s great and 400 people saying it’s sucks isn’t all the different from 600 people saying it sucks and 400 people saying those 600 are just bad photographers.

After Release 

There are some after-the-fact things a manufacturer can do to improve quality in an existing product. The manufacturers service center notes if a lot of lenses are returning for a particular problem, the company figures out why that problem occurs, and substitutes an improved part or assembly technique the next time that lens has a manufacturing run. They can replace a weak electronic or mechanical part pretty easily. Changing the optics, though, is much more difficult.

Two copies of the same lens, before and after a manufacturing change. In later versions (left) the manufacturer simply incorporated the electronics you see on the right into other boards. If you own one of these, from the outside you have no idea which type you have. In this case, BTW, it makes no difference they perform identically. It was probably a cost-saving change.  

(For those who don’t know, manufacturers make a given lens in batches. An assembly line is tooled for a given lens and makes X number of them. Then the line is broken down and set up to make a different lens. So Canon 35mm f/1.4 lenses, for example, may get manufactured every X number of months. I’ve heard as often as every 6 months or as rarely as once every couple of years, depending on the lens, but that’s just what I’ve heard.)

One of the biggest surprises in my journey through the lens world is how often this ‘after-release’ feedback loop doesn’t exist at all. I’ve talked to optical and optomechanical designers and told them their lens has this or that problem and found they were shocked. In one case, the mechanical designer had designed several different lenses with the same weak part over many years and had never been told the part failed frequently.

In general, a large company that does everything, including repair, in-house will have a strong after-release feedback loop. Canon, Nikon, Zeiss, and Sigma, for example, all get this information and do make changes. Small companies, especially those that outsource repairs, are less likely to have strong after-release feedback loops

Company attitude also varies quite a bit. We’ve all seen companies that when confronted with a very obvious problem on a new item issue a recall and fix the very obvious problem. We’ve seen others that when confronted with a very obvious problem simply repeat over-and-over, there is no problem.

Most of the time things are done pretty quietly. If the company identifies a problem occurring in some of their lenses, they may quietly fix it at no charge for all that are sent in. This often occurs along with a change in future lenses. For example, years ago a certain lens very commonly had a failure of its stabilizing unit. A year or so after release, we noticed that some copies sent in for repair for other reasons also had the stabilizing unit replaced at no charge. Not long after that, we noticed that new copies no longer had stabilizer failure. It’s now one of the most reliable lenses in our fleet.

For things like optical copy-to-copy variation, though, after-release fixes are less likely. Changing the optical formula isn’t an option; that would require a complete lens redesign. Changing the mechanical design isn’t much more likely but could happen. It’s possible, I suppose, that a certain lens element might be made to tighter tolerances, or that a group might be cemented more accurately by a new assembly technique.

So is Everything Doom and Gloom?

No, not in the big picture it’s not. In fact, I’m really quite positive about improving quality control and reduced sample variation going forward.

I know at least one manufacturer that recently realized their optical testing standards just weren’t adequate and they are spending money improving them. Testing is a back-burner, non-profit area that they just haven’t thought about recently. Once the QA problem was identified and got their attention, they addressed it.

Other manufacturers are looking to reduce variability by installing better, during-assembly optical tests of each component as the lens passes down the assembly line. New glass types, improved grinding techniques, better mechanicals and more accurate robotic assembly is both cost-effective and reduces variation.

A couple of manufacturer’s have quite obviously started improving copy-to-copy variation already. At least one of them has started to use this very quietly in their marketing. I really hope that more of them are going to see the advantage in publishing something like our variation number, or even just a real-world resolution number that they guarantee their lenses will meet. Some manufacturer, someday, will realize the advantage of not having people question how much their lenses vary. Once one of them does this, maybe optical quality and consistency will become just as important a marketing tool as computer generated, little-to-do-with-the-copy-you-get MTF charts are now.

I expect we will see a lot more copy-to-copy consistency and fewer problems in lenses released in the near future. I do NOT expect, though, that we’ll suddenly see an existing lens that has been known to have high variance suddenly have less. It could happen, but the changes required would make that very difficult.

Roger Cicala, Brandon Dube, and Aaron Closz

Lensrentals.com

February, 2016