Why We’re Going to Start Testing Cinema Lenses. And Why We Haven’t Before.


Here we go again. This isn’t my first time pissing off the status quo; I’ve been doing it for years. We spent the last few years developing better metrology (lens testing) for photography lenses. The reason was relatively simple; photography cameras with higher resolutions began to see flaws that the photography company’s (and our existing) metrology couldn’t see. Test charts and lens test projectors just didn’t cut it anymore.

CineLens MTF Testing LensRentals

We don’t keep our methods secret. We write some up on this blog and publish others in peer-reviewed journals. Of course, the industry isn’t always enthusiastic about it. Their existing methods said their lenses were fine, and change is expensive. But as more and more users complained about defects they could see in their images that the manufacturers couldn’t see in their testing we had engineers from most of the major manufacturers visit to see what we’re doing. Over the last few years, most have started improving their metrology. A few are enthusiastic about it, most a bit grudging, and some just repeat ‘it isn’t necessary’ over and over.

During this time we’ve tested many thousands of photo lenses and maybe a dozen video lenses. Why? Well, because there was no reason to MTF test video lenses. So why start now? Because pretty soon there will be a reason to. So before I start rolling out MTF charts of video lenses I thought I’d put up a bit of a post about our ‘why now.’


Why We Haven’t Bothered

Lenses for video and cinematography are different than photo lenses in some distinct ways, but many of you will be surprised at some of the ways they aren’t different.

Cinematographers are very concerned about the ‘look’ of a lens, its color rendering, focusing accuracy, lack of focus breathing, parfocal zooming, how it handles flare and a dozen other things. They are interested in having a ‘sharp’ lens, but sharp is a vague term and very different in video and photo work.

Video is shot on much lower resolutions sensors than a photo, and there is less concern about the highest resolution. Photographers, with their higher resolution cameras, worry as much about absolute resolution as they do about contrast. In testing, fine resolution is referred to as high frequency. Test charts or lens projectors are perfectly great at low-frequency testing. There’s no need to use fancy optical machines for that.

Why We Are Bothering Now

The key here is that if you shoot on a high-resolution sensor (assuming a good lens), you see fine detail that was invisible before. Two very close points look like one point at low resolution but become two points in the image as the resolution increases.

If you quadruple the number of pixels on a sensor, you theoretically double the resolution of your image; you can see the detail that wasn’t in the low-resolution image. (This assumes you have a good lens in front of the sensor. If it’s a not-so-good lens, resolution improves some, but not as much.)

Photographers in recent years have jumped from 16-megapixel sensors to 36 and even 50 megapixels. Not quite a doubling of resolution, but enough to allow the cameras to see weaknesses in lenses that weren’t apparent before. Mainstream video has moved from 720p (about 1.3 megapixels) to 1080p (about 2.3 megapixels) to 4k (8-9 megapixels) in recent years, so there’s already been a resolution doubling although the resolution is still much lower than for photography.

But now, the higher-end video is 6k (about 19 megapixels) or even 8k (33 megapixels). Video resolution is nearly doubling again, in a very short time. When photo cameras reached 32 megapixels, photographers started seeing flaws in their lenses that the manufacturers didn’t see on their existing testing. I expect the same thing is about to happen with video.

Now 72,934 cinematographers are all going to assume that those more expensive video lenses aren’t going to have the problems that cheaper photo lenses have because they haven’t seen it (yet). And 643 places that test video lenses on charts and projectors are going to tell them they don’t need to worry about it because their tests show the lenses are perfect (so far). And then, one day soon, a cinematographer somewhere is going to look at his high-resolution footage, shot with a lens he’s used many times before at lower resolution, and scream because one side of the image is blurrier than the other.

If you want to stop reading here and go ahead and comment that I’m completely wrong because you just know I must be you may proceed. If not, you can read further, and I’ll give you some introductions to the Cinema lens tests we’ll be publishing in the next few weeks and months.


MTF Full-Field Display Examples

We’ve started using these because they provide a really intuitive way to look at the MTF of a lens. In previous posts, we’ve used the frequency of 30 line pairs /mm because that’s a good compromise between high and low resolutions.

For those of you who don’t speak MTF, only a high-resolution sensor can differentiate high frequencies, like 50 lp/mm. If you’re shooting a 3.2-megapixel image, you don’t see these. If you’re shooting a 30-megapixel image, you definitely can. A high MTF, the blue shades in the graphs below, mean things have high contrast. Low MTF, the red shades in the graphs below, is just a gray smear.

For the examples below I’m going to use a nice Cinema lens that many people have shot with, the Zeiss 35mm T2.1 CP.2 lens. This first set of graphs shows how the average of 10 copies of this lens performs at different frequencies.

10 lp/mm

At low frequencies, the lens is very good. The dark blue means the MTF is very high, and it stays that way from one side to the other.

Olaf Optical Testing, 2017


20 lp/mm

At a slightly higher frequency, things are still good, but you can see that the lens is a little softer away from centers. If you’ve shot with this lens, even at 1080p you’ve noticed this.

Olaf Optical Testing, 2017


30 lp/mm

This is our usual testing frequency, and things look as you’d expect, MTF is lower even in the center now. How would this matter? In theory, if you’re shooting at 1080, you probably would see the same resolution in most decent 35mm lenses, but at 4k, you will start to notice some are better or worse with fine detail. I’m not making any judgment about the CP.2 (yet), just showing that there is room to be better or worse than this lens at higher frequencies.

Olaf Optical Testing, 2017


40 lp/mm

Olaf Optical Testing, 2017


50 lp/mm

At higher frequencies we’re getting a lot of orange now, meaning the lens is just barely handling fine details. Red would be low enough that fine details would be lost if you’re shooting very high resolution video. That may or may not matter a bit to you, just pointing it out.

Olaf Optical Testing, 2017


So That Didn’t Matter Much at All, but This Does.

The images above were averages of 10 copies, and behaved very nicely, just like the lens manufacturer’s data does. But what if we look at individual copies of that lens? Well, that’s where it gets interesting.

Here are nine copies (because nine fit nicely in the frame) of the 35mm T2.1 CP.2 tested at 10 lp/mm. They all look great and perform nearly identically; you certainly couldn’t see any difference between them shooting 1080p video. (BTW – the green check is just from our software letting us know these files have been backed up.)

Olaf Optical Testing, 2017


Here are the same nine lenses but this time, shown at 30 lp/mm. At this higher frequency, it’s apparent the lenses are not all identical, although all are acceptable. But on a couple, the very edge is getting unacceptably soft (red areas) and in one (upper right) the center isn’t quite as sharp as the others. Looking at the images above, this difference isn’t very apparent at the 10 lp/mm frequency (these are the same lenses in the same order).

Olaf Optical Testing, 2017


If we look at them at 50 lp/mm we see there’s even more variation.

Olaf Optical Testing, 2017


Does this variation at higher frequencies matter at all? If you’re shooting 1080p video, absolutely not, you can’t see it. If you’re shooting 4k, you probably still wouldn’t notice it. But at 6K or 8K you certainly could.

So What Does It Mean?

Mostly this post is just an introduction to the topic so that when I start showing Cine lens tests, you understand why I’m showing these higher frequencies and whether those frequencies are important to you. And also to explain why I’m saying you may start seeing things you’ve never seen before when you start shooting 6k and 8k.

And as people transition into higher resolution video, the differences in lenses will become more important. We’ll be doing a lot of comparisons in future posts, but to give you an idea, the Canon CN-E, Zeiss CP.2, and Sigma Cine 35mm T1.5 have nearly identical performance at low frequencies on average.

Left to right: Zeiss, Canon, Sigma 35mm Cine lenses at 10 lp/mm; Olaf Optical Testing, 2017


But begin to look a bit different at higher frequencies.

Left to right: Zeiss, Canon, Sigma 35mm Cine lenses at 30 lp/mm; Olaf Optical Testing, 2017


Moreover, as we showed above, variation between lenses is greater at higher frequencies and will be more noticeable on higher resolution sensors.

I just wanted to show why, as we start publishing Cine lens MTF charts, we’ll be emphasizing how the lenses differ at different testing frequencies. Depending on the resolution you’re shooting at it may make a huge difference to you, or none at all.


Roger Cicala, Aaron Closz, and Brandon Dube

July, 2017


. . . .

Author: Roger Cicala

I’m Roger and I am the founder of Hailed as one of the optic nerds here, I enjoy shooting collimated light through 30X microscope objectives in my spare time. When I do take real pictures I like using something different: a Medium format, or Pentax K1, or a Sony RX1R.

Posted in Equipment
  • Brandon Dube

    We are very unlikely to test for lens breathing in a quantitative way. Parfocal is… a maybe.

  • Mike Earussi

    I’m especially looking forward to your parfocal and lens breathing info because no one else tests for these and I hope you decide to expand these to your regular lens tests as well (no one tests for true focal length either).

  • Andre, it’s early — it takes a year or two to change methods. But I have seen improvement in identifying bad lenses from at least two manufacturers and improvement in service center capabilities from one of those two. (Changing at the main factory is first, the service centers follow slowly.) But Zeiss and Leica, who’ve always done true optical testing have always had consistent lenses. Sigma started doing it a few years ago (and I had nothing to do with that, Yamaki-san revamped all of their testing when the Art lenses started coming out) also has very good consistency in their prime lenses.

  • Lo-Wok Li Ya

    Thanks you so much Roger, Aaron and Brandon!!!
    Indeed, now is the right time to start publish and test video lenses since the 8k sensors are already here at the gates.
    I’m eagerly awaiting for the further reads on cine lenses.

  • Athanasius Kirchner

    Absolutely. Also, there’s a lot of people involved in a typical film, and so, a DP can easily obsess over resolution and moiré, only to be overruled at the last minute by the main director and have that beautiful glass massacred by a silk stocking (*gasp*).
    I mean, we haven’t even moved from 24 FPS, which can be the greatest roadblock to resolution, but I’m sure there’s plenty of Hollywood DP’s that will pore over LR’s results, looking for that perfect 8K lens.

  • Brandon Dube

    There’s no sense duplicating effort, and having metadata like this doesn’t put us at any sort of competitive advantage. Our custom software has significant value, a spreadsheet like this does not.

  • I like that you’re publishing these on github. Very cool.

  • Roger, the comment about manufacturers improving their own metrology is very interesting. Have you been able to see improvements in either performance or consistency in lenses from those manufacturers who’ve improved their testing methods? And are ones who are in denial still turning out lenses at the same performance levels?

  • Nate Weaver

    This is good, and I’m glad you’re doing it, but as a working Director of Photography (been shooting 10+ years), I have a couple comments.

    When I started investing in my own lenses and going down “the path”, one of the things I was interested in was sharpness. I had nothing else to go by. I had not shot with a huge selection of lenses, but I had a short list of reliable standbys that I went to.

    When I decided I was going to start investing in my own, I started doing tests of everything I could beg, borrow or steal. It was the beginning of starting to learn some of the other parameters you might decide you care about, and others you do not care about. As this process went on over a couple/few years, the importance I put on sharpness started sliding down the scale. I started caring more about how what detail there was there was rendered, how they flare, what color they flare, breathing, focus scales to make life easier on my 1st AC, etc etc.

    I suspect what is going to happen in thee tests is that some people are going to see some really expensive lenses resolve terribly compared to stills lenses at 5% of the cost. It’s not hard to make a sharp lens these days. Computers make designing a sharp lens easy, and new manufacturing techniques make previous tolerances cheaper to hold.

    In the cinema market, we have a whole other set of parameters, and since most of us, while we are sometimes shooting 4K and sometimes 6K, most of our jobs are delivering 1080p. I can count on one hand the number of jobs I’ve shot with a 4K camera or better in the last 10 years where absolute sharpness mattered because they were punching in on a 4 or 5k master in post. Some DPs do that kind of work every day. Most I know are not. But more than anything else, we need consistency. Consistency between single focal lengths in a prime set, and consistency in a family of zooms. Size, weight, stop, all has to merge together as one tool across many lenses. That’s one of the biggest things to achieve as price goes down. Getting to a T2 on a 18 is a much different task as getting a T2 out of a 50. That 18 is going to cost a lot.

    For most of the work DPs are doing, lenses were “sharp enough” long ago (spherical lenses, that is. Anamorphics have been a tougher road). We get sensitive to the finer points of how a lens or lens set renders and tend to try to match the character to the job. That attention to detail is lost on a lot of viewers, but some can appreciate it, if only in a subconscious way.

    A good example is how Cooke is “re-issuing” Panchros. They resolve terribly by modern standards. But they are beautiful. Go watch “The Crown” on HBO to see what I mean. And Cooke will sell all they can make for 2 years, and they will rent for $800/day. Early buyers will have them paid off in 18-24 months. To cinematographers who need that look, they are worth their weight in gold. You don’t get that look with a Glimmerglass 1.

    So I’ll be tuning in, but I’m sure a lot of people will be “WTF”. Expensive cinema glass is a business. DPs spend other people’s money to achieve a result. The world will keep turning.

  • Brandon Dube

    The mounts we can handle, and we have our handy dandy big ass table of lenses 🙂

  • Kemalettin Sert

    Holy moly.I have been waiting this for a long time.

  • Carleton Foxx

    Totally there with you. When you have Scarlett Johansson on one side of the screen and Leonardo DiCaprio on the other, you want both of them in focus. And of course you’re not quite the humble mechanic you claim to be, you’re more like the guy who invented the idea of wrenches.

  • Louie G

    Eager to see the results. It will be a lot of homework to do then, as not all cine lenses are designed for full frame. In addition to this, many lenses use different mounts for different cameras. Nonetheless, it will be more than interesting and useful to have the MTFs.

  • I totally get that. My job (at least as I see it) is to provide the tool that gives them all options. But I’m far less worried about a bit of overall softness. At higher frequencies depth of field narrows, too, and tilts become far more obviously.

    My thought is the worst case scenario is going to be that detailed shot that has one side of the footage obviously different than the other side. A wall of foliage, distant nature footage, architectural things across the image are the problems I foresee.

    Or perhaps why does the subject look one way when they’re on the right side of the image, and another when they’re on the left side? That same lens may be just fine when there’s a centered subject.

    I have zero cinema skills and don’t pretend too. I’m a lens mechanic. It’s tuning a race car for me — I want the driver to have all possible capabilities, but I don’t want to drive the car.

  • Carleton Foxx

    Ironically, on cinematography forums there is a fair amount of complaining about too-sharp lenses and discussing which diffusion filter to use for which project.
    A lot of them like the Schneider’s Hollywood Black Magic, others prefer the look of Tiffen Pro-Mists and GlimmerGlass, the cinematography outlaws prefer to stretch silk stockings over the rear element of lenses and hold them in place with something called “snot tape” which, thankfully, they don’t identify.
    So even though these people may say they want perfection in their lenses, they also want to be able to undo all your hard work by just dropping a filter in to the matte box.

  • Brandon Dube

    I’d hope it’s readable… many many hours of my life were lost to that paper 🙂 We’re trying to tell a story, MTF vs freq is the norm in optics because… reasons, MTF vs field is the norm in photography because… reasons, here’s MTF vs Field (2d), which we think is much better because…reasons.

  • I think it’s very similar to what we’ve gone through with photo for the last couple of years: you see what you test for. In adjusting lenses I know we can often get low frequency perfect but then a further tweak is necessary for the higher frequencies. If we weren’t looking, we wouldn’t see it. If adjustments are being done on test charts and projectors chances are very high they’re doing the same thing.

    There’s one other thing that may skew my findings, a bit: A lot of what we’ve tested so far are Zeiss CP.2 and CP.3, Canon CN-E, Sigma Cine, and Xeen. Those are all photo lenses adapted to video. I have fewer results yet on pure video lenses like Schneider’s, Leicas, and Cooke’s.

  • Athanasius Kirchner

    Well, that is odd, then… I’d have expected all that painstaking alignment to reflect better at higher frequencies, but then again, if it were like that you wouldn’t have written the piece, right? 😀

  • I am, indeed! Jim does amazing work and is very knowledgeable.

  • It was 30 lp/mm, actually. Thank you-corrected the legend.

  • I think you’ll find some of the comparisons we’re going to do, especially with Cine Zooms, very interesting from that standpoint.

  • Athanasius answered better than I would have! Thank you, sir.
    What I do hope to do is eventually have sliders so that you can look at the FFD MTF at whatever frequency is appropriate. If you’re shooting 8k in a scene with fine detail that you want to bring out, you’ll want to look at 50 lp data. If you’re shooting 1080p by candlelight, 10 or 20 lp will be all you want to see.

  • Thank you, Bruno. Fixed that.

  • l_d_allan

    Roger … I’m curious if you are familiar with and possibly have evaluated Jim Kasson’s use of a slanted razor blade with ImaTest and/or MtfMapper

    My impression is that his point is to get valid, reproducible numbers such as lp/mm or lp/ph with ultra-high resolution sensors like the Sony a7Rii. He seeks to get beyond the limitations of printed targets. His protocol also involves use of a motorized / computer-driven focus rail. Ouch?

    Mea culpa … I wasn’t able to get far enough along the learning curve of MtfMapper to get as much as preliminary numbers, even after correspondence with the author of FOSS MtfMapper.

  • Héng Ji?ng

    The last chart is measured at 40 lp/mm I guess?

  • l_d_allan

    Thx …

    This “video pre-newbie” speculates that a related issue is “screen captures” from individual frames of 4k and higher video.

  • Athanasius Kirchner

    Thanks for the great work, Roger and team! This will be certainly interesting. It’s evident that many cine lenses are optimized for contrast over high resolution, but those Zeiss (and the single Canon and Sigma) are really even across the field, which is part of what explains their much higher price. That’s the kind of QA that’d be extraordinary in photo lenses.

  • Athanasius Kirchner

    It’s not that simple, unfortunately, as it’s not like the different frequencies correspond neatly with each recording resolution. Also, different scenes will present different amounts of resolvable detail, which could become apparent (or not) depending on encoding and sample size… it’s really complicated, and if you’re a cinematographer using 10K or 50K lenses, you’d do very well to learn basic MTF.

  • Turniphead

    Thank you for your hard work as ever Roger!

    A suggestion for the data; how about describing the test charts as being for 1080, 4k, and 8k users? Most folks aren’t going to know MTF, but they are going to know (hopefully) what resolution they’re shooting at…

    I say hopefully, as I’ve lost count of the number of editor types who (without telling you what usage they plan) insist on a 300dpi image without giving any physical dimensions 😉

  • Bruno

    Very interesting insight, as usual, Roger. There is a typo though. The legend of the last graph says 10lp/mm which obviously it is not.

Follow on Feedly