LensRentals.com

Wavelength-Detecting Sensor Eliminates Bayer Filter, Triples Resolution

Posted by

“Current camera sensor technology is completely backwards.” Dr. Eno Lirpa

Everyone knows that in order to generate color, a digital camera’s sensor is overlaid with a Bayer filter. The filter makes each pixel sensitive to either red, blue, or green light.

 

Standard Bayer filter, courtesy Wikipedia Commons

 

Software than interpolates this red, green and blue image into the final color image we see.

 

Representation of an scene (above) and the raw Bayer data captured by a digital camera (below). courtesy Wikipedia Commons

 

In effect, our 24-megapixel color camera doesn’t resolve any better than a 14 or 15 megapixel black-and-white camera would.

There have been several attempts to improve on the Bayer-array method of detecting color. The Foveon sensor, which stacks red, green and blue pixels at different depths at each pixel (sensor site), certainly provides higher resolution than a standard Bayer sensor, although the Foveon sensor has it’s own limitations.

 

Foveon X3 sensor stack, courtesy Wikipedia Commons

 

Fuji has altered the array in their sensors, creating a more random pattern. This gives (arguably) some improvement over the standard Bayer array but still uses the same basic principle with inevitable loss of resolution.

 

Fuji X-trans array, courtesy fujifilmusa

 

At WPPI, I had the chance to spend time with the team from Baceolus Imaging, a small Italian imaging technology company with a growing patent portfolio and plans to make a big splash.

Using Energy to Detect Color

I was able to talk with Dr. Eno Lirpa, one of the optical physicists on the Baceolus team, about their new sensors.

“When looked at from a physics point of view, current sensor technology is just a foolish design”, Lirpa says. “You give away so much resolution just to detect color. It’s just not necessary. Every photon already carries a color message.”

Every first-year physics student learns the simple formula E = hc/ λ where E equals photon energy and λ it’s wavelength. If you know a photon’s energy it’s simple to calculate wavelength and therefore determine the photon’s color.

For example, blue light, with a wavelength of 400nm has an energy of 3 electron volts per photon, while 700nm wavelength red light has an energy level of 1.77 electron volts, and green light 2.43.

“It’s a fairly simple engineering matter to measure a photon’s energy as it strikes the camera sensor, but everyone has been so focused on cramming more megapixels onto the chip,” Lirpa continues. “There hasn’t been much interest in adding additional technologies to the chip.”

There is no Bayer array over a Baceolus sensor. Other manufacturers have used new back-illuminated sensor technology to move the wiring behind the actual photosensor, rather than in front of the sensor.

 

Difference between standard and back-illuminated CMOS sensors. Sony: http://www.sony.net/SonyInfo/News/Press/200806/08-069E/

 

Baceolus goes one step further. They’ve taken advantage of a back illuminated sensor to place energy sensing circuitry in front of each photo well. A simple calculation converts energy level to wavelength, determining that photon’s color.

 

Schematic of Baceolus sensor, courtesy Baceolus Imaging

 

“It is a simple thing, in addition to detecting that a photon has struck the sensor, to also determine the energy level of that photon,” Lirpa explains. “Our sensors record that energy and pass it along to an in-camera chip that uses this information to not only detect the intensity of light striking each pixel, but also its energy level. The chip converts that energy level to wavelength, which shows the color of the photon.”

“Right here we can tell,” Lirpa continues, showing a diagram, “that a photon with energy level of 2.14 eV struck this pixel. That’s a lovely yellow photon, probably from a sodium vapor streetlight. A blue photon struck this pixel and an orange-red photon over here.”

Numerous Advantages

According to Dr. Latot Parc, the researcher designing the computer chips to process the images in-camera, a 20 megapixel Baceolus sensor not only provides the same resolution as a 40 megapixel Bayer array camera but an amazing 32 bits of color depth. I was only allowed to keep one image taken with the camera, but below are 100% crops taken with a Canon 5D III and a Baceolus preproduction camera using a Canon 135mm f/2 lens. (Yes, the Baceolus camera will mount any Canon EF lens, so a huge lens selection is already available. The Baceolus team would not comment, but I have the impression a Nikon-mount version may be in development.)

 

Overall Image

 

100% crops comparing Baceolus sensor (left) compared with standard sensor using same lens (right).

 

There are other advantages to an energy-detecting sensor. High-energy ultraviolet and low-energy infrared light can be screened out in firmware, as the camera processes the image, so an infrared filter is not required. A simple flip of a switch tells the camera to change from visible light to infrared or even ultraviolet.

 

 

A simple switch on the camera changes from normal mode . . .

 

 

To combined ultraviolet, visible, and infrared mode, or any combination of these. 

 

Additionally, the sensor can act as a ‘photon trap’, detecting only photons of a given energy range. This could allow an astrophotographer, for example, to set the camera to only accept light with wavelengths of 410 nm, 434 nm, 486 nm, and 656 nm  (the spectrum of hydrogen light) or to capture simultaneous UV and IR images of a nebula.

 

Image of 30 Doradus in full spectrum of UV through IR (above) as compared to normal Bayer sensor image (below). Credit F. Paresce and R. O’Connell

 

I did ask Dr. Parc about rumors that some beta testers had set their Baceolus sensor to detect only flesh tones, allowing a photographer to basically take an image of a person’s body right through their clothes.

“It’s certainly possible but would require long exposure times.” Dr. Parc said. “Doing so would completely overexpose the subjects face, hands, and other areas not covered by clothing, so we doubt anyone would be interested in doing that. Besides, our extensive research shows that very few photographers are interested in shooting nudes. At any rate, a person concerned about being photographed through their clothing can simply wear thick woolen underwear, which is an effective photon blocker.”

“The advantages our sensor brings to resolution and color, especially in the studio, are amazing,” Dr. Parc adds. “For example, everyone shooting a digital SLR today is frustrated by their camera’s inability to properly reproduce the color Periwinkle. We hear it every day. Whether it’s a model’s periwinkle blouse, or a beautiful periwinkle flower, the photo is drab and lifeless because the Bayer array’s red, green, and blue filters just can’t reproduce periwinkle accurately. With our sensor, periwinkle is simply 2.513 electron volts, rendered every bit as accurately as any other color.”

 

True Periwinkle (left) is often rendered inaccurately by Bayer-sensor cameras, appearing too purple (center) or aquamarine.

 

Even more important, the Baceolus team believes, will be the ability to bring cell-phone staples like face detection and smile detection to full SLR cameras. They envision a setting in which the camera identifies areas of 2.03 to 2.13 Ev color (flesh tones), locking that area in as face detection. It then automatically takes a picture when a minimum of 5% of that area changes to 3.1 Ev – the near-ultraviolet color of bleached teeth — so you get the perfect smiley face every time.

And One Disadvantage

There is only one thing not improved on a Baceolus sensor. The circuitry used to measure photonic energy does detract from dynamic range slightly. Pre-production Baceolus sensors have a DR of about 12 electron volts, less than most current SLR cameras.

“This is no problem for good photographer,” states Yug Diputs, who recently joined Baceolus as director of marketing, in slightly broken English. “Combining two images at different exposure is best way to take picture anyway. Only bad photographers are limited by dynamic range. You see this repeated on every forum on internet — problem is always bad photographer, not bad equipment.”

When Can You Get One?

Very soon, according to what I’m hearing. Release in Europe and Japan is expected by early summer.

“We expect to have our camera on the market well before the Canon 200-400 f/4 IS lens is released,” Parc told me. “We got very lucky because Nikon has, for some reason, a large supply of autofocus sensors they can’t use which we were able to buy very cheaply. That moved our release date up several months.”

Release in the United States, unfortunately, will be delayed for quite a while. “In the entire world, only the United States, Burma and Liberia do not use the metric system. Because our system converts the electron volts into nanometers, it is calibrated for metric light and we can’t guarantee image quality in areas using nonmetric light. We hope to have a firmware upgrade that will convert electron volt measurements to inches by September or October.”

“Dual-system countries, like Great Britain and Canada, appear to have sufficient metric-wavelength light to allow our cameras to work properly,” Diputs was quick to add.

Price has not yet been determined.

Roger Cicala

Lensrentals.com

April 1, 2013

65 Responses to “Wavelength-Detecting Sensor Eliminates Bayer Filter, Triples Resolution”

NoOne said:

Eno Lirpa :D Nice one !

Feng Chun said:

I was about to believe this until I read the “when can you get one”…

Roger, Did you make this yourself?

Joachim / CH said:

April 1st…

good one, though :) Nikon indeed seems to collect AF sensors they can’t use. Or maybe, they should keep to use and sell the better ones.

Jean-franois said:

Lolll, non metric light…. Good one. It was interesting though

aschk said:

That was even better than Google’s joke! Thanks, laughed out loud like crazy when reading the last few lines!

John King said:

I can’t imagine anything topping this story today. At least not from any non-metric light emitting countries.

Richard Hatch said:

had me going…

mazo said:

OMG, almost got me! I was really starting to ask myself how they are able to build energy-aware single-photon-detectors. Well done, very well done!

Pedro Vera said:

Damn you Roger, you had me going until the disadvantage section. Well played, sir. Well played.

Ben said:

Roger, great one. Help me out here. I assume the reason that something like this can’t work is because it would essentially need to be an array of single photon detectors. Many photons striking the photosite at the same time would make performing calculations on each photon impractical. As I understand current sensors, over the duration of the exposure time photons strike the photosite causing charge to build up and then are read off at the end. So, what is recorded is an average brightness for light that made it through that photosite’s color filter over the duration of the exposure.

And that type of accumulation and read out at the end wouldn’t be practical (possible?) in the system you describe. Is that about right?

LensRentals Employee

Roger Cicala said:

Ben, I believe you’ve hit the nail right on the head. The thing that makes it sound kind of logical is the “single photon” idea, but that’s not reality.

Ben said:

Roger, the only way I could imagine it working is if this notional single photon detector somehow went into read-only mode once any single photon touched it. After readout it would then be reset to await another photon strike.

What do I know…just a software engineer who took some physics 15 years ago.

Trenton said:

Very good one, almost believed it )

Trenton said:

Non-metric light. That’s fantastic, I am still rolling OTFLMAO.

And I just wanted to add Lobachevsky intersecting light source to my Einstein flash… Non-metric light. That will do the trick.

Lynn said:

Had me going until the “non metric light” part! Great one, Roger!!!

Milton Simoes said:

This new sensor is even worst than the X-trans, look at the artifacts close to the duck head. It sucks. Garbage.

By the way, what are you using to hypnotize the Duck?
It didn’t move a mm between the shots. Thats the real secret here. You and your friends are developing a Freezing People Devise as part of the conspiracy of the Slow Street Photographers Association, obviously.

Nice.

Andrew Burday said:

What I want to know is, is anybody working on integrating this with the Google Smell technology? You could combine an incredibly high resolution shot with the appropriate odor to give the viewer the feeling of truly being there. Not just a picture of clouds breaking over a mountain lake, but the smell of newly fallen rain as well. Not just an image of a rock concert, but the scent of thousands of sweaty teenagers who haven’t showered since Wednesday. The possibilities are endless. I feel fortunate to be alive in a time of such innovation.

Brandon Meyer said:

Very Nice! I was getting excited until the metric wavelength. Brilliant.

John H. Seidel said:

Damn inch-foot light! We’re behind the technology curve again.

Joshin Y. said:

(Slow clap)

Pitch perfect. This one had me going right up until the end, and then the slow reveal as you turned up the ludicrous knob just gradually enough to give me a humorous journey was awesome.

Thank you!

Samuel H said:

You are a mean, mean man…

LensRentals Employee

Roger Cicala said:

Milton, we used a double barrel shotgun to hypnotize the ducks. Those are possum ducks, found only in the Southern U. S. They play dead at the sight of a shotgun.

Paul Buckley said:

The name “Eno Lirpa” sort of set the tone…

LensRentals Employee

Roger Cicala said:

I really rather liked “Latot Parc” myself :-)

Lee Saxon said:

You know, this will probably actually happen one day, it’s probably just a matter of massive computational requirements…

Jerry Russell said:

Roger, I think you should do a patent search. This just might be a really good idea. :-)

Byron said:

Well played! Particularly the references to the Canon 200-400, and the Nikon autofocus sensors. Given the AF sensors, I think these cameras may be particularly prone to hemispatial neglect!

nandadevieast said:

I got it when i read about the underwear thing :)

Benjamin Anderson said:

Best Baceolus review I’ve seen so far today.

Luckily, I’ll soon be in enough money to afford one, according to the email I received from a very nice prince.

Daniel said:

Well, I didn’t know that it is fools day today (I am somehow confused by the coincidence with the Easter Monday).
Well, I woke up once I read about the non-metric light in USA :)
Good one!

Ralf C. Kohlrausch said:

If the unions manage to raise my income every April 1st, I hope I will eventually be able to afford to buy one. Otherwise I’ll just rent a camera with this technique, once they become available and autofocus- and oilspots have been sortet out. Don’t think, software will be able to autoretouch those supersharp dustbunnies or worse. Handlingtimes in postprocessing will increase dramatically, rendering this technology useless for the average pro. Maybe CIA will be interested, stands for Cameras 1 April anyway, doesn’t it?

Greets
Ralf C.

Michael Steinbach said:

I didn’t know that the light that hits the US parts of Canada and the British Isles was non metric! :)

James Sinks said:

You left off my favorite feature–the accelerometer at the bottom of every pixel well so you can discard photons that are moving fast enough to be motion blurred at your current shutter speed!

Dan said:

You had me until the very end, I was gaffing at the AF sensor bit, and the broken english conversations but it wasn’t till I saw the date when I got it. Best prank of the day so far

Mike said:

I bought special bulbs from London, so I have metric light in my home studio. I am so excited!

Elmer said:

I totally believed the article until I read the “When Can You Get One” and saw how it is difficult to use in non-metric countries due to measurement of light.

Nqina Dlamini said:

Had me going there for a moment. The single photon issue was niggling me but decided to believe you. Dam You got me good. The last part was hillarious.

Richard said:

Doctor Parc and Mr Yug were my favourites as well :-)

simon said:

nice. :)

Ben said:

Damn it you got me good. It’s April 2nd here down under so I was off my guard. I even told a few people about this awesome tech coming. Wasn’t til I just now read the comments. Nicely played.

Rhombus said:

The time-energy uncertainty principle means that the more accurately you learn the photon’s energy, the less you can discern its arrival time. So Dr. Lirpa’s sensor requires long exposures, hence small aperture, and then all those nasty Nikon oil spots will show up!

Steven said:

Loved the non-metric light line. When I was an undergrad in Mech. Eng. I wished every day that the US would move to the metric system.

I’d like to point out this is a real product though, used in Sigma cameras (they make cameras too).

http://en.wikipedia.org/wiki/Foveon_X3_sensor

To summarize: “…Bayer images [are] ahead on fine monochrome detail, such as the lines between bricks on a distant building, but the Foveon images are ahead on color resolution.”

Wayne Morellini said:

Good laugh. I was looking at a similar idea in the 90′s, but the problem anybody thinking about this as they read it, is how do you seperate out the colors of millions of different colored photons hitting the sensor at once, you might as well say it is a small cube powered by a dwarf holding it jumping on one foot. The foveon actually tries to achieve this. However, after all these years, and a lot more experience, I looked at it again, and dud come up with a solution (this is April the second).

After my quick glance through some joke busting moments:

- Scientist think how sensors work is foolish. I doubt their us a sane scientist that does. Existing sensors are an efficient compromise.

- Existing sensor makers are interested in only more pixels rather than adding to the pixel itself. Much research goes into improving the pixels, including adding things to them to improve performance, which enables smaller pixels etc.

- foveon actually tries to achieve this.

- As above, separating out colors, you can get a average value maybe. (hmm, just thought of another way to do it).

- The duck. Bayer is not going to produce such a large size of detail loss on the hires Canon, but comparing still mode to the lacklustre compression of lower res video mode would, or just shifting focus. It is obvious that it is the sane frame and neither the duck or the camera has changed position.

- If the face and exposed skins burns out in cloths piercing nudie shots, people would not be interested in doing it, right, or increase sakes by ten. Anybody heard of hdr techniques, or adding hdr technologies used in other sensors, or getting the sensor to detect other ranges if too bright for the pixel. Actually the idea itself has merit, without long exposure. I thought of the possible mechanics of doing this a few tears back when solving another problem.

- Expecting everybody else to wear thick woolen underwear, that’s funny.

- The color periwinkle, they hear it every day, what’s periwinkle. Try blue or red clothing or lighting on Bayer, then displayed on 4:2:0 video signal.

- dynamic range, you would add hdr sensor technologies.

Just to make the less observant wake up that there is something fishy going on, you add all the metric etc light stuff keeping it out of the US, brilliant. I think I’m going to post a thread about this over on a certain camera forum and see who picks up on it by the time they get to the comments. :)

I don’t like lying, but this was a pretty brilliant effort Roger.

Thanks.

NancyP said:

Best photo April Fool’s Day joke of 2013. Hands down.

Nicholas Condon said:

As someone who does sceintific spectroscopy professionally and photography as a hobby, I have to say that I want one of these sensors for both work and play. You’ve clearly outlined the advantages in imaging, but you have no idea how happy I’ll be to get rid of the expensive, large, and now outmoded junk in my lab, like monochromators, Fourier transform spectrometers, bandpass filters, and the like. One prosumer-grade camera body, and my lab is all set! I’ll go start smashing my monochromators and using their 10 cm, 2400 l/mm gratings as drink coasters! Thanks!

Gavin Melville said:

I still don’t get the Dr Eno Lirpa reference — is this a USA only in joke ?

Gavin Melville said:

Ok — I got it.

LensRentals Employee

Roger Cicala said:

Eno Lirpa backwards is April One

jb said:

NICE !!
I was completely fooled until I read 3 times the end about “metric-light”…
You’re funny when serious, and the other way too…
Keep going roger

Leave a Reply