LensRentals.com

Wavelength-Detecting Sensor Eliminates Bayer Filter, Triples Resolution

Posted by

“Current camera sensor technology is completely backwards.” Dr. Eno Lirpa

Everyone knows that in order to generate color, a digital camera's sensor is overlaid with a Bayer filter. The filter makes each pixel sensitive to either red, blue, or green light.

 

Standard Bayer filter, courtesy Wikipedia Commons

 

Software than interpolates this red, green and blue image into the final color image we see.

 

Representation of an scene (above) and the raw Bayer data captured by a digital camera (below). courtesy Wikipedia Commons

 

In effect, our 24-megapixel color camera doesn’t resolve any better than a 14 or 15 megapixel black-and-white camera would.

There have been several attempts to improve on the Bayer-array method of detecting color. The Foveon sensor, which stacks red, green and blue pixels at different depths at each pixel (sensor site), certainly provides higher resolution than a standard Bayer sensor, although the Foveon sensor has it’s own limitations.

 

Foveon X3 sensor stack, courtesy Wikipedia Commons

 

Fuji has altered the array in their sensors, creating a more random pattern. This gives (arguably) some improvement over the standard Bayer array but still uses the same basic principle with inevitable loss of resolution.

 

Fuji X-trans array, courtesy fujifilmusa

 

At WPPI, I had the chance to spend time with the team from Baceolus Imaging, a small Italian imaging technology company with a growing patent portfolio and plans to make a big splash.

Using Energy to Detect Color

I was able to talk with Dr. Eno Lirpa, one of the optical physicists on the Baceolus team, about their new sensors.

“When looked at from a physics point of view, current sensor technology is just a foolish design”, Lirpa says. “You give away so much resolution just to detect color. It's just not necessary. Every photon already carries a color message.”

Every first-year physics student learns the simple formula E = hc/ λ where E equals photon energy and λ it’s wavelength. If you know a photon's energy it's simple to calculate wavelength and therefore determine the photon's color.

For example, blue light, with a wavelength of 400nm has an energy of 3 electron volts per photon, while 700nm wavelength red light has an energy level of 1.77 electron volts, and green light 2.43.

"It's a fairly simple engineering matter to measure a photon's energy as it strikes the camera sensor, but everyone has been so focused on cramming more megapixels onto the chip," Lirpa continues. "There hasn't been much interest in adding additional technologies to the chip."

There is no Bayer array over a Baceolus sensor. Other manufacturers have used new back-illuminated sensor technology to move the wiring behind the actual photosensor, rather than in front of the sensor.

 

Difference between standard and back-illuminated CMOS sensors. Sony: http://www.sony.net/SonyInfo/News/Press/200806/08-069E/

 

Baceolus goes one step further. They've taken advantage of a back illuminated sensor to place energy sensing circuitry in front of each photo well. A simple calculation converts energy level to wavelength, determining that photon’s color.

 

Schematic of Baceolus sensor, courtesy Baceolus Imaging

 

“It is a simple thing, in addition to detecting that a photon has struck the sensor, to also determine the energy level of that photon,” Lirpa explains. “Our sensors record that energy and pass it along to an in-camera chip that uses this information to not only detect the intensity of light striking each pixel, but also its energy level. The chip converts that energy level to wavelength, which shows the color of the photon."

“Right here we can tell,” Lirpa continues, showing a diagram, “that a photon with energy level of 2.14 eV struck this pixel. That's a lovely yellow photon, probably from a sodium vapor streetlight. A blue photon struck this pixel and an orange-red photon over here.”

Numerous Advantages

According to Dr. Latot Parc, the researcher designing the computer chips to process the images in-camera, a 20 megapixel Baceolus sensor not only provides the same resolution as a 40 megapixel Bayer array camera but an amazing 32 bits of color depth. I was only allowed to keep one image taken with the camera, but below are 100% crops taken with a Canon 5D III and a Baceolus preproduction camera using a Canon 135mm f/2 lens. (Yes, the Baceolus camera will mount any Canon EF lens, so a huge lens selection is already available. The Baceolus team would not comment, but I have the impression a Nikon-mount version may be in development.)

 

Overall Image

 

100% crops comparing Baceolus sensor (left) compared with standard sensor using same lens (right).

 

There are other advantages to an energy-detecting sensor. High-energy ultraviolet and low-energy infrared light can be screened out in firmware, as the camera processes the image, so an infrared filter is not required. A simple flip of a switch tells the camera to change from visible light to infrared or even ultraviolet.

 

 

A simple switch on the camera changes from normal mode . . .

 

 

To combined ultraviolet, visible, and infrared mode, or any combination of these. 

 

Additionally, the sensor can act as a ‘photon trap’, detecting only photons of a given energy range. This could allow an astrophotographer, for example, to set the camera to only accept light with wavelengths of 410 nm, 434 nm, 486 nm, and 656 nm  (the spectrum of hydrogen light) or to capture simultaneous UV and IR images of a nebula.

 

Image of 30 Doradus in full spectrum of UV through IR (above) as compared to normal Bayer sensor image (below). Credit F. Paresce and R. O'Connell

 

I did ask Dr. Parc about rumors that some beta testers had set their Baceolus sensor to detect only flesh tones, allowing a photographer to basically take an image of a person’s body right through their clothes.

“It’s certainly possible but would require long exposure times.” Dr. Parc said. “Doing so would completely overexpose the subjects face, hands, and other areas not covered by clothing, so we doubt anyone would be interested in doing that. Besides, our extensive research shows that very few photographers are interested in shooting nudes. At any rate, a person concerned about being photographed through their clothing can simply wear thick woolen underwear, which is an effective photon blocker.”

“The advantages our sensor brings to resolution and color, especially in the studio, are amazing,” Dr. Parc adds. “For example, everyone shooting a digital SLR today is frustrated by their camera's inability to properly reproduce the color Periwinkle. We hear it every day. Whether it’s a model’s periwinkle blouse, or a beautiful periwinkle flower, the photo is drab and lifeless because the Bayer array’s red, green, and blue filters just can’t reproduce periwinkle accurately. With our sensor, periwinkle is simply 2.513 electron volts, rendered every bit as accurately as any other color.”

 

True Periwinkle (left) is often rendered inaccurately by Bayer-sensor cameras, appearing too purple (center) or aquamarine.

 

Even more important, the Baceolus team believes, will be the ability to bring cell-phone staples like face detection and smile detection to full SLR cameras. They envision a setting in which the camera identifies areas of 2.03 to 2.13 Ev color (flesh tones), locking that area in as face detection. It then automatically takes a picture when a minimum of 5% of that area changes to 3.1 Ev – the near-ultraviolet color of bleached teeth -- so you get the perfect smiley face every time.

And One Disadvantage

There is only one thing not improved on a Baceolus sensor. The circuitry used to measure photonic energy does detract from dynamic range slightly. Pre-production Baceolus sensors have a DR of about 12 electron volts, less than most current SLR cameras.

“This is no problem for good photographer,” states Yug Diputs, who recently joined Baceolus as director of marketing, in slightly broken English. “Combining two images at different exposure is best way to take picture anyway. Only bad photographers are limited by dynamic range. You see this repeated on every forum on internet -- problem is always bad photographer, not bad equipment.”

When Can You Get One?

Very soon, according to what I’m hearing. Release in Europe and Japan is expected by early summer.

“We expect to have our camera on the market well before the Canon 200-400 f/4 IS lens is released,” Parc told me. “We got very lucky because Nikon has, for some reason, a large supply of autofocus sensors they can’t use which we were able to buy very cheaply. That moved our release date up several months.”

Release in the United States, unfortunately, will be delayed for quite a while. "In the entire world, only the United States, Burma and Liberia do not use the metric system. Because our system converts the electron volts into nanometers, it is calibrated for metric light and we can't guarantee image quality in areas using nonmetric light. We hope to have a firmware upgrade that will convert electron volt measurements to inches by September or October."

"Dual-system countries, like Great Britain and Canada, appear to have sufficient metric-wavelength light to allow our cameras to work properly," Diputs was quick to add.

Price has not yet been determined.

Roger Cicala

Lensrentals.com

April 1, 2013

65 Responses to “Wavelength-Detecting Sensor Eliminates Bayer Filter, Triples Resolution”

Benjamin Anderson said:

Best Baceolus review I've seen so far today.

Luckily, I'll soon be in enough money to afford one, according to the email I received from a very nice prince.

nandadevieast said:

I got it when i read about the underwear thing :)

Byron said:

Well played! Particularly the references to the Canon 200-400, and the Nikon autofocus sensors. Given the AF sensors, I think these cameras may be particularly prone to hemispatial neglect!

Jerry Russell said:

Roger, I think you should do a patent search. This just might be a really good idea. :-)

Lee Saxon said:

You know, this will probably actually happen one day, it's probably just a matter of massive computational requirements...

LensRentals Employee

Roger Cicala said:

I really rather liked "Latot Parc" myself :-)

Paul Buckley said:

The name "Eno Lirpa" sort of set the tone...

LensRentals Employee

Roger Cicala said:

Milton, we used a double barrel shotgun to hypnotize the ducks. Those are possum ducks, found only in the Southern U. S. They play dead at the sight of a shotgun.

Samuel H said:

You are a mean, mean man...

Joshin Y. said:

(Slow clap)

Pitch perfect. This one had me going right up until the end, and then the slow reveal as you turned up the ludicrous knob just gradually enough to give me a humorous journey was awesome.

Thank you!

John H. Seidel said:

Damn inch-foot light! We're behind the technology curve again.

Brandon Meyer said:

Very Nice! I was getting excited until the metric wavelength. Brilliant.

Andrew Burday said:

What I want to know is, is anybody working on integrating this with the Google Smell technology? You could combine an incredibly high resolution shot with the appropriate odor to give the viewer the feeling of truly being there. Not just a picture of clouds breaking over a mountain lake, but the smell of newly fallen rain as well. Not just an image of a rock concert, but the scent of thousands of sweaty teenagers who haven't showered since Wednesday. The possibilities are endless. I feel fortunate to be alive in a time of such innovation.

Milton Simoes said:

This new sensor is even worst than the X-trans, look at the artifacts close to the duck head. It sucks. Garbage.

By the way, what are you using to hypnotize the Duck?
It didn't move a mm between the shots. Thats the real secret here. You and your friends are developing a Freezing People Devise as part of the conspiracy of the Slow Street Photographers Association, obviously.

Nice.

Lynn said:

Had me going until the "non metric light" part! Great one, Roger!!!

Trenton said:

Non-metric light. That's fantastic, I am still rolling OTFLMAO.

And I just wanted to add Lobachevsky intersecting light source to my Einstein flash... Non-metric light. That will do the trick.

Trenton said:

Very good one, almost believed it )

Ben said:

Roger, the only way I could imagine it working is if this notional single photon detector somehow went into read-only mode once any single photon touched it. After readout it would then be reset to await another photon strike.

What do I know...just a software engineer who took some physics 15 years ago.

LensRentals Employee

Roger Cicala said:

Ben, I believe you've hit the nail right on the head. The thing that makes it sound kind of logical is the "single photon" idea, but that's not reality.

Ben said:

Roger, great one. Help me out here. I assume the reason that something like this can't work is because it would essentially need to be an array of single photon detectors. Many photons striking the photosite at the same time would make performing calculations on each photon impractical. As I understand current sensors, over the duration of the exposure time photons strike the photosite causing charge to build up and then are read off at the end. So, what is recorded is an average brightness for light that made it through that photosite's color filter over the duration of the exposure.

And that type of accumulation and read out at the end wouldn't be practical (possible?) in the system you describe. Is that about right?

Pedro Vera said:

Damn you Roger, you had me going until the disadvantage section. Well played, sir. Well played.

mazo said:

OMG, almost got me! I was really starting to ask myself how they are able to build energy-aware single-photon-detectors. Well done, very well done!

Richard Hatch said:

had me going...

John King said:

I can't imagine anything topping this story today. At least not from any non-metric light emitting countries.

aschk said:

That was even better than Google's joke! Thanks, laughed out loud like crazy when reading the last few lines!

Jean-franois said:

Lolll, non metric light.... Good one. It was interesting though

Joachim / CH said:

April 1st...

good one, though :) Nikon indeed seems to collect AF sensors they can't use. Or maybe, they should keep to use and sell the better ones.

Feng Chun said:

I was about to believe this until I read the "when can you get one"...

Roger, Did you make this yourself?

NoOne said:

Eno Lirpa :D Nice one !

Leave a Reply