Transcript A Digital Camera to Rival the Human Eye

A Digital Camera to
Rival the Human Eye
Dr. Dil Joseph, Assistant Professor
Electrical and Computer Engineering
University of Alberta
Synopsis


Although digital cameras have in many ways
surpassed the capabilities of film cameras, the
human eye remains the ultimate standard for
comparison and it vastly outperforms the best
cameras in many respects
Inspiration from biology may help us to
“perfect” the technology of imaging
Outline






Film cameras
Digital cameras
Human eyes
My research
Your questions
However, let us first put
things in perspective…
In Perspective



Society has invested over millennia in developing technology
to record observed scenes on an independent medium
Artistic license aside, the aim has been to render images with a
maximum of perceptual accuracy using a minimum of effort
The digital camera is merely a culmination of the above but
that does not mean the technology development is over
Norbert Aujoulat ©
National Centre of
Prehistory, France
http://www.culture.
gouv.fr/culture/arcn
at/lascaux/en/
History of Film Cameras


Two scientific processes
required development:
optical and chemical
During the Renaissance,
the optical process was
understood well enough
for artists to begin using
the camera obscura as a
drawing aid (see right)
History of Film Cameras



Thomas Wedgwood published his work on
photograms in 1802; he recorded short-lived
images using silver nitrate on leather
John Herschel, who popularized the word
“photography”, discovered in 1819 how to
dissolve (unexposed) silver salts
Although Nicéphore Niépce produced the first
photograph in 1827, Louis Daguerre was the
most famous of the early inventors
History of Film Cameras



The Daguerreotype was
revealed to the public in
1839 to much acclaim
Images were produced
on copper plates coated
with silver iodide
The drawbacks of the
process were the cost,
long exposure time and
irreproducibility
History of Film Cameras



Fox Talbot introduced
the Calotype in 1840,
the first process to
involve negatives
Positive prints could be
made cheaply on paper
Others went on to create
processes that had both
better image quality and
shorter exposure times
Film Cameras Today



Film cameras today have come a long way
since the nineteenth century, especially by
improved spatial and contrast resolution
Thanks to the early work of Maxwell and
“recent” work of others, including industry,
colour photography thrives today
However, except for low cost and large format
photography, amateurs and professionals are
replacing their film cameras with digital ones
History of Digital Cameras



Einstein published three
papers in 1905, one of
which explained the
photoelectric effect
Photons striking a metal
liberate electrons, which
can carry a current
Kinetic energy depends
on light frequency while
the number of electrons
depends on intensity
History of Digital Cameras



The advent of Quantum Mechanics in the early
1900s led to advances in solid-state physics
Russell Ohl invented the p-n junction, a
photosensitive diode that (usually) passes
current in only one direction, in 1940
John Bardeen, Walter Brattain and William
Shockley invented the transistor in 1947, a
semiconductor device that can act as an
amplifier or a switch
History of Digital Cameras




Working independently, Jack Kilby and Robert
Noyce invented the integrated circuit in 1959
Transistors, diodes, capacitors, resistors and
wiring could be fabricated on a single crystal
of semiconductor material, e.g. silicon
George Smith and Willard Boyle invented the
CCD camera in 1969
Digital, as opposed to analog, CCD cameras
came in the late 1980s
Digital Cameras Today



A digital camera consists of many components
(optics, housing, battery, memory etc.), of
which the image sensor is principal
With market revenues of $1.7 billion in 2003,
there is widespread research and development
in a variety of image sensor designs
Modern designs may be either charge coupled
device (CCD) sensors or complementary
metal-oxide-semiconductor (CMOS) sensors
CCD Image Sensors





March photo-generated
charge systematically from
an array of pixels to an
output amplifier
Established technology
High resolution, high
sensitivity, low noise
Fabrication process is
optimised for imaging
Market share of 93% in
1999 (49% in 2004?)
CMOS Image Sensors





Work like memory array
with photosensitive pixels
instead of memory cells
Signal processing may be
included on the same die
High yield of working chips
and good video performance
May be fabricated by the
makers of microchips
Market share of 7% in 1999
(51% in 2004?)
Test Your Knowledge


In what ways are the
digital cameras on the
market better than
human eyes?
And, in what ways are
human eyes better than
the digital cameras on
the market?
History of the Human Eye


Darwin wrote in 1859: “To suppose that the
eye…could have been formed by natural
selection, seems, I freely confess, absurd in the
highest possible degree. Yet reason tells
me…the difficulty of believing that a perfect
and complex eye could be formed by natural
selection…can hardly be considered real.”
Biologists have since filled in many gaps
The Human Eye Today
© John W. Kimball
http://users.rcn.com
/jkimball.ma.ultran
et/BiologyPages/


There are 6–7 million cones, for bright light vision
with fine detail and colour, and 75–150 million rods,
for dim light vision with coarse detail and no colour
Today, the resolution of digital pixels (5–10 μm) is
closing on the resolution of foveal cones (2–3 μm)
The Human Eye Today



The human eye is a remarkable organ not only
because of its ability to sense images but
especially because of its ability to process the
image before sending a signal to the brain
The eye encodes the abundant visual input in
such a way that the limited neural output
retains the most significant descriptors of the
scene while the rest are discarded
Most digital cameras do little pre-processing
Dynamic Range



The human eye will capture brightness and colour detail (left)
that a typical digital camera will fail to capture (right)
These images were computed from a series of photographs—
no human eye or digital camera was harmed in the process
Photographic data came from Paul Debevec and the visual
model came from Gregory Larson and his colleagues
Colour Constancy



The human eye factors
out the illumination to
large measure when
identifying colours
A typical digital camera
uses a simple method to
balance colours that will
fail noticeably at times
Better methods exist but
they are complicated
Acquired colours
Perceived colours
Typical correction
Better correction
© Computational Vision Lab, Simon
Fraser University (annotation added)
http://www.cs.sfu.ca/~colour/research/col
our-constancy.html
Fixed Pattern Noise



Any two photoreceptors
in a retina and any two
photodetectors in an
image sensor are never
perfectly identical
A varying response to
light stimulus causes
“fixed pattern noise”
Camera designers work
to correct or reduce the
FPN; neurons adapt (?)
Temperature Stability



Unlike the human body,
a digital camera cannot
regulate temperature
The response of a pixel
to a light stimulus will
depend on temperature
When the temperature
dependence varies from
one pixel to another,
FPN will appear
My Research



Pixel & readout circuits
and image processing
may be improved
Camera designers face
challenges when using
CMOS processes with
feature sizes ≤ 180 nm
Photodetectors may be
fabricated in a thin film
deposited on top of the
CMOS microchip
Linear Pixels



Linear pixels (either
CCD or CMOS type)
“count” photons over a
discrete period of time
They produce a voltage
directly proportional to
the light intensity
Unfortunately, the
response may saturate
white or black easily
© IMS Chips
http://www.ims-chips.de/
Logarithmic Pixels



Logarithmic pixels
(CMOS only) measure
the “rate” of photon
incidence continuously
They produce a voltage
directly proportional to
the logarithm of the
light intensity
The response is similar
to that of human vision
© IMS Chips
http://www.ims-chips.de/
My Research




Logarithmic pixels are
great for high dynamic
range imaging but…
FPN is worse compared
to typical linear pixels
Colours are worse than
for typical linear pixels
Temperature stability is
hardly understood for
log (and linear) pixels
Fixed Pattern Noise



By studying the causes
of FPN, I developed a
method to correct it
Images of a uniform
surface are used to
define corrections
My correction reduces
the FPN to the same
order as the random
temporal noise
Colour Rendition

Reference

I also showed how to
render accurate colours
with logarithmic pixels
Images of a reference
chart are used to define
a colour mapping
The perceptual error of
the rendered colours is
comparable to the error
of typical cameras
Rendered

Temperature Stability



The dark response of a
pixel depends only on
temperature
Thus, it may be used to
correct FPN due to
temperature in the light
response
Experiments support
this conclusion but
simulation results are
shown for clarity
My Research



A digital camera should render images with a
maximum of perceptual accuracy (as per the
human eye) using a minimum of effort
A logarithmic imager responds to light much
like the human eye; pre-processing and better
electronics will improve the image quality
My past work has focused on maximising the
perceptual accuracy; my future work will focus
on minimising the effort required
Your Questions
Resources




Robert Leggat, “A History of Photography”,
http://www.rleggat.com/photohistory/
Mary Bellis, “The History of Computers”,
http://inventors.about.com/
Bob Patterson, “The Evolution of Eyes”,
http://www.origins.tv/darwin/eyes.htm
Microsoft Office Online, Clip Art and Media,
http://office.microsoft.com/clipart/