Transcript G(x,y) - MIT Media Lab

Northeastern University, Fall 2005
CSG242: Computational Photography
Ramesh Raskar
Mitsubishi Electric Research Labs
Northeastern University
Nov 30, 2005
Course WebPage :
http://www.merl.com/people/raskar/photo/course/
Ramesh Raskar, CompPhoto Class Northeastern, Fall 2005
Plan for Today
• Assignment 4
• Paper reading
• Topics
– Tools: Gradient Domain and Graph Cuts
– Type of Sensors
• Project Updates
• Peter Sand, Video Special Effects
• Course feedback
Ramesh Raskar, CompPhoto Class Northeastern, Fall 2005
Reading Paper Presentations
15 minutes: 10 minute presentation, 5 minutes for discussion
Use Powerpoint slides,
Bring your own laptop or put the slides on a USB drive, (print the slides to be safe)
Format:
Motivation
Approach (New Contributions)
Results
Your own view of what is useful, what are the limitations
Your ideas on improvements to the technique or new applications (atleast 2 new ideas)
It is difficult to explain all the technical details in 15 minutes. So focus on the key concepts and
give an intuition about what is new here. Ignore second order details in the paper, instead
describe them in the context of the results.
Keep the description of the approach simple, a rule of thumb: no more than 3 equations in your presentation.
Most authors below have the powerpoint slides on their websites,
so feel free to use those slides and modify them. Be careful, do not simply present all their slides in sequence.
You should focus on only the key concepts and add your own views.
If the slides are not available on the author website, copy paste images from the PDF to create your slides.
Sometimes you can send email to the author, and s/he will send you the slides.
Ramesh Raskar, CompPhoto Class Northeastern, Fall 2005
A
Introduction
Digital photography compared to film photography
Image formation, Image sensors and Optics
B
Understanding the Camera
Parameters: Pixel Resolution, Exposure, Aperture, Focus, Color depth, Dynamic range
Nonlinearities: Color response, Bayer pattern, White balance, Frequency response
Noise: Electronic sources
Time factor: Lag, Motion blur, Iris
Flash settings and operation
Filters: Polarization, Density, Decamired
In camera techniques: Auto gain and white balance, Auto focus techniques, Bracketing
C
Image Processing and Reconstruction Tools
Convolution, Overview
Gradient domain operations, Applications in fusion, tone mapping and matting
Graph cuts, Applications in segmentation and mosaicing
Bilateral and Trilateral filters, Applications in image enhancement
Today
D
Improving Performance of Camera
Dynamic range: Variable exposure imaging and tone mapping,
Frame rate: High speed imaging using multiple cameras
Pixel resolution: Super-resolution using jitter
Focus: Synthetic Aperture from camera array for controlled depth of field
E
Image Processing and Reconstruction Techniques
Brief overview of Computer Vision techniques: Photometric stereo, Depth from defocus, Defogging
Scene understanding: Depth edges using multiple flashes, Reflectance using retinex
Denoising using flash and no flash image pairs
Multi-image fusion techniques:
Fusing images taken by varying focus, exposure, view, wavelength, polarization or illumination
Photomontage of time lapse images
Matting
Omnidirectional and panoramic imaging
F
Computational Imaging beyond Photography
Optical tomography, Imaging beyond visible spectrum,
Coded aperture imaging, multiplex imaging, Wavefront coded Microscopy
Scientific imaging in astronomy, medicine and geophysics
Next week
G
Future of Smart and Unconventional Cameras
Overview of HDR cameras: Spatially adaptive prototypes, Log, Pixim, Smal
Foveon X3 color imaging
Programmable SIMD camera, Jenoptik, IVP Ranger
Gradient sensing camera
Demodulating cameras (Sony IDcam, Phoci)
Future directions
Ramesh Raskar, CompPhoto Class Northeastern, Fall 2005
Tentative Schedule
• Nov 30th
– Project Update
– Special lecture: Video Special Effects
• Dec 2nd (Friday)
– Hw 4 due Midnite
• Dec 7th
– Computational Imaging Beyond Photography
– Special lecture: Mok3
• Dec 15th (Exam week)
– In class exam (instead of Hw 5)
– Final Project Presentation
Ramesh Raskar, CompPhoto Class Northeastern, Fall 2005
Assignment 4: Playing with Epsilon Views
See course webpage for details
•
Resynthesizing images from epsilon views (rebinning of rays)
•
http://groups.csail.mit.edu/graphics/pubs/siggraph2000_drlf.pdf
In this assignment, you will use multiple pictures under slightly varying position
to create a large synthetic aperture and multiple-center-of-projection (MCOP) images
You will create
(i) An image with programmable plane of focus
(ii) A see-through effect
--------------------------------------------------------(A) Available set
http://www.eecis.udel.edu/~yu/Teaching/toyLF.zip
Use only 16 images along the horizontal translation
(B) Your own data set
Take atleast 12-16 pictures by translating a camera (push broom)
The forground scene is a flat striped paper
Background scene is a flat book cover or painting
Choose objects with vibrant bright saturated colors
Instead of translating the camera, you may find it easier to translate the scene
Put the digital camera in remote capture time lapse interval mode (5 second interval)
Effect 1: Programmable focus
Rebin rays to focus on first plane
Rebin rays to focus on back plane
Rebin rays to focus on back plane but rejecting first plane
Effect 2: MCOP images
Rebin rays to create a single image with multiple views
Useful links
http://groups.csail.mit.edu/graphics/pubs/siggraph2000_drlf.pdf
Ramesh Raskar, CompPhoto Class Northeastern, Fall 2005
Image Tools
• Gradient domain operations,
– Applications in tone mapping, fusion and matting
• Graph cuts,
– Applications in segmentation and mosaicing
• Bilateral and Trilateral filters,
– Applications in image enhancement
Ramesh Raskar, CompPhoto Class Northeastern, Fall 2005
But, Simple Pixel Blending Creates
Ugly Artifacts
Ramesh Raskar, CompPhoto Class Northeastern, Fall 2005
Pixel Blending
Our Method:
Integration of
blended Gradients
Ramesh Raskar, CompPhoto Class Northeastern, Fall 2005
Gradient field
Nighttime image
x
I1
G1
Y
G1
Mixed gradient field
Y
x
G
G
Importance
image W
I2
Y
Final result
x
G2
G2
Daytime image
Gradient field
Ramesh Raskar, CompPhoto Class Northeastern, Fall 2005
Reconstruction from Gradient Field
• Problem: minimize error
• Estimate I’ so that
|I’ – G|
G =  I’
• Poisson equation

GX
2I’ = div G
• Full multigrid
solver
GY
I’
Ramesh Raskar, CompPhoto Class Northeastern, Fall 2005
Intensity Gradient in 1D
Intensity
Gradient
105
105
1
1
I(x)
Gradient at x,
G(x) =
G(x)
I(x+1)- I(x)
Forward Difference
Ramesh Raskar, CompPhoto Class Northeastern, Fall 2005
Reconstruction from Gradients
Gradient
Intensity
105
105
?
1
?
1
G(x)
I(x)
For n intensity values, about n gradients
Ramesh Raskar, CompPhoto Class Northeastern, Fall 2005
Reconstruction from Gradients
Gradient
Intensity
105
105
?
1
1
G(x)
I(x)
1D Integration
I(x) = I(x-1) + G(x)
Cumulative
sum
Ramesh Raskar, CompPhoto Class Northeastern, Fall 2005
Intensity Gradient in 2D
Gradient at x,y as Forward Differences
Gx(x,y) = I(x+1 , y)- I(x,y)
Gy(x,y) = I(x , y+1)- I(x,y)
G(x,y)
=
(Gx , Gy)
Grad X
Grad Y
Ramesh Raskar, CompPhoto Class Northeastern, Fall 2005
Intensity Gradient Vectors in Images
Gradient Vector
Ramesh Raskar, CompPhoto Class Northeastern, Fall 2005
Reconstruction from Gradients
Given
G(x,y)
How to compute
=
(Gx , Gy)
I(x,y) for the image ?
For n 2 image pixels, 2 n 2 gradients !
Grad X
2D
Integration
Grad Y
Ramesh Raskar, CompPhoto Class Northeastern, Fall 2005
Intensity Gradient in 2D
Recovering Original Image
Grad X
2D
Integration
Grad Y
Ramesh Raskar, CompPhoto Class Northeastern, Fall 2005
Intensity Gradient Manipulation
Recovering Manipulated Image
Grad X
Grad Y
Gradient
Processing
New Grad X
New Grad Y
Ramesh Raskar, CompPhoto Class Northeastern, Fall 2005
Intensity Gradient Manipulation
Recovering Manipulated Image
Gradient
Processing
New Grad X
2D
Integration
New Grad Y
Ramesh Raskar, CompPhoto Class Northeastern, Fall 2005
Intensity Gradient Manipulation
Recovering Manipulated Image
Gradient
Processing
New Grad X
2D
Integration
New Grad Y
Ramesh Raskar, CompPhoto Class Northeastern, Fall 2005
Intensity Gradient Manipulation
A Common Pipeline
Grad X
Grad Y
Gradient
Processing
New Grad X
2D
Integration
New Grad Y
Ramesh Raskar, CompPhoto Class Northeastern, Fall 2005
Reconstruction from Gradients
Ramesh Raskar, CompPhoto Class Northeastern, Fall 2005
Euler-Lagrange Equation
Ramesh Raskar, CompPhoto Class Northeastern, Fall 2005
Application: Compressing Dynamic Range
How could you put all this
information into one
Image ?
Ramesh Raskar, CompPhoto Class Northeastern, Fall 2005
Attenuate High Gradients
Intensity
105
1
Intensity
105
1
I(x)
Gradient
105
1
I(x)
G(x)
Maintain local detail at the cost
of global range
Fattal et al Siggraph 2002
Ramesh Raskar, CompPhoto Class Northeastern, Fall 2005
Basic Assumptions
• The eye responds more to local intensity
differences (ratios) than global illumination
• A HDR image must have some large
magnitude gradients
• Fine details consist only of smaller magnitude
gradients
Ramesh Raskar, CompPhoto Class Northeastern, Fall 2005
Gradient Compression in 1D
Ramesh Raskar, CompPhoto Class Northeastern, Fall 2005
Gradient Domain Method
Ramesh Raskar, CompPhoto Class Northeastern, Fall 2005
Basic Method
• Take the log of the luminances
• Calculate the gradient at each point
• Scale the magnitudes of the gradients with a
progressive scaling function (Large
magnitudes are scaled down more than small
magnitudes)
• Re-integrate the gradients and invert the log
to get the final image
Ramesh Raskar, CompPhoto Class Northeastern, Fall 2005
Summary: Intensity Gradient Manipulation
Grad X
Grad Y
Gradient
Processing
New Grad X
2D
Integration
New Grad Y
Ramesh Raskar, CompPhoto Class Northeastern, Fall 2005
Graph and Images
Credits: Jianbo Shi
Ramesh Raskar, CompPhoto Class Northeastern, Fall 2005
Brush strokes
Computed labeling
Ramesh Raskar, CompPhoto Class Northeastern, Fall 2005
Image objective
0 for any label
0 if red
∞ otherwise
Ramesh Raskar, CompPhoto Class Northeastern, Fall 2005
Graph Based Image Segmentation
Segmentation = Graph partition
i
Wij
j
Wij
V: graph node
Image pixel
E: edges connection nodes
Link to neighboring pixels
Wij: Edge weight
Pixel similarity
Ramesh Raskar, CompPhoto Class Northeastern, Fall 2005
Minimum Cost Cuts in a graph
Cut: Set of edges whose removal makes a graph disconnected
Si,j : Similarity between pixel i and pixel j
Cost of a cut,
A
A
Ramesh Raskar, CompPhoto Class Northeastern, Fall 2005
Graph Cuts for Segmentation and Mosaicing
Cut ~ String (loop) on a height field
Places where the string rests
Brush strokes
Computed labeling
Ramesh Raskar, CompPhoto Class Northeastern, Fall 2005
Cameras
we consider 2 types :
1. CCD
2. CMOS

Department of Computer and Information Science
CCD
separate photo sensor at regular positions
no scanning
charge-coupled devices (CCDs)
area CCDs and linear CCDs
2 area architectures :
interline transfer and frame transfer
photosensitive
storage
Department of Computer and Information Science
The CCD camera
Department of Computer and Information Science
CMOS
Same sensor elements as CCD
Each photo sensor has its own amplifier
More noise (reduced by subtracting ‘black’ image)
Lower sensitivity (lower fill rate)
Uses standard CMOS technology
Allows to put other components on chip
‘Smart’ pixels
Foveon
4k x 4k sensor
0.18 process
70M transistors
Department of Computer and Information Science
CCD vs. CMOS
•
•
•
•
Mature technology
Specific technology
High production cost
High power
consumption
• Higher fill rate
• Blooming
• Sequential readout
•
•
•
•
•
•
•
•
•
Recent technology
Standard IC technology
Cheap
Low power
Less sensitive
Per pixel amplification
Random pixel access
Smart pixels
On chip integration
with other components
Department of Computer and Information Science
Color cameras
We consider 3 concepts:
1. Prism (with 3 sensors)
2. Filter mosaic
3. Filter wheel
… and X3
Department of Computer and Information Science
Prism color camera
Separate light in 3 beams using dichroic prism
Requires 3 sensors & precise alignment
Good color separation
Department of Computer and Information Science
Prism color camera
Department of Computer and Information Science
Filter mosaic
Coat filter directly on sensor
Demosaicing (obtain full color & full resolution image)
Department of Computer and Information Science
Filter wheel
Rotate multiple filters in front of lens
Allows more than 3 color bands
Only suitable for static scenes
Department of Computer and Information Science
Prism vs. mosaic vs. wheel
approach
# sensors
Separation
Cost
Framerate
Artefacts
Bands
Prism
3
High
High
High
Low
3
High-end
cameras
Mosaic
1
Average
Low
High
Aliasing
3
Low-end
cameras
Wheel
1
Good
Average
Low
Motion
3 or more
Scientific
applications
Department of Computer and Information Science
New color CMOS sensor
Foveon’s X3
better image quality
smarter pixels
Department of Computer and Information Science
Foveon: Thick Sensor
Pixim
High Dynamic Range
http://www.cybergrain.com/tech/hdr/
Fuji's SuperCCD S3 Pro camera has a chip with high and low sensitivity sensors
per pixel location to increase dynamic range
Gradient Camera
Sensing Pixel Intensity Difference with
Locally Adaptive Gain
Ramesh Raskar, MERL
Work with Jack Tumblin, Northwestern U,
Amit Agrawal, U of Maryland
Natural Scene Properties
Intensity
Gradient
105
105
1
1
x
x
Intensity Histogram
1
5
Gradient Histogram
5
5
Original Image
Intensity Camera Image
Intensity values ranging from 0 to 1800
Intensity ramp plus low contrast logo
8 bit camera for 1:1000 range
Problem: . saturation at high intensity regions
Log Camera Image
8 bit log for 1:106 range
Problem: Visible quantization effects at high intensities
Locally Adaptive Gain
Pixel divided by the average of local neighborhood.
Thus the low frequency contents are lost and only
detail remains.
Gradient Camera Image
In proposed method, we sense intensity
differences. We use a 8 bit A/D with
resolution of log(1.02) to capture 2%
contrast change between adjacent pixels.
Notice that the details at both high and low
intensities are captured.
Gradient Camera
• Two main features
1. Sense difference between neighboring pixel intensity
At each pixel, measure (x , y ) ,
x = Ix+1,y - Ix,y
,
y = Ix,y+1 - Ix,y
2. With locally adaptive gain
• Gradient camera is very similar to locally adaptive gain camera
• Locally Adaptive Gain Camera
– Gain is different for each pixel
– Problem: Loses low frequency detail and preserves only high frequency features (edges)
• Gradient Camera
– The gain is same for four adjacent pixels
– Difference between two pixels is measured with same gain on both pixels
– Reconstruct original image in software from pixel differences by solving a linear system
(solving Poisson Equation)
Camera Pipeline
On-board Hardware
Difference
between
pixels
Local gain
adaptive to
difference
Software
2D Integration to
reconstruct the
image
Detail Preserving
Intensity Camera
Log Intensity Camera
Gradient Camera
Intensity cameras capture detail but lose range
Log cameras capture range but lose detail
Quantization
Intensity Histogram
1
105
Gradient Histogram
Original Image
Uniform
quantization 3 bits
-105
105
GradCam requires
fewer bits
Log Uniform
quantization 3 bits
Log Uniform gradients
quantization 3 bits
In the reconstructed
image, error is pushed
to high gradient pixel
positions which is
visually imperceptible
High Dynamic Range Images
Scene
Intensity camera
saturation map
Gradient camera
saturation map
Intensity camera fail to capture range
Gradients saturate at very few isolated pixels
3D Cameras
• Time of flight or phase
– ZCam
– Canesta
• Phase Decoding of modulated illumination
– Phase difference = depth
– Magnitude = reflectance
• Structured Light
– Binary coded light and triangulation
Novel Sensors
•
•
•
•
•
•
Gradient sensing
HDR Camera, Log sensing
Line-scan Camera
Demodulating
Motion Capture
3D
Fantasy Configurations
• ‘Cloth-cam’: ‘Wallpaper-cam’
elements 4D light emission and 4D capture in
the surface of a cloth…
• Floating Cam: ad-hoc wireless networks form
camera arrays in environment…
• Other ray sets:
Multilinear cameras, canonical ‘basis’ cameras
(linear combination of 8 types)
McMillan’04, ‘05