Transcript dror_sctv_01 - MIT Computer Science and Artificial Intelligence

Surface Reflectance Estimation
and Natural Illumination Statistics
Ron Dror, Ted Adelson, Alan Willsky
Artificial Intelligence Lab, Lab for Information and Decision Systems
http://www.ai.mit.edu/people/rondror
July 13, 2001
1
Reflectance Estimation Problem


Surface appearance depends on surface
reflectance, illumination, and geometry.
We wish to estimate reflectance under
unknown illumination.
2
Human vision
3
Machine vision
4
Motivation

Recognize materials.




Reflectance, like texture, is a primary visual
characteristic of materials.
Material recognition is important in its own right
and as a complement to shape recognition.
Capture real-world reflectances for rendering
purposes.
Rectify classical motion, stereo, and shapefrom-shading algorithms.
5
Reflectance estimation is ill-posed




A surface’s BRDF f(i,i; r,r)
specifies how much of the light
incident from any one direction
is emitted in any second
direction.
The brightness of a surface patch to a viewer is a
weighted integral over illumination from all directions.
Goal: estimate reflectance (function of 4 variables) from
an image (function of 2 variables) under unknown
illumination from every direction (function of 2 variables
at every point on the surface).
More degrees of freedom than measurements, even
assuming known geometry, homogeneous reflectance. 6
Bayesian formulation


Find the most likely reflectance given image
data.
Given image data R, find most likely reflectance
f by marginalizing over illumination I.
fˆ  arg max P( f | R)  arg max P( f )  P( I ) P( R | f , I )dI
f



f
I
P(f) – prior probability of a reflectance function
P(I) – prior probability of an illumination field
Challenges:


P(f) and P(I) are not readily available.
Integration over all illuminations is computationally
daunting!
7
Two simplified formulations
1. Classification (finite but arbitrary classes):
2. Parameter estimation using a reflectance
model (regression).
8
Prior information: illumination

Assuming distant light sources, we can represent
illumination by a single spherical image.
Projection of
spherical map
Rendered surfaces
9
Statistical models of illumination

Illumination maps possess statistical
regularities akin to those of “natural images”.
Histogram of pixel
intensities
Histogram of wavelet
coefficients
10
Importance of illumination
statistics for humans

People recognize reflectance more easily
under realistic illumination than simplified
illumination.
11
Ward reflectance model

A physically realizable variant of the Phong model
(satisfies energy conservation and reciprocity).
d
1
exp(  tan 2  /  2 )
f ( i , i ; r , r ) 
 s

4 2
cos i cos r
diffuse
component



specular
lobe
d: proportion of incident radiation reflected diffusely.
s: proportion of incident radiation reflected specularly.
: surface roughness, or blur in specular component.
12
Effect of Ward model parameters
on pixel intensity histogram
probability
Original
pixel intensity
13
Effect of Ward model parameters
on pixel intensity histogram
d=.1
probability
Original
pixel intensity
14
Effect of Ward model parameters
on pixel intensity histogram
d=.2
probability
Original
pixel intensity
15
Effect of Ward model parameters
on pixel intensity histogram
d=.3
probability
Original
pixel intensity
16
Effect of Ward model parameters
on pixel intensity histogram
d=.4
probability
Original
pixel intensity
17
Effect of Ward model parameters
on pixel intensity histogram
d=.4
s=.1
probability
Original
pixel intensity
18
Effect of Ward model parameters
on pixel intensity histogram
d=.4
s=.2
probability
Original
pixel intensity
19
Effect of Ward model parameters
on pixel intensity histogram
d=.4
s=.3
probability
Original
pixel intensity
20
Effect of Ward model parameters
on pixel intensity histogram
d=.4
s=.4
probability
Original
pixel intensity
21
Effect of Ward model parameters
on pixel intensity histogram
d=.4
s=.4
=0
probability
Original
pixel intensity
22
Effect of Ward model parameters
on pixel intensity histogram
d=.4
s=.4
=.05
probability
Original
pixel intensity
23
Effect of Ward model parameters
on pixel intensity histogram
d=.4
s=.4
=.1
probability
Original
pixel intensity
24
Effect of Ward model parameters
on pixel intensity histogram
d=.4
s=.4
=.15
probability
Original
pixel intensity
25
Dependence of statistics on
reflectance parameters
Real-world illuminations
d
s

Random checkerboard illuminations
d
s

skew
kurt
10%
50%
90%
mean
var
skew
kurt
10%
50%
90%
mean
var
skew
kurt
10%
50%
90%
mean
var

normalized
derivative
normalized
derivative

Each reflectance clusters in
feature space
black matte
black shiny
white matte
white shiny
gray shiny
chrome
27
A system for classification


“Learn” relationships between features of the
observed image and reflectance classes.
For a distant viewer and convex object, radiance
depends only on local surface orientation.
28
Implementation flow chart

This leaves two open questions:


How to select relevant statistics?
How to build a classifier?
29
SVM classifier


Support vector machines are relatively robust to the
inclusion of extraneous features.
A sample classifier based on just two statistics:
black matte
black shiny
white matte
white shiny
gray shiny
chrome
30
Training data sets
6 Ward model reflectances, 9 illuminations (Debevec)
11 Ward model reflectances, 100 illuminations (Teller)
9 real spheres, photographed at seven locations
31
Performance
Rendered:
6 BRDFs,
9 illums
16.7%
Rendered:
11 BRDFs,
100 illums
9.1%
Photos:
9 spheres,
7 illums
11.1%
6 handselected
features
98.1%
98.5%
93.7%
6 autoselected
features
96.3%
94.4%
74.6%
6 PCA
features
79.6%
86.8%
71.4%
Chance
32
Conclusions



Our classifier rivals human performance when
geometry is known and reflectance is homogeneous.
Although ill-posed, reflectance estimation under
unknown natural illumination is tractable.
The statistical structure of natural illumination plays
an essential role in visual reflectance estimation by
humans and machines.
33
Future directions

In progress or submitted:




Extension to complex or unknown geometry; robustness to
incorrect assumed geometry.
Quantitative study of natural illumination statistics.
Measurement of human ability to estimate reflectance from a single
image without contextual information.
Additional goals:


Rigorous theoretical foundation – link illumination statistics directly
to selected features.
Estimate spatially varying reflectance.
34
Misclassifications
Illumination
Misclassified
image
Potential source
of confusion
35
Feature selection


By hand, based on insights
developed through work with
Ward model.
Using automated feature
selection method, which iterates
the following steps:



Estimate marginal probability
density of each feature for each
class.
Select the feature that minimizes
Bayes error.
Regress remaining features against
selected features, and subtract off
predicted values.
36
Auto-selected features

6 features selected based on images of spheres with
6 Ward model reflectances under 9 illuminations:







10th percentile of 4th finest vertical subband
90th percentile of pixel intensity
variance of 3rd finest diagonal subband
10th percentile of pixel intensity
90th percentile of 4th finest vertical subband
median of 3rd finest horizontal subband
Hand-selected features




mean and 10th percentile of original image
variance of two finest vertical subbands
ratio of these two variances
kurtosis of second finest vertical subband
37
Complex vs. simple illumination


People recognize reflectance more easily under
realistic illumination than simplified illumination.
A reflectance estimation algorithm which takes
advantage of natural illumination statistics will fail for
atypical illumination.
38
Human reflectance estimation


Pool balls
Note ambiguity in overall color and brightness
when matte spheres are viewed in isolation.
39
Related Work

Yu, Debevec, Malik, and Hawkins, ’99


Sato, Wheeler, and Ikeuchi, ’97


Reflectance and geometry under simple lighting, using color
separation.
Pellacini, Ferwerda, Greenberg, ’00


Reflectance under known illumination.
Tominaga and Tanaka, 1999, ’00


Reflectance and geometry from photos and laser range
finder, with known illumination.
Marschner, Greenberg, et al., ’98, ’99


Reflectance and illumination from multiple photos.
Perceptually uniform gloss space for graphics.
Ramamoorthi and Hanrahan, ’01

Determine when reflectance estimation problem is wellposed.
40
Photographic data

Nine different spheres under the same illumination.
41
Photographic data

Same spheres under a second illumination.
42
Photographic data

Same spheres under a third illumination.
43
Illumination conditions
44
Rendered data set

6 spheres under one illumination condition
45
Rendered data set

6 spheres under a 2nd illumination condition
46
Is this task even possible?

Humans are good at it.
Photographs of three
spheres under two
illumination conditions


In psychophysical tests, we found that humans could match
synthetic images of surfaces with similar reflectances
rendered under different real-world illuminations.
Two conclusions:


Humans rely on prior information in estimating reflectance.
Humans estimate reflectance without explicitly estimating
illumination.
47
Additional applications


Rectify motion, stereo, and shape-fromshading algorithms.
Gideon Stein, unpublished
Capture real-world reflectances for rendering
purposes.
Yu, Debevec, Malik,
Hawkins, SIGGRAPH 1999
48
Debevec spheres
49
mean
var
skew
kurt
10%
50%
90%
s
Random checkerboard illuminations
d
s
skew
kurt
10%
50%
90%
mean
var
skew
kurt
10%
50%
90%
mean
var
skew
kurt
10%
50%
90%
mean
var
Real-world illuminations
d
skew
kurt
10%
50%
90%
mean
var
skew
kurt
10%
50%
90%

mean
var
normalized
derivative

normalized
derivative
Dependence of statistics on
reflectance parameters

