Transcript I rock!

Online Construction of
Surface Light Fields
By Greg Coombe, Chad Hantak, Anselmo
Lastra, and Radek Grzeszczuk
1
Image-Based Modeling
*
Benefits: *
 Photorealistic content
 Modeling by acquisition
Difficulties:
 Sampling issues
 Lack of feedback
2
Motivation
Casual Capture
Pick up a video camera, wave it around the
model, and capture reflectance model
3
Surface Light Field
 A surface light field is a
representation of the appearance
of a model with known geometry
and static lighting:
(θ,Φ)
surface position
f ( ,  , u, v)
(u,v)
viewing direction
4
Our system
 Incremental system for online capture,
construction, and rendering of surface light
fields
 Each image is processed as it is captured

Interactive feedback

Guide user to undersampled regions
 Data-driven heuristic
 Structured method for handling missing data
5
Previous Work
 Regular parameterizations light field data [Levoy96, Gortler96]
 Sparse, scattered data [Debevec96, Debevec98, Buehler01]
 BRDF capture systems [Dana99, Lafortune97, Marschner99,
Debevec01, Gardner03]
 Function fitting

Lafortune BRDF [McAllister02], Torrance-Sparrow BRDF [Sato97],
Clusters of BRDFs [Lensh01], Homomorphic Factorization
[McCool?], Bi-quadratic polynomials [Malzbender01]
 Online Methods
 Fixed viewpoint, progressive refinement [Matusik04]
 Adaptive Meshing of light field [Schirmacher99]
 Streaming non-linear optimization [Hillesland04]
 Data Mining [Brand02, Roweis97]
6
SLFs using PCA
 Represent 4D function as a matrix
 Use Principal Component Analysis [Noshino01,
Chen02]


For each surface patch, break the 4D function into two
2D functions
Store the principal components in texture maps
rank
f ( ,  , u, v)   g ( ,  ) * h(u, v)
7
SLFs using PCA
 Problems:



Requires that the entire data set be available
at once.
Difficult to locate undersampled regions
Requires recomputation when new images are
added
SVD
*
8
Online SVD
 Online SVD [Brand02] is a incremental PCA
 Update output matrices one sample at a time
 Advantages:


Never store the entire data matrix at once
Stream images
...
Online SVD
*
9
Online SVD Algorithm
1. Existing rank-r PCA
A  USV T
2. Project new image samples onto eigenspace
m UTa
p
3. Compute the orthogonal component
j
p  a - Um
4.
If ||p|| is less than a threshold, then we just compute
the rotations and update the eigenspace.
U’ = URu
5.
V’ = VRv
Otherwise, the current rank r is insufficient, so we
append a new eigenvector.
U’ = [U; m]Ru V’ = VRv
6. The rotations are computed by re-diagonalizing this
matrix
 diag ( s ) m  diagonalize

 [U ', S ',V ']
p
 0
10
Error
Rank
Reference PCA
Online SVD
11
Imputation
 Matrix factorization approach
requires fully resampled data
matrices
 Missing data is a common problem

Occlusion, meshing errors
 Imputation is the process of filling
in these holes with “reasonable”
guesses

Better than zeros or mean
 Use the current Online SVD
approximation to generate these
missing values

In practice, need about 5-10 initial
images, and 50%+ coverage
Missing data
in red
Imputed
 Further details in [Brand03]
12
Automatic Feedback
 Aid the user in capturing
surface light fields

Direct attention towards
undersampled areas
 Use a data-driven
quality heuristic


E = sqrt( e0^2 + e1^2)
e0 – variation over
hemisphere
e1 – variation over
surface
13
System
Video
Camera
Real Time Pose
Estimation
Capture PC
Visibility*
Resampling*
Render PC
Online SVD
Rendering
Quality
Heuristic
* From Intel’s OpenLF system
14
Results
Captured image
SLF before
incorporating
new image
SLF after
incorporating
new image
15
Results
16
Timing
17
Conclusion
 We present a method with


Incremental construction
Data-driven quality heuristic
 Online SVD is well-suited for streaming model


Combine distribution and solution
Requires only one pass over data
18
Future Work
 We are still a long ways from “Casual
Capture”


Geometry must be known a priori
Fixed illumination conditions
 Implementation issues

Real-time pose estimation errors
19
20
Online SVD
 Problem: Multiple small rotations can
accumulate error
 Brand proposes splitting output matrices to
avoid accumulating error
A SVD

 USV T
A SVD

 UU ' SV 'V T
 Advantages:
 Due to small working
set, most data in cache
(fast)
21
Higher-Dimensional Problems
 We would like to extend Online SVD to
higher-order factorizations

Moveable light source
 Introduces another dimension of data
f (1 , 1 , u, v, 2 , 2 )   g (1 , 1 ) * h(u, v) *k (2 , 2 )
View maps
Surface maps
Light maps
 Requires tensor product expansion
22
2. Convergence
Averaged over twenty random vertices
23
3. Sensitivity
24
Imputation
Incomplete
sample: (-1, ??)
Existing SVD
approximation
Impute
using SVD
25