a4academics.com
Download
Report
Transcript a4academics.com
MOTION DETAIL
PRESERVING OPTICAL
FLOW ESTIMATION
Li Xu, Jiaya Jia and Yasuyuki Matsushita
CONTENTS
INTRODUCTION
CONVENTIONAL OPTICAL FLOW
OPTICAL FLOW MODEL
ROBUST DATA FUNCTION
EDGE-PRESERVING REGULARIZATION
MEAN FIELD APPROXIMATION
OPTIMIZATION FRAMEWORK
EXTENDED FLOW INITIALIZATION
CONTINUOUS FLOW OPTIMIZATION
OCCLUSION-AWARE REFINEMENT
CONCLUSION
REFERENCE
INTRODUCTION
Optical flow is the apparent motion of brightness
patterns in the image
Ideally, optical flow would be the same as the motion field
The motion field … is the projection into the image of
three-dimensional motion vectors
3
CONVENTIONAL OPTICAL FLOW
Dominant Scheme: Coarse-to-Fine Warping Framework
The input image is represented as a tree of
regions
The optical flow is estimated by optimizing an
energy function
optical flow estimation on the coarser level
region-tree is used for defining region-wise finer
displacement samplings for finer level regiontrees
Middlebury optical flow evaluation
MULTI-SCALE PROBLEM IN COARSE-TO-FINE WARPING
(a)
(b)
(c)
Coarse level (e)
(d)
Fine level
(f)
(a)-(b) Two input patches.
(c) Flow estimate using the coarse-to-fine variational setting. (d) Our
flow estimate. (e)-(f) Two consecutive levels in the pyramid. Flow fields
are visualized using the colour code
Large displacement optical flow may not be well estimated
• Inclination to diminish small motion structures when
spatially significant and abrupt change of the displacement
exists.
•
Solution
•
Improve flow initialization to reduce the reliance of the
initialization from coarser levels and enables recovering
many motion details at each scale
Our Work
Framework
Extended coarse-to-fine motion estimation for both large and
small displacement optical flow
Model
A new data term to selectively combine constraints
Solver
Efficient numerical solver for discrete-continous optimization
OPTICAL FLOW MODEL
ROBUST DATA FUNCTION
Objective function for development of a new optimization
procedure
u denotes the flow field that represents the displacement
between frames I1 and I2 ,x represents the 2D coordinates
Data Constraints
Color constraint
D1(u,x)=||I2(x+u)-I1(x)||
Gradient constraint
D I(u,x)=ζ|| I2(x+u)- I1(x)||
Data term
ED (u,x)=∑1/2 D1(u,x)+1/2 D
(u,x)
I
Data cost distributions w.r.t different displacement values
A good model should only use the more fitting constraint, but not
both of them
Define a binary weight map α(x):Z―>{0,1} to switch between the two
terms. When α(x)=1, the gradient constraint is favored. Otherwise,
we select color constraint
EDGE-PRESERVING REGULARIZATION
Smoothness term, it maintains motion discontinuity
The final objective function is defined as E(u,α)= ED(u,α)+λEs(u)
where λ is the regularization weight.
MEAN FIELD APPROXIMATION
The effective energy is written as Eeff(u)= EeffD(u)+λEs(u)
The effective data function
β is the inverse temperature
β plays a key role in shaping the data function.
Small β makes the distribution close to the original one with α=0.5
A relatively large β yields the distribution approaching the lower envelope of
the costs with α=0 and α=1
OPTIMIZATION FRAMEWORK
INPUT: A pair of images for optical flow estimation
1.
Construct pyramids for both of he images and set the initial level l=0 and
uɭ=0 for all pixels
2.
Propagate uɭ to level l+1
Extended Flow Initialization
3.1. Detect and match SIFT features in level l+1
3.2 Perform patch matching in level l+1
3.3 Generate multiple flow vectors as candidates
3.4 Optimize flow
1.
Continous Flow Optimization
4.1 Compute the ᾱ map
4.2 Solve the energy function
5. Occlusion-aware Refinement
6.
If l≠n-1 where n is the total number of levels, l=l+1 and go to step 2
OUTPUT: The optical flow field
1.
EXTENDED FLOW INITIALIZATION
Finding multiple extended displacements (denoted as {u0,u1,….,un}) to
improve estimation in uc
uc which is the flow field computed in the immediately coarser level.
The steps adopted to obtain the extended displacements.
SIFT Feature Detection
Selection
Expansion
Patch Matching
Matching Field Fusion
A)SIFT Feature Detection
SIFT feature detection and matching can efficiently capture large
motion for objects undergoing translational and rotational motion
Employ only the sparse matching of discriminative points, which
avoids introducing many ambiguous correspondences and outliers.
employ discrete optimization to only select the most credible
candidates.
B) Selection
The computed displacement vectors by feature matching are
denoted as {s0, . . . ,sn}
Robustly screen out the duplicated vectors
Compute the euclidean distance between each
si and all uc
If all results are greater than 1 (pixel), we regard si as a new flow
candidate.
Repeat this process for all si, and denote the m remaining
candidate vectors {sk0, . . . ,skm-1}
C)Expansion
The m remaining vectors {sk0, . . . ,skm-1} represent possible
missing motion in the present flow field uc
Determine whether or not they are better estimates to replace the
original ones
D) Patch Matching
sometimes it still misses some motion vectors
because small texture-less objects may not have distinct
features
SIFT descriptors, the patches on which they operate
should at least contain 16x16 samples as suggested.
Compute the matching field un by minimizing energy
Total of five color and gradient channels used.
Noise can be quickly rejected in the following optimization step
with the collection of a set of flow candidates for each pixel.
E) Matching Field Fusion
The m+1 new motion fields {u0,..um-1,un} together with the
original uc, comprise several motion candidates for each pixel in
the present image scale.
Selection of the optimal flow among the m+2 candidates for
each pixel is a labeling problem
Solved by discrete optimization efficiently
Extended flow initialization.
CONTINUOUS FLOW OPTIMIZATION
Refine flow u0 through continuous optimization
We propose decomposing the optimization into three simpler problems,
each of which can have the globally optimal solution.
Auxiliary variables p and w, representing the substituted data cost and
flow derivatives
The optimal solution is given by the shrinkage
formula
INPUT: Images Ik ,initial flow field u0,weights αk
Perform linerization at u0
η= η0
repeat
Compute pk
θ= θ0
repeat
Compute w
Compute du
θ=θ/3
until θ=θmin
η= η/3
until η< ηmin
ur=u0+du
OUTPUT: Refined flow field ur
θ and η are critical parameters that should be small.
fixing them to constants typically results in slow convergence
Initially sets θ and η to large values to allow warm-starting and
then decreases them in iterations toward the desired
convergence
θ and η minimum values are set to 0.1
OCCLUSION-AWARE REFINEMENT
Multiple pixels mapping to the same point in the target image
using forward warping are possibly occluded by each other
detect occlusion using the mapping uniqueness criterion
expressed as
o(x)=T0.1(f(x+u(x))-1)
f(x+u(x)) is the count of reference pixels mapped to position
x+u(x) in the target view using forward warping
measure of the data confidence based on the occlusion detection
is expressed as c(x)=max(1-o(x),0.01)
CONCLUSION
A unified framework to preserve motion details in both
small and large-displacement scenarios.
It include the selective combination of the color and
gradient constraints, sparse feature matching, and dense
patchmatching to collect appropriate motion candidates
Limitations
--Texture-less motion details
--Large occlusions
REFERENCE
1.
2.
3.
4.
L. Alvarez, J. Esclarin, M. Lefebure, and J. Sanchez, “A PDE Model for
Computing the Optical Flow”
P. Anandan, “A Computational Framework and an Algorithm for the
Measurement of Visual Motion”
S. Baker, D. Scharstein, J. Lewis, S. Roth, M.J. Black, and R. Szeliski, “A
Database and Evaluation Methodology for Optical Flow ”
C. Barnes, E. Shechtman, A. Finkelstein, and D.B. Goldman,
“Patchmatch: A Randomized Correspondence Algorithm for Structural
Image Editing”