Transcript mf_bmvc12_ver3

Improved Initialisation and Gaussian
Mixture Pairwise Terms for Dense
Random Fields with Mean-field Inference
Vibhav Vineet, Jonathan Warrell,
Paul Sturgess, Philip H.S. Torr
http://cms.brookes.ac.uk/research/visiongroup/
1
Labelling problem
Assign a label to each image pixel
Object segmentation
Stereo
Object detection
2
Problem Formulation
Find a labelling that maximizes the conditional probability
or minimizes the energy function
3
Problem Formulation
• Grid CRF leads to over smoothing around boundaries
Inference
4
Problem Formulation
• Grid CRF leads to over smoothing around boundaries
• Dense CRF is able to recover fine boundaries
Inference
Inference
5
Inference in Dense CRF
• Very high time complexity
• graph-cuts based methods not feasible
6
Inference in Dense CRF
• Filter-based mean-field inference method takes 0.2 secs*
• Efficient inference under two assumptions
• Mean-field approximation to CRF
• Pairwise weights take Gaussian weights
*Krahenbuhl
et al. Efficient Inference in Fully Connected CRFs with Gaussian
Edge Potentials, NIPS 11
7
Efficient Inference in Dense CRF
• Mean-fields methods (Jordan et.al., 1999)
• Intractable inference with distribution
• Approximate distribution
from tractable family
P
8
Naïve Mean Field
• Mean-field approximation to CRF
• Assume all variables are independent
9
Efficient Inference in Dense CRF
• Assume Gaussian pairwise weight
Mixture of Gaussians
Bilateral
Spatial
10
Marginal update
• Marginal update involve expectation of cost over
distribution Q given that x_i takes label l
Expensive message passing step is solved using highly
efficient permutohedral lattice based filtering approach
• MPM with approximate distribution:
11
Q distribution
Q distribution for different classes
across different iterations
Iter 0
Iter 1
Iter 2
Iter 10
12
Two issues associated with the method
• Sensitive to initialisation
• Restrictive Gaussian pairwise weights
13
Our Contributions
Resolve two issues associated with the method
• Sensitive to initialisation
• Propose SIFT-flow based initialisation method
• Restrictive Gaussian pairwise weights
• Expectation maximisation (EM) based strategy
to learn more general Gaussian mixture model
14
Sensitivity to initialisation
• Experiment on PascalVOC-10 segmentation dataset
Mean-field
Alpha-expansion
Unary potential
28.52 %
27.88%
Ground truth label
41 %
27.88%
• Good initialisation can lead to better solution
Propose a SIFT-flow based better initialisation method
15
SIFT-flow based correspondence
Given a test image, we first retrieve a set of nearest
neighbours from training set using GIST features
Test image
Nearest neighbours retrieved from training set
16
SIFT-flow based correspondence
K-nearest neighbours warped to the test image
23.31
13.31
14.31
18.38
22
22
Test image
22
30.87
27.2
Warped nearest neighbours and corresponding flows
17
SIFT-flow based correspondence
Pick the best nearest neighbour based on the flow value
Test image
Nearest neighbour
Warped image
Flow: 13.31
18
Label transfer
• Warp the ground truth according to correspondence
• Transfer labels from top 1 using flow
Ground truth of test image
Ground truth of
the best nearest
neighbour
Flow
19
Warped ground truth according to flow
SIFT-flow based initialisation
Rescore the unary potential
Test image
Ground truth image
Qualitative improvement
in accuracy after using
rescored unary potential
Without rescoring
After rescoring
20
SIFT-flow based initialisation
Initialise mean-field solution
Test image
Ground truth image
Qualitative improvement
in accuracy after
Without initialisation
initialisation of mean-field
With initialisation
21
Gaussian pairwise weights
Mixture of Gaussians
bilateral
spatial
22
Gaussian pairwise weights
Mixture of Gaussians
• Zero mean
bilateral
spatial
23
Gaussian pairwise weights
Mixture of Gaussians
• Zero mean
bilateral
• Same Gaussian mixture
model for every label pair
spatial
24
Gaussian pairwise weights
Mixture of Gaussians
• Zero mean
bilateral
• Same Gaussian mixture
model for every label pair
• Arbitrary standard
deviation
spatial
25
Our approach
Incorporate a general Gaussian mixture model
26
Gaussian pairwise weights
• Learn arbitrary mean
• Learn standard deviation
27
Gaussian pairwise weights
• Learn arbitrary mean
• Learn standard deviation
• Learn mixing coefficients
28
Gaussian pairwise weights
• Learn arbitrary mean
• Learn standard deviation
• Learn mixing coefficients
• Different Gaussian mixture
for different label pairs
29
Learning mixture model
Propose piecewise learning framework
30
Learning mixture model
• First learn the parameters of unary potential
31
Learning mixture model
• First learn the parameters of unary potential
• Learn the label compatibility function
32
Learning mixture model
• First learn the parameters of unary potential
• Learn the label compatibility function
• Set the Gaussian model following Krahenbuhl et.al
33
Learning mixture model
• First learn the parameters of unary potential
• Learn the label compatibility function
• Set the Gaussian model following Krahenbuhl et.al
• Learn the parameters of the Gaussian mixture
34
Learning mixture model
• First learn the parameters of unary potential
• Learn the label compatibility function
• Set the Gaussian model following Krahenbuhl et.al
• Learn the parameters of the Gaussian mixture
• Lambda is set through cross validation
35
Our model
• Generative training
• Maximise joint likelihood of pair of labels and
features:
: latent variable: number of mixture components
36
Learning mixture model
• Maximize the log-likelihood function
• Expectation maximization based method
Zero-mean Gaussian
Our learnt mixture model
37
Inference with mixture model
• Involves evaluating M extra Gaussian terms:
• Perform blurring on mean-shifted points
• Increases time complexity
38
Experiments on PascalVOC-10
Qualitative results of SIFT-flow method
Image
Warped nearest
ground truth
image
Output without
SIFT-flow
Output with
SIFT-flow
39
Experiments on PascalVOC-10
Quantitative results PascalVOC-10 segmentation dataset
Algorithm
Time(s)
Overall(%-corr)
Av. Recall
Av. U/I
Alpha-exp
3.0
79.52
36.08
27.88
AHCRF+Cooc
36
81.43
38.01
30.9
Dense CRF
0.67
71.63
34.53
28.4
Ours1(U+P+GM)
26.7
80.23
36.41
28.73
Ours2 (U+P+I)
0.90
79.65
41.84
30.95
Ours3 (U+P+I+GM)
26.7
78.96
44.05
31.48
• Our model with unary and pairwise terms achieves better accuracy
than other complex models
• Generally achieves very high efficiency compared to other methods 40
Experiments on PascalVOC-10
Qualitative results on PascalVOC-10 segmentation dataset
Image
Alpha-expansion
Dense CRF
Ours
• Able to recover missing object parts
41
Experiments on Camvid
Quantitative results on Camvid dataset
Algorithm
Time(s)
Overall(%-corr)
Av. Recall
Av. U/I
Alpha-exp
0.96
78.84
58.64
43.89
APST(U+P+H)
1.6
85.18
60.06
50.62
denseCRF
0.2
79.96
59.29
45.18
Ours (U+P+I)
0.35
85.31
59.75
50.56
• Our model with unary and pairwise terms achieve better accuracy
than other complex models
• Generally achieve very high efficiency compared to other methods
42
Experiments on Camvid
Qualitative results on Camvid dataset
Image
Alpha-expansion
Ours
• Able to recover missing object parts
43
Conclusion
• Filter-based mean-field inference promises high
efficiency and accuracy
• Proposed methods to robustify basic mean-field method
• SIFT-flow based method for better initialisation
• EM based algorithm for learning general Gaussian
mixture model
• More complex higher order models can be incorporated
into pairwise model
44
45
Thank You 
46
Q distribution
47
Learning mixture model
• For every
label pair:
• Maximize the log-likelihood function
48
Learning mixture model
• For every
label pair:
• Maximize the log-likelihood function
• Expectation maximization based method
49