Transcript Micro expression Detection using Strain Patterns

- Sridhar Godavarthy
 Expressions
 Microexpressions
 FACS
 Evolutionary Psychology
 Proposed method
 Outline
 Video
 Face Detection, Alignment and Splitting
 Motion Field
 Optic Flow
 Optic Strain




Datasets
Current Work
Future Work
Results




Initial
WACV
Optic Flow
Thresholding
 Expressions
 Microexpressions
 FACS
 Evolutionary Psychology
 Primary means of social emotion conveyance
 Non verbal
 Conveys emotional state
 Voluntary or involuntary
 Minuscule differences in muscle movement
• 6 primary expressions(Not all clearly
distinguishable)
•Can you identify the ones in this
picture?
 What are microexpressions?
 Subtle movements of the human face
 Usually caused when attempting to mask a macro-




expression
Quick enough to be completed within the blink of an
eye
Last from 1/25th to 1/5th of a second
Restricted to certain muscles(regions) of the face
Almost impossible to fake
 Examples:
 Raising an eyebrow
 Shrugging of shoulders
 Pout of lips
 Fast blinking of eye
 Non Examples:
 Talking
 Smiling
 Laughing
 Anger
 For compartmentalization and categorization of




human expressions
32 Action Units (with muscle involvement)
14 Action Descriptors ( ! -do-)
Can be used to code any possible expression
Widely used in pain recognition and facial expression
simulations
 1000 page manual.
 Devised and written by one man.
 Requires extensive training.
 Some success in automating [ Bartlett et. al 1999] .
 Study of everything we discussed until now
 The child of ONE man - Paul Ekman.
 Over thirty years of research
 One of the world’s leading experts on lying.
 About 2 dozen books and innumerable articles
 Developed FACS
 Scientific Advisor to “Lie to Me”
 Co creator of Microexpression Training Tool
(METTx)

Expressions

Microexpressions

FACS

Evolutionary Psychology
 Proposed method
Outline
 Video
 Face Detection, Alignment and Splitting
 Motion Field

 Optic Flow
 Optic Strain

Datasets

Current Work

Future Work

Results

Initial

WACV

Optic Flow

Thresholding
Face Detection & Translation
Eye Detection/Alignment
Split into ROI
Motion Field Estimation
(Optical Flow)
Optical Strain
Thresholding for period, strain
Combine and count ROI
Face Detection & Translation
Eye Detection/Alignment
Split into ROI
Motion Field Estimation
(Optical Flow)
Optical Strain
Thresholding for period, strain
Combine and count ROI
 Video is a collection of individual images also known
as frames
 In reality: spatial and temporal compression using
properties of the scene.
 Any video can be decoded into a series of frames.
 24/30 frames per second of video.
 The science of encoding a
video in a manner such
that
 Minimum number of bits
are used
 Motion compensated
prediction can be
performed from a
previous frame.
Face Detection & Translation
Eye Detection/Alignment
Split into ROI
Motion Field Estimation
(Optical Flow)
Optical Strain
Thresholding for period, strain
Combine and count ROI
Face Detection & Translation
Eye Detection/Alignment
Split into ROI
Motion Field Estimation
(Optical Flow)
Optical Strain
Thresholding for period, strain
Combine and count ROI
 2D vector field of velocities of the image points induced by
the relative motion.
• Feature-based methods
 Extract visual features (corners, textured areas) and
track them over multiple frames
 Sparse motion fields, but more robust tracking
 Suitable when image motion is large (10s of pixels)
• Direct methods
 Directly recover image motion at each pixel from spatio-
temporal image brightness variations
 Dense motion fields, but sensitive to appearance
variations
 Suitable for video and when image motion is small
• Def: Optical Flow is the apparent motion of brightness
patterns in the image
• Ideally, same as the motion field
• Have to be careful: apparent motion can be caused by
lighting changes without any actual motion
 Brightness constancy
 Under most circumstance, the
apparent brightness of moving
objects remain constant
• Key assumptions
 Optical Flow Equation
 Relation of the apparent motion
dE ( x, y, t )
0
dt
E u  E v  Et  0
• the
Brightness
projection
x of they
with
spatial andconstancy:
temporal
sameofpoint
looksbrightness
the same in every frame
derivatives
the image
• Small motion: points do not move very far
 Aperture
problem
• Spatial
coherence: points move like their
 Only the component of the motion
neighbors
field in the direction of the spatial
image gradient can be determined
 The component in the direction
perpendicular to the spatial gradient is
not constrained by the optical flow
equation
 Constant Flow Method
Assumption: the motion field is well approximated by a
constant vector within any small region of the image plane
 Solution: Least square of two variables (u,v) from NxN
Equations – NxN (=5x5) planar patch
 Condition: ATA is NOT singular (null or parallel gradients)

 Weighted Least Square Method
Assumption: the motion field is approximated by a constant
vector within any small region, and the error made by the
approximation increases with the distance from the center
where optical flow is to be computed
 Solution: Weighted least square of two variables (u,v) from

NxN Equations – NxN patch
 Affine Flow Method
Assumption: the motion field is well approximated by a affine
parametric model uT = ApT+b (a plane patch with arbitrary
orientation)
 Solution: Least square of 6 variables (A,b) from NxN

Equations – NxN planar patch
Face Detection & Translation
Eye Detection/Alignment
Split into ROI
Motion Field Estimation
(Optical Flow)
Optical Strain
Thresholding for period, strain
Combine and count ROI
 Different materials have different elasticity
 Elasticity can be modeled
stress
Elasticity   
strain
Known
Calculate
 What is Facial Strain?
 Strain on soft tissue when expressions are made.
 Anatomical method
 Uses a pair of frames to measure deformation
 Finite Element Method
 Forward modeling when Dirichlet condition is satisfied
 Good at handling irregular shapes
 Computationally expensive
 This method is an approximation to the solution
 Finite Difference Method
 Strain, a tensor, can be expressed derivatives of the
displacement vector
 This can be approximated by a Finite Difference Method.
 Very efficient when carried out on a regular grid.
 This method is an approximation to the differential equation
 Finite Difference Method
 Compute spatial derivatives from discrete points.
 Forward Difference Method
 Central Difference Method
 Richardson extrapolation
Optical Flow
Optical Strain

Expressions

Microexpressions

FACS

Evolutionary Psychology

Proposed method

Outline

Video

Face Detection, Alignment and Splitting

Motion Field


Optic Flow
Optic Strain
 Datasets
 Current Work
 Future Work

Results

Initial

WACV

Optic Flow

Thresholding
Name
#videos
~ #Sequences
Political
10
USF
(30)
8-9
~250
Found Videos
4(+10)
1-2
~20
ASL
1
TBGt
-
7-9
Total
80
 Eye detection/face alignment to accommodate head





movement/rotation.
Automatic thresholding
Dataset collection
Testing on “interesting” videos
Trying a different Optical Flow( Black and Anandan)
Run expression detection to remove Macro expressions
first.
 Micro expression recognition

Expressions

Microexpressions

FACS

Evolutionary Psychology

Proposed method

Outline

Video

Face Detection, Alignment and Splitting

Motion Field


Optic Flow
Optic Strain

Datasets

Current Work

Future Work
 Results
 Initial
 WACV
 Optic Flow
 Thresholding
Images resized non-uniformly for presentation
~22 frames
~5 frames
 P. Ekman and W. Friesen. Facial Action Coding
System: A Technique for the Measurement of Facial
Movement. Consulting Psychologists Press, Palo Alto,
1978
 Malcolm Gladwell,” Blink: The Power of Thinking Without
Thinking”, Back Bay Books (April 3, 2007)
 G. Donato, M. Bartlett, J. Hager, P. Ekman, and T.
Sejnowski.
 Classifying facial actions. IEEE Transactions
 on Pattern Analysis and Machine Intelligence,
 21(10):974–989, 1999
Sridhar Godavarthy
Dept. Of Computer Science and Engineering
University of South Florida
[email protected]