Transcript Slide 1
The Display of Emotion
Brent Lance
CS 543 Lecture
7/7/2015 12:27 AM
Copyright 2005, USC/ISI. All rights reserved.
1
Overview
Motivation
Psychological View of Nonverbal Behavior
Encoding Vs. Decoding
Nonverbal Behavior
Expression of Emotion through Nonverbal
Behavior
Artistic View of Emotional Expression
Realizing Emotional Expression
7/7/2015 12:27 AM
Copyright 2005, USC/ISI. All rights reserved.
2
Motivation
What is interesting about the display of
emotion?
The development of Virtual characters
Believable
Capable of open-ended interaction
Engaging
For this, a character must be able to signal its
internal state through its external behavior
7/7/2015 12:27 AM
Copyright 2005, USC/ISI. All rights reserved.
3
Virtual Characters
Keyframe Animation
Classic technique from 2D cell animation
Skilled animator develops key frames
Interpolation between them creates
movement
Slow, iterative process
Produces most realistic behavior
Takes very long time to develop
Animations are not reusable
7/7/2015 12:27 AM
Copyright 2005, USC/ISI. All rights reserved.
4
KeyFrame Animation
7/7/2015 12:27 AM
Copyright 2005, USC/ISI. All rights reserved.
5
Motion Capture
Motion Capture Animation
Cameras record movement of sensors placed on an
actor
Software reconstructs movement and approximation
of character’s anatomy
Allows for quicker generation of movement
Requires expensive hardware and analysis tools
Captured sequences not reusable
Can be difficult to retarget capture motion to 3D
model being animated
7/7/2015 12:27 AM
Copyright 2005, USC/ISI. All rights reserved.
6
Motion Capture
7/7/2015 12:27 AM
Copyright 2005, USC/ISI. All rights reserved.
7
Procedural Animation
Generation of movement without human
oversight
Virtual Agents
Video Game Characters
Can use libraries of keyframe or motion
capture animation
Allow for dynamic or interactive environments
Do not have the quality of previous two
methods
7/7/2015 12:27 AM
Copyright 2005, USC/ISI. All rights reserved.
8
Procedural Animation
7/7/2015 12:27 AM
Copyright 2005, USC/ISI. All rights reserved.
9
Motivation - Revisited
Hopefully, knowledge of how emotion is
expressed through nonverbal behavior can
improve interactive virtual agents to the
point where they are as engaging as
traditionally animated characters
7/7/2015 12:27 AM
Copyright 2005, USC/ISI. All rights reserved.
10
Overview
Motivation
Psychological View of Nonverbal Behavior
Encoding Vs. Decoding
Nonverbal Behavior
Expression of Emotion through Nonverbal
Behavior
Artistic View of Emotional Expression
Realizing Emotional Expression
7/7/2015 12:27 AM
Copyright 2005, USC/ISI. All rights reserved.
11
Nonverbal Behavior
Encoding
The display of information through external
behavior
Decoding
The comprehension of information through
observation of external behavior
7/7/2015 12:27 AM
Copyright 2005, USC/ISI. All rights reserved.
12
Encoding
Types of encoding
[Ekman & Friesen, 1969] define three types
of coding
Arbitrary
No relation between coding and meaning
Iconic
Coding is representation of meaning
Intrinsic
Coding is meaning
7/7/2015 12:27 AM
Copyright 2005, USC/ISI. All rights reserved.
13
Complications
Encoding may not be deliberate
Encoding may be used to mask actual
information
Encoding may be idiosyncratic
Men and women encode & decode
differently
Different cultures have different coding
rules
7/7/2015 12:27 AM
Copyright 2005, USC/ISI. All rights reserved.
14
Realistic vs. Believable
Decoding research is easier than Encoding
research
Generating Believable expressive behavior
is easier than generating Realistic
expressive behavior
“Realistic” -> behavior as a human would
have performed it
“Believable” -> behavior understandable by
an observer
7/7/2015 12:27 AM
Copyright 2005, USC/ISI. All rights reserved.
15
Overview
Motivation
Psychological View of Nonverbal Behavior
Encoding Vs. Decoding
Nonverbal Behavior
Expression of Emotion through Nonverbal
Behavior
Artistic View of Emotional Expression
Realizing Emotional Expression
7/7/2015 12:27 AM
Copyright 2005, USC/ISI. All rights reserved.
16
Nonverbal Communication
Technically, communication without words
Extremely broad field [Knapp & Hall, 1997]
Includes:
Environment
Appearance
Dress, grooming
Height, weight
Race & gender
None of these really express emotion
However, they may affect appraisal ratings
7/7/2015 12:27 AM
Copyright 2005, USC/ISI. All rights reserved.
17
Overview
Motivation
Psychological View of Nonverbal Behavior
Encoding Vs. Decoding
Nonverbal Behavior
Expression of Emotion through
Nonverbal Behavior
Artistic View of Emotional Expression
Realizing Emotional Expression
7/7/2015 12:27 AM
Copyright 2005, USC/ISI. All rights reserved.
18
Expression of Emotion
What nonverbal behaviors display emotion?
Torso Posture
Head Position
Facial Expression
Hand Gestures
Gaze Behavior
Movement
Expression of Emotion in Man and Animals
[Darwin, 1872].
7/7/2015 12:27 AM
Copyright 2005, USC/ISI. All rights reserved.
19
Posture - Demonstration
7/7/2015 12:27 AM
Copyright 2005, USC/ISI. All rights reserved.
20
Posture - Demonstration
7/7/2015 12:27 AM
Copyright 2005, USC/ISI. All rights reserved.
21
Posture - Demonstration
7/7/2015 12:27 AM
Copyright 2005, USC/ISI. All rights reserved.
22
Torso Posture
Torso Posture is affected less by emphasis
and other speech-related behaviors [Bull
& Connelly, 1985]
Positive and Negative emotions are both
easily distinguished both from static body
postures [Coulson, 2004], [Schouwstra &
Hoogstraten, 1995], and from dynamic
trunk movement [de Meijer, 1989].
7/7/2015 12:27 AM
Copyright 2005, USC/ISI. All rights reserved.
23
Expression of Emotion
What nonverbal behaviors display
emotion?
Torso Posture
Head Position
Facial Expression
Hand Gestures
Gaze Behavior
Movement
7/7/2015 12:27 AM
Copyright 2005, USC/ISI. All rights reserved.
24
Head Position
7/7/2015 12:27 AM
Copyright 2005, USC/ISI. All rights reserved.
25
Head Position
Movement of the head is tightly connected
to speech [Hadar et al., 1985], [McClave,
1999].
Static posture of the head is still useful for
the display of emotions
Head which is tilted upwards is viewed as
more dominant, and displaying a more
positive than one tilted downwards [Kappas et
al., 1994], [Mignault & Chaudhuri, 2003].
7/7/2015 12:27 AM
Copyright 2005, USC/ISI. All rights reserved.
26
Expression of Emotion
What nonverbal behaviors display
emotion?
Torso Posture
Head Position
Facial Expression
Hand Gestures
Gaze Behavior
Movement
7/7/2015 12:27 AM
Copyright 2005, USC/ISI. All rights reserved.
27
Facial Expression
Facial expression is one of the stronger
channels for emotional expression
Paul Ekman
Performed many of early experiments on
facial expression
Found that facial expression was highly cross
cultural
Developed the Facial Action Coding System
(FACS)
7/7/2015 12:27 AM
Copyright 2005, USC/ISI. All rights reserved.
28
Happiness
7/7/2015 12:27 AM
Copyright 2005, USC/ISI. All rights reserved.
29
Anger
7/7/2015 12:27 AM
Copyright 2005, USC/ISI. All rights reserved.
30
Sadness
7/7/2015 12:27 AM
Copyright 2005, USC/ISI. All rights reserved.
31
Fear
7/7/2015 12:27 AM
Copyright 2005, USC/ISI. All rights reserved.
32
Disgust
7/7/2015 12:27 AM
Copyright 2005, USC/ISI. All rights reserved.
33
Facial Expression
Other researchers have shown that
Children who are Deaf/Blind from birth share
many of the same facial expressions
As do very young infants
As well as monkeys
[Knapp & Hall, 1997]
7/7/2015 12:27 AM
Copyright 2005, USC/ISI. All rights reserved.
34
Expression of Emotion
What nonverbal behaviors display
emotion?
Torso Posture
Head Position
Facial Expression
Hand Gestures
Gaze Behavior
Movement
7/7/2015 12:27 AM
Copyright 2005, USC/ISI. All rights reserved.
35
Hand Gestures
Much more research has been done on
the association of hand gestures with the
communication of concepts
However, research has demonstrated that
level of arousal, and to a lesser extent the
valence of emotion can be determined
through arm movement [Pollick et al.,
2001]
7/7/2015 12:27 AM
Copyright 2005, USC/ISI. All rights reserved.
36
Expression of Emotion
What nonverbal behaviors display
emotion?
Torso Posture
Head Position
Facial Expression
Hand Gestures
Gaze Behavior
Movement
7/7/2015 12:27 AM
Copyright 2005, USC/ISI. All rights reserved.
37
Gaze Behavior
What do we mean by gaze behaviors?
Eye shape and eyebrow position are part of
facial expression
Length of gaze & mutual gaze
Gaze attraction vs. aversion
7/7/2015 12:27 AM
Copyright 2005, USC/ISI. All rights reserved.
38
Non-Emotional Gaze
Gaze is closely linked to speech-related
behaviors
Patterns of gaze for individuals speaking are
often different from patterns of gaze for
listeners [Kleinke, 1986].
Attention and Saliency also control eye
movement [Argyle & Cook, 1973].
7/7/2015 12:27 AM
Copyright 2005, USC/ISI. All rights reserved.
39
Gaze and Dominance
Gaze is very important in signaling
dominance [Exline, 1971]
However this relationship is not fully
understood
Seen in the way monkeys perform dominance
displays when looked at by experimenters
Highly dominant individuals look less while
listening, and more while speaking than do
individuals of low dominance [Knapp, 1997].
7/7/2015 12:27 AM
Copyright 2005, USC/ISI. All rights reserved.
40
Gaze and Arousal
Gaze is also closely related to
physiological arousal [Argyle & Cook,
1973]
Receiving gaze and engaging in mutual gaze
increases arousal
Gaze is used as a regulation for physiological
arousal
Increased blinks, and increased pupil dilation
are also strong signals of arousal
7/7/2015 12:27 AM
Copyright 2005, USC/ISI. All rights reserved.
41
Gaze and Preference
Gaze also has a drastic effect on liking
Individuals who gaze very little at people they
interact with are rated lower on scales of
preference, credibility, attractiveness, and are
less likely to be hired [Burgoon et al., 1985],
[Argyle & Cook, 1973].
Individuals are also more likely to look at
people that they like [Argyle & Cook,
1973]
7/7/2015 12:27 AM
Copyright 2005, USC/ISI. All rights reserved.
42
Expression of Emotion
What nonverbal behaviors display
emotion?
Torso Posture
Head Position
Facial Expression
Hand Gestures
Gaze Behavior
Movement
7/7/2015 12:27 AM
Copyright 2005, USC/ISI. All rights reserved.
43
Movement & Locomotion
Emotion can be recognized through how
people walk
But further, emotion can be recognized
through the movement of simple
geometric shapes [Rime et al., 1985]
7/7/2015 12:27 AM
Copyright 2005, USC/ISI. All rights reserved.
44
Movement
7/7/2015 12:27 AM
Copyright 2005, USC/ISI. All rights reserved.
45
Locomotion
7/7/2015 12:27 AM
Copyright 2005, USC/ISI. All rights reserved.
46
Synchrony
The interactions of all of these behaviors
is as important as any single behavior
If not, emotional signals lack coherency
The ability of users to recognize emotion
decreases
Believability and engagement decrease
7/7/2015 12:27 AM
Copyright 2005, USC/ISI. All rights reserved.
47
Conclusion – Nonverbal
While there are many different ways
emotion can be signaled through
nonverbal behavior
Only facial expression (and possibly prosody)
can clearly differentiate between individual
emotional categories
Much nonverbal behavior research done using
dimensional models of emotion that can be
applied to factor analyses of nonverbal
behavior results
7/7/2015 12:27 AM
Copyright 2005, USC/ISI. All rights reserved.
48
Overview
Motivation
Psychological View of Nonverbal Behavior
Encoding Vs. Decoding
Nonverbal Behavior
Expression of Emotion through Nonverbal
Behavior
Artistic View of Emotional Expression
Realizing Emotional Expression
7/7/2015 12:27 AM
Copyright 2005, USC/ISI. All rights reserved.
49
Art & Emotion
Looking at three artistic views of
emotional expression
Delsarte’s Structural Acting System
Laban Movement Analysis, a system derived
from choreography
Emotion in 2D Disney Animation
7/7/2015 12:27 AM
Copyright 2005, USC/ISI. All rights reserved.
50
Structural Acting System
Francois Delsarte (1811-1871)
French opera singer who damaged his voice
Turned to the expression of emotion through
acting
Compiled extremely extensive library of
movements, and the meanings of those
movements
However, this library has not been empirically
verified
7/7/2015 12:27 AM
Copyright 2005, USC/ISI. All rights reserved.
51
Delsarte
Recently, [Marsella et al., 2006] performed
an evaluation of some of Delsarte’s work
on hand gestures
“Delsarte’s cube”
Small imaginary cube directly in front of chest
Placement of hands on cube sends different
signals
i.e. hands placed on inside surface of near
face of cube to reveal possession
7/7/2015 12:27 AM
Copyright 2005, USC/ISI. All rights reserved.
52
Laban Movement Analysis
Rudolph Laban (1879-1958)
Hungarian Dancer and Choreographer
Developed Labanotation, a notation for
recording dance
Rendered mostly obsolete through widespread
distribution of video recording technology
Developed, in conjunction with several of his
students, Laban Movement Analysis, a
taxonomy for describing movement
7/7/2015 12:27 AM
Copyright 2005, USC/ISI. All rights reserved.
53
Laban Movement Analysis
Qualitative, hierarchical taxonomy
Effort
Space
Indirect/Direct
Time
Sustained/Sudden
Weight
Light/Strong
Flow
Free/Bound
Shape
7/7/2015 12:27 AM
Copyright 2005, USC/ISI. All rights reserved.
54
EMOTE
[Chi et al., 2000] used Laban parameters
Effort and Shape to animate emotion
Theory was that instead of specifying
emotional state, would instead specify LMA
parameters to animate a character
Extended by [Zhao & Badler, 2005] which
learned LMA parameters from motion capture
or video recording
7/7/2015 12:27 AM
Copyright 2005, USC/ISI. All rights reserved.
55
Disney Animation
Principles of Disney animation laid out in
[Thomas & Johnston, 1981], extended to
3D animation in [Lasseter, 1994].
7/7/2015 12:27 AM
Copyright 2005, USC/ISI. All rights reserved.
56
Keyframe Animation
Skilled animators
drew key frames
2nd Tier drew the “inbetweens”
Developed principles
such as “Squash &
Strech”,
“Anticipation”, and
“Follow Through”
7/7/2015 12:27 AM
Copyright 2005, USC/ISI. All rights reserved.
57
2-D Animation
Extremely slow process
Highly iterative
Highly intuitive
Very difficult to clearly explain
Requires great deal of training, and a certain
amount of talent
Results are excellent
7/7/2015 12:27 AM
Copyright 2005, USC/ISI. All rights reserved.
58
Analyzing 2D Animation
[Lance et al., 2004], and [Chafai et al.,
2006] were attempts to analyze 2D
animation for information on nonverbal
behavior for gaze and gesture,
respectively
Moderately successful
Extremely labor intensive
7/7/2015 12:27 AM
Copyright 2005, USC/ISI. All rights reserved.
59
Overview
Motivation
Psychological View of Nonverbal Behavior
Encoding Vs. Decoding
Nonverbal Behavior
Expression of Emotion through Nonverbal
Behavior
Artistic View of Emotional Expression
Realizing Emotional Expression
7/7/2015 12:27 AM
Copyright 2005, USC/ISI. All rights reserved.
60
Realizing Behavior
7/7/2015 12:27 AM
Copyright 2005, USC/ISI. All rights reserved.
61
Animation Curves
7/7/2015 12:27 AM
Copyright 2005, USC/ISI. All rights reserved.
62
Animation of Behavior
Qualitative descriptions of nonverbal
behavior need to be transformed into
quantitative changes in animation
Two Choices:
Directly manipulate the animation curves
yourself
Optimization
Develop and utilize an abstraction, such as
EMOTE or the Body Markup Language (BML)
7/7/2015 12:27 AM
Copyright 2005, USC/ISI. All rights reserved.
63
Conclusion
Many sources of information for the expression
of emotion through human behavior
Lots of them are qualitative
Or not experimentally verified
Or describe more subtle relationships than direct
emotional state -> behavior
Picture is still very incomplete
Still lots of good information for a starting point!
7/7/2015 12:27 AM
Copyright 2005, USC/ISI. All rights reserved.
64
References
Argyle, M., Cook, M. Gaze and Mutual Gaze. Cambridge University Press. 1976.
Bull, P., Connelly, G. Body Movement and Emphasis in Speech. Journal of Nonverbal Behavior,
9(3). Fall 1985.
Burgoon, J., Manusov, V., Mineo, P., Hale, J. Effects of Gaze on Hiring, Credibility, Attraction and
Relational Message Interpretation. Journal of Nonverbal Behavior. 9(3). Fall 1985.
Chafai, N., Pelachaud, C., Pele, D., Breton, G. Gesture Expressivity Modulations in an ECA
Application. Proceedings of the Intelligent Virtual Agents IVA 2006 Conference. 2006.
Coulson, M. Attributing Emotion to Static Body Postures: Recognition Accuracy, Confusions, and
Viewpoint Dependence. Journal of Nonverbal Behavior. 28(2). Summer 2004.
Chi, D., Costa, M., Zhao, L., Badler, N. The EMOTE Model for Effort and Shape. Proceedings of
the 27th Annual Conference on Computer Graphics and Interactive Techniques. ACM
Press/Addison-Wesley Publishing, New York, NY. 2000.
Darwin, C., The Expression of Emotions in Man and Animals. 1872.
Ekman P., Friesen, W.V., The Repertoire of Nonverbal Behavior: Categories, Origins, Usage, and
Coding. Semiotica. 1, 49-98. 1969.
Exline, R. Visual Interaction: The Glances of Power and Preference. In Weitz, S. (ed), Nonverbal
Communication: Readings with Commentary. Oxford University Press, 1974.
Hadar, U., Steiner, T. J., Rose, F. C. Head Movement During Listening Turns in Conversation.
Journal of Nonverbal Behavior, 9(4). Winter 1985.
Kappas, A., Hess, U., Barr, C., Kleck, R. Angle of Regard: The Effect of Vertical Viewing Angle on
the Perception of Facial Expressions. Journal of Nonverbal Behavior. 18(4). Winter 1994.
Kleinke, C. Gaze and Eye Contact: A Research Review. Psychological Bulletin. v. 100, n. 1.
1986.
7/7/2015 12:27 AM
Copyright 2005, USC/ISI. All rights reserved.
65
References (Cont.)
Knapp, M., Hall, J. Nonverbal Communication in Human Interaction. Fourth Edition. Harcourt
Brace Publishers. 1997.
Lance, B., Marsella, S., Koizumi, D. Towards Expressive Gaze Manner in Embodied Virtual
Agents. AAMAS Workshop on Empathic Agents. 2004.
Lasseter, J. Principles of Traditional Animation Applied to 3D Computer Animation. Computer
Graphics. v. 21, n 4. July 1987.
Marsella, S., Carnicke, S. M., Gratch, J., Okhmatovskaia, A., Rizzo, A. An Exploration of Delsarte’s
Structural Acting System. Proceedings of the Intelligent Virtual Agents IVA 2006 Conference.
2006.
McClave, E. Linguistic Functions of Head Movements in the Context of Speech. Journal of
Pragmatics. v. 32, 2000.
de Meijer, M. The Contribution of General Features of Body Movement to the Attribution of
Emotions. Journal of Nonverbal Behavior. 13(4). Winter 1989.
Mignault, A., Chaudhuri, A. The Many Faces of a Neutral Face: Head Tilt and Perception of
Dominance and Emotion. Journal of Nonverbal Behavior. 27(2). Summer 2003.
Pollick, F., Paterson, H., Bruderlin, A., Sanford, A. Perceiving Affect from Arm Movement.
Cognition. i 82, B51-B61. 2001.
Rime, B., Boulanger, B., Laubin, P., Richir, M., Stroobantst, K. The Perception of Interpersonal
Emotions Originated by Patterns of Movement. Motivation and Emotion. v. 9, n. 3. 1985.
Schouwstra, S., Hoogstraten, J. Head Position and Spinal Position as Determinants of Perceived
Emotional State. Perceptual and Motor Skills. v. 81, 1995.
Thomas, F., Johnston, O. The Illusion of Life: Disney Animation. Walt Disney Productions. 1981.
Zhao, L., Badler, N. Acquiring and Validating Motion Qualities from Live Limb Gestures.
Graphical Models. v. 67 i. 1. January, 2005.
7/7/2015 12:27 AM
Copyright 2005, USC/ISI. All rights reserved.
66