Transcript Cursive: A novel interaction technique for controlling expressive

What your avatar
can reveal about
your handwriting
Avatar gesture from
pen gestures
Francesca A. Barrientos
Computer Science Division
23 May 2002

Ph.D. Dissertation Seminar
Interacting in avatar worlds
A wedding
Laurel’sMay
herb8,farm
1996in(Damer
AlphaWorld
‘98) (Damer ‘98)
23 May 2002
Ph.D. Dissertation seminar
2
Avatar nonverbal
communication
• In physical world,
language embedded in
matrix of sounds and
visuals
• Avatar, as a virtual body,
can send nonverbal
communication
23 May 2002
Ph.D. Dissertation seminar
3
Problem statement
We want an interaction technique for
controlling avatar gesture
– Controls expressive movement
– Seamless with verbal communication
23 May 2002
Ph.D. Dissertation seminar
4
Avatar gesture from pen gesture
• Body gesture has symbolic and qualitative aspects
• Pen gesture carries symbolic and continuous data
• Pen gesture simultaneously selects avatar gesture and
modulates multiple expressive qualities
23 May 2002
Ph.D. Dissertation seminar
5
Control using handwritten letter
• Writing pen
gesture triggers
animation
• Body gesture is
“sweep” to side
• Symbol is letter ‘s’ l
• Quality being
varied is size
23 May 2002
Ph.D. Dissertation seminar
6
Overview
• Background on nonverbal
communication
– Why it’s important
– Why previous control techniques are
inadequate
• Description of interaction technique
• Description of implementation
• Conclusions
23 May 2002
Ph.D. Dissertation seminar
7
Kinds of nonverbal cues
•
•
•
•
•
Vocalics
Appearance
Proxemics & Haptics
Affective display
Kinesics
–
–
–
–
23 May 2002
Facial expression
Gaze
Posture
Gesture
Ph.D. Dissertation seminar
8
Gesture types
•
•
•
•
•
•
Deictic
Iconic
Metaphoric
Beat
Adaptors
Regulators
23 May 2002
Ph.D. Dissertation seminar
9
Importance of gesture
• Encodes ideas shared with speech
– Clarifies meaning when speech is ambiguous
– Useful cue when outside noise interferes with
speech
• Aids utterance generation
• Smooths over social intercourse
• Communicates mood/emotion
23 May 2002
Ph.D. Dissertation seminar
10
Interaction with speech
• Coverbal gesture shares
meaning units with spoken
language
• Language-like gestures fill in for
a word or phrase in a sentence
• Emblems have standard forms
and well understood within a
group
23 May 2002
Ph.D. Dissertation seminar
11
Observed virtual nonverbal
displays
• Presence
• Appearance
• Proxemics
– Conversation
group formation
– Personal space
WorldsAway from Fujitsu
Avabar by Zuidema (Damer)
23 May 2002
Ph.D. Dissertation seminar
12
Designed behaviors
• Static expressions
– Facial
– Posture
– Gesture
• Animated motions
– Gesture
– Entertaining dances
– Custom animations
23 May 2002
WorldsAway from Fujitsu
Ph.D. Dissertation seminar
13
Related work
• Commercial worlds
– Blend of text virtual communities and computer graphics
– Worlds Chat – first 3D world in 1995
• Multi-user virtual environment research
–
–
–
–
Vlnet (Guye-Vuillème et al ’98)
ComicChat (Kurlander+Skelly+Salesin ‘96)
Autonomous avatars (Vilhjálmsson+Cassell’98, Cassell et al ‘94)
Acting in virtual reality (Slater et al ‘00)
• Synthetic characters
– Improv (Perlin+Goldberg ’96)
– Alive (Blumberg+Galyean ’95, Maes et al ‘97)
– Jack (Badler ’97)
• Computer mediated conversation visualization
– Collaboration-at-a-glance (Donath ’95)
– Chat Circles (Viegas+Donath ’99)
23 May 2002
Ph.D. Dissertation seminar
14
Vlnet gesture/mimics panel
(Guye-Vuillème et al)
• Button for each expression or gesture
• Modulate speed with slider
23 May 2002
Ph.D. Dissertation seminar
15
ComicChat
• Text inference
– Emoticons
– Sentence
Structure
– Keywords
• Emotion wheel
(Kurlander et al)
23 May 2002
Ph.D. Dissertation seminar
16
BodyChat avatar agents
• Automatic conversation
regulation behaviors
– Salutations
– Envelope feedback
– Facial expression
• User specifies high-level
intentions
• Avatar expressions driven
by chat text
• Avatar software
manages gestures and
gaze behavior
(Vilhjálmsson+ Cassell)
23 May 2002
Ph.D. Dissertation seminar
17
Limitations of current techniques
• Mainly speech independent nonverbal displays
– Emotional facial display and posture
– Emblematic gesture
– Not general
• Cumbersome interface
– Graphical interface requires hunt and select
– Cannot scale
• Lack control over multiple expressive features of
movement
23 May 2002
Ph.D. Dissertation seminar
18
Why pen gestures
• Natural
– People doodle while
talking and listening
• Expressive
– Reflects emotional state
– Very personal
– Can be intentionally
manipulated
• Dual nature
–
–
–
–
Analog and digital
Symbolic and qualitative
Discrete and continuous
Information and emotion
23 May 2002
Ph.D. Dissertation seminar
19
Continuous interaction
Computers fragment our thinking by
substituting discrete events for
continuous actions.
-Malcolm McCullough
(Abstracting Craft, p. 53)
[Gestures] can...enhance the
experience of agency through
kinesthetic involvement and the
feeling of directness.
-Brenda Laurel
(Computers as Theatre, p. 158)
23 May 2002
Ph.D. Dissertation seminar
20
Interaction technique
Library
Generated
offline
Input
User writes
letter in GUI
Generation
Selection
and synthesis
Animation
Gesture
performed
wsrlibrary
23 May 2002
GUI
Gesture
generator
Ph.D. Dissertation seminar
Avatar
animator
21
Interaction schematic
23 May 2002
Ph.D. Dissertation seminar
22
Design issues
• Pen gesture input set
– Using letters of the alphabet
• Mapping from pen gesture features to
avatar movement parameters
– Handwriting features to extract
– Gesture movement parameterization
• Avatar gesture animation synthesis
method
23 May 2002
Ph.D. Dissertation seminar
23
Handwriting features
• Desired properties
– Controllable
– Computable
• Most important feature types (according
to handwriting analysis)
– Size
– Speed
– Pressure
23 May 2002
Ph.D. Dissertation seminar
24
Motion parameterization
• Formal systems
– Shape-Effort (Labanotation)
– Formal sign language systems
• Physical
– Size
– Speed
– Sustain
• Emotional
– Emphatic
– Listless
– Tentative
23 May 2002
Ph.D. Dissertation seminar
25
Physical mapping
Handwriting
feature
Size
Speed
23 May 2002
Movement
parameter
Size
Duration
Ph.D. Dissertation seminar
26
Gesture synthesis
• Input: gesture and movement parameter
values
• Multi-linear interpolation from set of
sample gestures
– Each avatar gesture comprises a set of
sample motions
– Each motion sample has different expression
– Samples are annotated with its movement
parameter values
23 May 2002
Ph.D. Dissertation seminar
27
Modulation through multilinear
interpolation
Gβ
• Joint trajectory I
– Rotation angles over time
• Gesture type β
Speed
u
– Semantic category
– Set of 2n prototypes - G
– n style parameters
– Prototypes represent
extremal trajectories
k
• Gesture instance Iβ(u,v )
Iβ(0,1)
Iβ(1,1)
β
I
(l,k)
I (0,k)
Iβ(0,0)
Iβ(1,k)
Iβ(1,0)
β
– Vector of joint trajectories
– Multilinear interpolation on
type produces instance
Size v
l
23 May 2002
Ph.D. Dissertation seminar
28
Interpolating speed
• Interpolation along curves of
different durations
• Time dilation step
– Determine duration for
interpolated curve
– Choose sample rate on
interpolated curve
– Compress slower curve sample at proportionately
slower rate
– Sample faster trajectory at
proportionately faster rate
23 May 2002
Angle θ
Ph.D. Dissertation seminar
t
29
Framework for applying pen
gesture to avatar gestures
• Subproblems
– Expression map design
– Gesture synthesis technique
• Can explore other mappings and
synthesis techniques
– Labanotation parameterization of movement
– Emotion parameterization
• Customize toward particular domain
23 May 2002
Ph.D. Dissertation seminar
30
Implementation
Cursive
– Application for interactively controlling VRML
avatars over the internet
– Use to test animations
• Architecture permits independence from
specific avatar world software
– Viewer can see animations without installing
new software
• Facilitate testing of shared object
behaviors in virtual worlds
23 May 2002
Ph.D. Dissertation seminar
31
Avatar gesture samples
• Built motion capture system
– Magnetic position/orientation sensors
(Flock of Birds from Ascension Corp.)
• Recorded gestures
– Vary size and speed
• Can create very personalized gesture
23 May 2002
Ph.D. Dissertation seminar
32
Cursive screenshot
Cursive window
VRML browser
VRML avatar
23 May 2002
Ph.D. Dissertation seminar
33
Architecture
23 May 2002
Ph.D. Dissertation seminar
34
Communication
Cursive communicates with any copies of user’s avatar
•
•
•
•
•
Driver logs into Vworld
server
Other viewers receive
notification
Other viewers request and
download avatar copy
Avatar opens socket
connection to Cursive
Cursive sends gesture
commands via socket
viewer host
driver host
Web server
open
socket
sends
commands
login
request
avatar
Download
avatar
notification
VWorld server
23 May 2002
Ph.D. Dissertation seminar
35
Evaluation
• Very simple to control one parameter at
a time
• More complicated to control size and
speed simultaneously
– Effective usage requires practice
– Require further investigation into mapping
handwriting features to movement
parameters
• Viable technique for controlling avatar
gesture
23 May 2002
Ph.D. Dissertation seminar
36
Expert use
…the art of finding and executing an effective
gesture is learned through the more indirect
means of observation, experimentation,
performance, and evaluation, and it is a skill
that continues to grow over time.
- Brenda Laurel
(Computers as Theatre, p. 155)
23 May 2002
Ph.D. Dissertation seminar
37
Summary
• Want to control richly expressive,
spontaneous gesture in avatar worlds
• Solution is an interaction technique
employing pen gesture input
• Cursive: an implementation of this
interaction technique
23 May 2002
Ph.D. Dissertation seminar
38
Contributions
• Novel interaction technique that augments the potential
repertoire of avatar nonverbal communication
– Possible to control spontaneous gesture
– Control over expressive characteristics of gestural movement
• Framework for applying pen gesture to avatar gesture
– Mapping handwriting features to movement parameters
– Synthesis of expressive gesture
• The algorithms and methods used to implement the
technique
• Cursive, a working application that applies the interaction
technique
23 May 2002
Ph.D. Dissertation seminar
39
Future work
• Explore other mappings
– emotional expression in handwriting and gesture
– Shape-Effort parameterization of movement
• Explore other gesture synthesis methods
– Reduce cost of obtaining gesture motion samples
• Develop a framework for determining avatar
gesture vocabulary
– How many gestures
– What types of gestures
23 May 2002
Ph.D. Dissertation seminar
40
Conclusion
• Transmitting bodily nonverbal communication
through the internet is an exciting idea.
• Think of the computer as a medium for personal
expression
– Continuous/rich interaction
– Playful behavior
– Sense of engagement
• Handwriting has a place in affective computing
23 May 2002
Ph.D. Dissertation seminar
41
Acknowledgements
A few of the people who helped
with my research and with this
talk.
My dissertation committee: Prof.
John Canny, Prof. James
Landay, Prof. John McWhorter.
My gesture model: Erin Dare.
My dissertation writing group:
Blanca Gordo, Jeffrey Ow, Dr.
Ellen Sacco-Fernandez, Lynne
Horiuchi.
Other talk critics: Miriam Walker,
Scott Klemmer, Dan Glaser,
Jeremy Risner, James Lin, Jason
Hong, Dr. Eric Paulos.
23 May 2002
Ph.D. Dissertation seminar
42
Partial bibliography
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
Damer, B., Avatars! Exploring and Building Virtual Worlds on the Internet. Berkeley, CA: Peachpit
Press, 1998.
Guye-Vuillème, A., T. Capin, I. Pandzic, N. Magnenat-Thalmann, and D. Thalmann, "Non-Verbal
Communication Interface for Collaborative Virtual Environments," in Proc. CVE 98, June 1998,
Manchester, 1998.
Vilhjálmsson, H. H. and J. Cassell, "BodyChat: Autonomous Communicative Behaviors in
Avatars," in Proceedings of the Second International Conference on Autonomous Agents, May
9-13, 1998, Minneapolis: ACM, 1998, pp. 269-276.
Cassell, J., C. Pelachaud, N. Badler, M. Steedman, B. Achorn, T. Becket, B. Douville, S. Prevost,
and M. Stone, "Animated Conversation: Rule-Based Generation of Facial Expression, Gesture
and Spoken Intonation for Multiple Conversational Agents," in Proceedings of SIGGRAPH '94.,
1994.
Slater, M., J. Howell, A. Steed, D. P. Pertaub, M. Gaurau, and S. Springel, "Acting in Virtual
Reality," in Collaborative Virtual Environments 2000: ACM, 2000, pp. 103-110.
Perlin, K. and A. Goldberg, "Improv: A System for Scripting Interactive Actors in Virtual Worlds," in
Proc. Siggraph 96, H. Rushmeier, Ed., New York: ACM Press, 1996, pp. 205-216.
Blumberg, B. M. and T. A. Galyean, "Multi-level direction of autonomous creatures for real-time
virtual environments," presented at SIGGRAPH, Los Angeles, CA, 1995.
Maes, P., T. Darrell, B. Blumberg, and A. Pentland, "The ALIVE system: wireless, full-body
interaction with autonomous agents," Multimedia Systems, 5, no. 2, 1997, pp. 105-12.
Badler, N., "Virtual Humans for Animation, Ergonomics, and Simulation," in IEEE Workshop on
Non-Rigid and Articulated Motion, June 1997, Puerto Rico, 1997.
Donath, J. S., "The illustrated conversation," Multimedia Tools and Applications, 1, no. 1, 1995,
pp. 79-88.
Viegas, F. B. and J. S. Donath, "Chat Circles," in CHI 99: ACM, 1999, pp. 6-19.
Laurel, B., Computers as Theatre. Reading, Massachusetts: Addison-Wesley, 1993.
McCullough, M., Abstracting craft : the practiced digital hand. Cambridge, Massachusetts: The
MIT Press, 1996.
23 May 2002
Ph.D. Dissertation seminar
43