Transcript CVE 2002 - Computer Science Division

Cursive:
Controlling Expressive
Avatar Gesture using
Pen Gesture
Francesca A. Barrientos
John F. Canny
UC Berkeley
Computer science division
CVE’02,
September 30-October 2, 2002, Bonn, Germany.
2
Avatar world communication
• Verbal communication
– Chat
– Spoken
• Nonverbal communication
– Change image or play animation
– Controls
• Select menu item
• Select image from palette of
images
The palace
• Interface creates disruption
between verbal and nonverbal
blaxxun
2 October 2002
CVE '02 • Bonn, Germany
Barrientos
Language embedded in physical
world
• In physical world, language
embedded in matrix of sounds
and visuals including movement
• Communication through
multiple channels is seamless
and continuous
• Avatar, as a virtual body, can
send nonverbal communication
2 October 2002
CVE '02 • Bonn, Germany
Barrientos
3
4
Problem
• Controlling a highly articulated avatar using
simple computer input devices is difficult
• Goals for gesture control technique
– Seamless with verbal communication
– Create expressive movement
– Unobtrusive interaction at a desktop or palmtop
computer
2 October 2002
CVE '02 • Bonn, Germany
Barrientos
5
Avatar gesture from pen gesture
• Body gesture has symbolic and
qualitative aspects
• Pen gesture (handwriting) carries
discrete and continuous data
• Pen gesture simultaneously selects
an avatar gesture and controls
expressive movement
2 October 2002
CVE '02 • Bonn, Germany
Barrientos
6
Illustration of technique
• Writing pen
gesture triggers
animation
• Body gesture is
“sweep” to side
• Symbol is letter ‘s’ l
• Quality being
varied is size
2 October 2002
CVE '02 • Bonn, Germany
Barrientos
7
Overview
• Background
– Nonverbal communication
– Previous approaches
• Interaction technique
• Cursive application
• Conclusions
2 October 2002
CVE '02 • Bonn, Germany
Barrientos
8
Communicative gesture
• Closely tied to speech
• Encodes ideas shared with speech
– Clarifies meaning when
speech is ambiguous
– Useful when outside noise
interferes with speech
• Multidimensional in meaning
and form
– Encodes through shape, timing, and
quality of movement
• Indicates mood/emotion
2 October 2002
CVE '02 • Bonn, Germany
Barrientos
9
Previous approaches
• Specialized widgets
Vlnet expression panel
1. Select
2. Modulate
3. Trigger
• 3 step process
ComicChat EmotionWheel
• Select and modulate
• Hunt and click diverts attention
• Avatar agency
BodyChat and ComicChat
• Driven by chat text
• Lacks direct control
2 October 2002
CVE '02 • Bonn, Germany
Barrientos
10
Continuous interaction
Computers fragment our thinking by
substituting discrete events for
continuous actions.
-Malcolm McCullough
(Abstracting Craft, p. 53)
[Gestures] can...enhance the
experience of agency through
kinesthetic involvement and the
feeling of directness.
-Brenda Laurel
(Computers as Theatre, p. 158)
2 October 2002
CVE '02 • Bonn, Germany
Barrientos
11
Why pen gestures
• Natural
– People doodle while
talking and listening
• Expressive
– Reflects emotional state
– Very personal
– Can be intentionally
manipulated
• Dual nature
– form: Analog and digital
• Multi-dimensional
2 October 2002
CVE '02 • Bonn, Germany
Barrientos
12
Interaction technique
Library
Generated
offline
Input
User writes
letter in GUI
Generation
Selection
and synthesis
Animation
Gesture
performed
wsrlibrary
GUI
2 October 2002
CVE '02 • Bonn, Germany
Gesture
generator
Avatar
animator
Barrientos
13
Interaction schematic
Cursive interaction technique.
Copyright © 2002 by Barrientos.
2 October 2002
CVE '02 • Bonn, Germany
Barrientos
14
Synthesizing expressive gesture
• An avatar gesture is represented by a set
of motion samples
– Each motion sample has different movement
qualities
– Samples recorded using human performance
capture system
– Samples made up of set of joint angle
trajectories
• Synthesize expressive gesture instance by
blending samples
2 October 2002
CVE '02 • Bonn, Germany
Barrientos
15
Motion blending example
sample 2
sample 1
interpolated gesture instance
2 October 2002
CVE '02 • Bonn, Germany
Barrientos
16
Modulation through multilinear
interpolation
Gβ
• Gesture sample I
– Rotation angles over time
• Gesture type β
– Set of 2n prototypes - G
– n style parameters
– Prototypes represent
extremal trajectories
• Gesture instance I (u,v )
β
Speed
u
k
– Vector of joint trajectories
– Multilinear interpolation on
type produces instance
Iβ(0,1)
Iβ(1,1)
β
I
(l,k)
I (0,k)
Iβ(0,0)
Iβ(1,k)
Iβ(1,0)
β
Size v
l
2 October 2002
CVE '02 • Bonn, Germany
Barrientos
17
Interpolating speed
• Interpolation along curves of
different durations
• Time dilation step
– Determine duration for
interpolated curve
– Choose sample rate on
interpolated curve
– Compress slower curve sample at proportionately
slower rate
– Sample faster trajectory at
proportionately faster rate
2 October 2002
Angle θ
CVE '02 • Bonn, Germany
t
Barrientos
18
Cursive
2 October 2002
CVE '02 • Bonn, Germany
Barrientos
Independent of specific avatar
world software
• Cursive communicates directly with all
avatar copies
• Avatar driver runs Cursive to control
gesturing avatar
• Other viewers can see gestures without
installing Cursive
2 October 2002
CVE '02 • Bonn, Germany
Barrientos
19
20
Communication
•
•
•
•
•
Driver logs into Vworld
server
Other viewers receive
notification
Other viewers request and
download avatar copy
Avatar opens socket
connection to Cursive
Cursive sends gesture
commands via socket
viewer host
driver host
Web server
open
socket
sends
commands
login
request
avatar
Download
avatar
notification
VWorld server
2 October 2002
CVE '02 • Bonn, Germany
Barrientos
21
Evaluation
• Very simple to control one parameter at
a time
• More complicated to control size and
speed simultaneously
– Effective usage requires practice
– Require further investigation into mapping
handwriting features to movement
parameters
• Viable technique for controlling avatar
gesture
2 October 2002
CVE '02 • Bonn, Germany
Barrientos
22
Summary
• Wanted: Richly expressive, spontaneous
gesture in avatar worlds
• Solution: An interaction technique
employing pen gesture input
• Cursive: An implementation of this
interaction technique
2 October 2002
CVE '02 • Bonn, Germany
Barrientos
23
Future work
• User studies
• Experiments with public multi-user virtual
worlds
• Investigate avatar gesture vocabulary
– How many gestures
– What types of gestures
2 October 2002
CVE '02 • Bonn, Germany
Barrientos
24
Conclusion
• Transmitting bodily nonverbal communication
through the internet is an exciting possibility
• Continuous interaction strengthens the
connection between the user and the avatar
• Pen gestures are a powerful technique for
controlling expressive avatar gesture
– Expressive writing is natural
– Interaction technique is compact and unobtrusive
– Maps continuous actions to continuous events
2 October 2002
CVE '02 • Bonn, Germany
Barrientos
25
Acknowledgements
My gesture model:
Erin Dare.
2 October 2002
CVE '02 • Bonn, Germany
Barrientos