Transcript L. Cairco Presentation - Socially Relevant Computing REU @ UNC

REU 2007 presents…
AVARI
Animated Virtual Agent Retrieving Information
Vision, Communication Shell,
& Conversational Flow
Lauren Cairco, primary
Toni Bloodworth, secondary
2
Your Name
2
Background & Motivation
[Thorrison, 1998]
[Simmons, 2002]
[Babu, 2005]
[Cassell, 2001]
3
Your Name
3
Who is AVARI?
4
Your Name
4
Who is AVARI?
• Animated Virtual Agent Retrieving
Information
• Answers questions about members of the
computer science faculty of UNC Charlotte
5
Your Name
5
General Architecture
Animated Agent
(Haptek)
Information Display
(HTML/Javascript)
Voice Recognition
(SALT)
Communication Shell
(Javascript)
Vision
(Matlab)
Question Resolution Algorithm
(PHP)
Your Name
Database
(MySQL)
6
6
My Contributions
Vision
Conversational Flow
Communication Shell
Your Name
7
7
Vision
• Background subtraction using Matlab
8
Your Name
8
Vision
• Camera looks at the floor
to see if anyone is
standing in front of Avari
• Begins conversation when
she sees someone
• Ends conversation when
person leaves
9
Your Name
9
Vision
10
Your Name
10
Communication Shell
• Writing code to hold all the pieces together
– PHP (PHP Hypertext Processor)
– SALT (Speech Application Language Tags)
– Haptek (for virtual human)
– Matlab (for vision)
• Communicating with group members
about needs from other pieces
• Version control
11
Your Name
11
Conversational Flow
Greeting
Get Professor Name
Get Category
Get Question
Say Answer
Goodbye
12
Your Name
12
Conversational Flow
Keep professor?
Greeting
Yes
No
Keep talking?
Get Professor Name
Yes
No
Prompt for professor
Get Category
User says professor name
Get Question
Say Answer
Keep category?
Yes
Goodbye
No
Prompt for category
User says category
13
Your Name
13
Conversational Flow
• Avari’s phrases that prompt users are
randomly selected from text files
• An additional screen displays information
that is relevant to the conversation
• Avari gives users clearer cues if she does
not recognize their speech multiple times
– Keywords on the other screen
– Example questions
Your Name
14
Sample Conversation
15
Your Name
15
Challenges in Design
• Predicting what people will say to Avari
• Deciding how to best direct conversation
while keeping it natural
16
Your Name
16
Challenges in Implementation
• Communication problems
– Server side and client side languages
– Integrating vision software
• Security restrictions in Internet Explorer
• Timing and efficiency considerations
17
Your Name
17
Skills I Learned
• PHP, SALT, MySQL, Matlab, Java,
Javascript, Haptek
• How to work with other people’s code
• Debugging without a compiler
18
Your Name
18
Vicky Fowler
Animated Agent
Morris Leblanc
Speech Recognition
Louis Fletcher
Database
Toni Bloodworth
Question Resolution Algorithm
Your Name
19
19
20
Your Name
20