Transcript helmut-talk
Designing and Evaluating Life-like Agents as Social Actors Helmut Prendinger Dept. of Information and Communication Eng. Graduate School of Information Science and Technology University of Tokyo [email protected] http://www.miv.t.u-tokyo.ac.jp/~helmut/helmut.html Short Bio education, experience Master’s in Logic (1994) Ph.D. in Artificial Intelligence (1998) U. of Salzburg, Austria, Dept. of Logic and Philosophy of Science Dynamic modal logic (completeness, decidability) Non-degree studies in Psychology, Linguistics, Literature U. of Salzburg, Dept. of Logic and Philosophy of Science and Dept. of Computer Science; U. of California, Irvine Incomplete reasoning (deduction, hypothetical reasoning, EBL) Post doctoral research U. of Tokyo, Ishizuka Lab JSPS Fellowship (4/1998-3/2000): Knowledge compilation, hypothetical reasoning “Mirai Kaitaku” project (since 4/2000): Life-like characters, affective communication with animated agents, markup languages for animated agents, emotion recognition Social Computing main objective and task Social Computing aims to support the tendency of humans to interact with computers as social actors. Develop technology that reinforces human bias towards social interaction by appropriate feedback in order to improve the communication between humans and computational devices. Social Computing realization Most naturally, social computing can be realized by using life-like characters. Life-like Characters at Work sample applications Tutoring, USC Knowledge Sharing, ATR Presentation, U. of Tokyo Sales, DFKI Entertainment, MIT Life-like Characters desiderata Life-like characters should be emphatic and engaging as tutors trustworthy as sales persona entertaining and consistent as actors stimulating as match-makers convincing as presenters (in short) … social actors [… and competent ] Life-like characters should enable effective and natural communication with humans Background computers as social actors Humans are biased to treat computers like real people Psychological studies show that people tend to treat computers as social actors (like other humans) Tendency to be nicer in “face-toface” interactions, ... Animated agents may support this tendency if they are designed as social actors Ref.: B. Reeves and C. Nass, 1998. The Media Equation. Cambridge University Press, Cambridge. Animated Agents as Social Actors requirements for life-likeness Features of Life-like Characters Embodiment Synthetic bodies Emotional facial display Communicative gestures Posture Affective voice Artificial Emotional Mind Affect-based response Personality Response adjusted to social context social role awareness Adaptive behavior social intelligence Outline designing and evaluating life-like characters The mind of life-like agents Emotion, social role awareness, attitude change Demo - Casino scenario Implementation and character behavior scripting Evaluating life-like characters Using biosignals to detect user emotions Experimental study with character-based quiz game Book project - character scripting languages and applications SCREAM System Architecture SCRipting Emotion-based Agent Minds Appraisal Module the cognitive structure of emotions Evaluates external events according to their emotional significance for the agent Outputs emotions joy, distress happy for, sorry for angry at resent, gloat … 22 in total Ref.: A. Ortony, G. Clore, A. Collins, 1988. The Cognitive Structure of Emotions. Cambridge University Press, Cambridge. Social Filter Module emotion expression modulating factors Ekman and Friesen’s facial “Display Rules” (’69) Expression and intensity of emotions is governed by social and cultural norms Brown and Levinson (’87) on linguistic style Linguistic style is determined by social variables: power, distance, imposition of speech acts Agent Model character profile, affect processing Character Profile Static features personality traits, standards Dynamic features static and dynamic features goals, beliefs, attitudes Attitudes (liking/disliking) are an important source of emotions toward other agents an agent’s attitude decides whether it has a positive or negative emotion (toward another agent) “happy for”– resent; “sorry for”– gloat an agent’s attitude changes as a result of communication dependent on “affective interaction history” Signed Summary Record computing attitude from affective interaction history distress (1) distress (3) angry at (2) hope (2) good mood(1) winning emotional states interaction history joy (2) gloat (1) happy for (2) time Attitude summary value positive emotions negative emotions joy (2) distress (1) hope (2) distress (3) good mood(1) angry at (2) happy for (2) gloat (1) = + Liking if positive Disliking if negative <emotion, intensity> pairs Ref.: A. Ortony, 1991. Value and emotion. In: W. Kessen, A. Ortony, and F. Craik (eds.), Memories, Thoughts, and emotions: Essays in the honor of George Mandler. Hillsdale, NJ: Erlbaum, 337-353. Updating Attitude weighted update rule If a high-intensity emotion of opposite sign occurs – e.g., a liked interlocutor makes the agent very angry Agent ignores “inconsistent” new information Agent updates summary value by giving greater weight to “inconsistent” information (“primacy of recency”, Anderson ’65) (Sitn) = 3 = (3 disliking (Sit liking w: intensity of (winning) emotion n1) h + w (Sitn) r 0.25) (5 h-weight angry , {+,} historical/recency 0.75) h/r: weight r-weight Consequence for future interaction with interlocutor Momentary disliking: new value is active for current situation Essential disliking: new value replaces summary record Life-like Agents making them act and speak Realization of embodiment Technology Microsoft Agent package (installed client-side) JavaScript based interface in Internet Explorer Microsoft Agent package 2D animation sequences Synthetic affective speech Controls to trigger character actions Text-to-Speech (TTS) Engine Voice recognition Multi-modal Presentation Markup Language (MPML) Easy-to-use XML-style authoring tool Interface with SCREAM system Life-like Characters in Interaction some demos Casino Scenario Life-like characters that change their attitude during interaction Comics Scenario Animated comics actors engaging in developing social relationships Business Scenario Animated agents that storify tacit corporate knowledge Casino Scenario life-like characters with changing attitude Animated advisor (“Genie”) Dealer (“James”), player (“Al”) User in the role of player of Black Jack game Emotion, personality Changes attitude dependent on interaction history with user Pre-scripted behavior Genie‘s Character Profile % Personality specification personality_type(genie,agreeableness,3). personality_type(genie,extraversion,2). % Social variables specification social_power(genie,user,0,_). social_distance(genie,user,1,_). % Goals wants(genie,user_wins_game,1,_). wants(genie,user_follows_advice,4,_). % Attitude Implemented with MPML and SCREAM attitude(genie,user,likes,1,init). Emotional Arc advisor’s dominant emotions depending on attitude Round 1 Round 2 Round 3 Round 4 Round 5 advisor has agreeable personality pos. attitude pos. attitude neg. attitude pos. attitude pos. attitude ignores advice ignores advice ignores advice follows advice ignores advice user looses user looses user looses user looses user wins distress (4) sorry for (4) gloat (5) sorry for (5) good mood (5) Internal intensity values advisor has agreeable personality, is socially slightly distant to user distress (1) sorry for (5) gloat (2) Intensity values of expressed emotions sorry for (5) good mood (5) Implementation Agent Scripting simple MPML script <!--Example MPML script --> <mpml> … <scene id=“introduction” agents=“james,al,spaceboy”> <seq> <speak agent=“james”>Do you guys want to play Black Jack?</speak> <speak agent=“al”>Sure.</speak> <speak agent=“spaceboy”>I will join too.</speak> <par> <speak agent=“al”>Ready? You got enough coupons?</speak> <act agent=“spaceboy” act=“applause”/> </par> </seq> </scene> … </mpml> Mind-Body Interface interface SCREAM MPML <!--MPML script showing interface with SCREAM --> <mpml> … <consult target=”[…].jamesApplet.askResponseComAct(‘james,’al’,’5’)”> <test value=“response25”> <act agent=“james” act=“pleased”/> <speak agent=“james”>I am so happy to hear that.</speak> </test> <test value=“response26”> <act agent=“james” act=“decline”/> <speak agent=“james”>We can talk about that another time.</speak> </test> … </consult> … </mpml> Alternative View smart characters vs. smart environments infers “I am happy” environment instructs agent “be happy now” “acts” expresses happiness “perceives” game state “Sense-think-act” cycle Classical AI approach Internet softbots search for information on the web, robots explore their environment All the intelligence is agent-side “tells” available behaviors behavior repository “Annotated” environments Shift from agent intelligence to environment intelligence Semantic web, ubiquitous computing, affordance theory Agents and environments can be developed independently Outline revisited designing and evaluating life-like characters The mind of life-like agents Emotion, social role awareness, attitude change Demo - Casino scenario Implementation and character behavior scripting Evaluating life-like characters Using biosignals to detect user emotions Experimental study with character-based quiz game Book project - character scripting languages and applications Affective Computing why should a computer recognize user emotions? Human-human communication Based on efficient grounding mechanisms including the ability to recognize interlocutors’ emotions (frustration, confusion,…) Humans may react appropriately upon detection of an interlocutor’s emotion (clarification upon confusion) Human-computer communication Computers typically lack ability to recognize user emotions Ignoring users’ emotions causes users’ frustration Recognizing and responding to users’ (often) negative emotions may improve users’ interaction experience Ref.: R. Picard, 1997. Affective Computing. The MIT Press. Emotion Recognition how can computers recognize users’ emotions? Stereotypes Communicative modalities A typical visitor of a casino wants… (to win) Facial display (face recognition) Prosody (speech analysis) Linguistic style (NLU) Gestures (gesture recognition) Posture (posture recognition) Physiological data Biosignals Physiological Data Assessment ProComp+ unit BVP EMG: Electromyography EEG: Electroencephalography EKG: Electrocardiography BVP: Blood Volume Pressure GSR: Galvanic Skin Response Respiration Temperature sensors GSR Inferring Emotions from Biosignals Lang’s 2-dimensional emotion model enraged excited Lang’s two dimensions joyful Arousal Biometric measures sad relaxed depressed Valence Valence - positive or negative dimension of feeling Arousal - degree of intensity of emotional response Skin conductivity increases with arousal (Picard ’97) Heart rate increases with negatively valenced emotions Note some named emotions in the arousal-valence space introverts reach a higher level of emotional arousal than extroverts Ref.: Lang, P. 1995. The emotion probe: Studies of motivation and attention. American Psychologist 50(5):372–385. Experimental Study effects of a character-based interface Biosignals to measure skin conductance and blood volume pressure (`objective’ assessment of user experience) Questionnaire (users’ subjective assessment) Instruction Show that a character with affective expression may improve users’ experience (= reduce frustration) of a simple quiz game Method Experimenter Analyser Aim of study Junichiro Mori - Addition/subtraction task (short-term memory load) Solve a series of 30 quizzes correctly and as fast as possible Frustration is deliberately caused by delay (in 6 out 30 quizzes) Subjects 20 university students (all male Japanese, approx. 24 years old) JPY 1000.- for participation, JPY 5000.- for best score Experimental Setup Instruction mathematical quiz game timer It is correct. (polite language) sometimes delay here (6 – 14 sec.) Add 5 numbers and subtract the i-th number (i < 5) 1 + 3 + 8 + 5 + 4 = [21] E.g.: subtract the 2nd number Result: 18 Select the correct answer by clicking the radio button next to the number Then the character tells whether answer is correct Two Versions of the Game affective vs. non-affective (independent variables) Affective Version Non-Affective Version Description Character expresses happiness (sorriness) for correct (wrong) answer Character shows empathy (when delay occurs) Character expresses affect both verbal and nonverbal Character does not show affective response Character ignores occurrence of delay Hypotheses Character may reduce user stress (SC) and decrease negative valence (heart rate) Character has no significant effect on user emotion (SC, heart rate) Character Responses examples of affective/non-affective feedback I am sorry. It is wrong. (hyper-polite language) I am sorry for the delay. (polite language) Hanging shoulder gesture to express sorriness non-verbally Character apologizes for the delay Non-affective feedback Non-affective feedback “Wrong.” No non-verbal emotion expression. Character ignores the occurrence of delay. Analyzing Physiological User Data BVP user response DELAY segment agent response BVP could not be taken reliably RESPONSE segment GSR delay starts delay ends Biograph Software (Thought Technologies) Preliminary Findings 9 subjects in each version (data of 2 subjects discarded) Hypothesis (design): delay induces frustration in subjects All 18 subjects showed significant rise of SC in DELAY segment Corresponds to finding in behavioral psychology (if an individual is prohibited from attaining a goal, the individual experiences primary frustration) Hypothesis (main): affective agent behavior reduces user frustration DELAY segment Non-affective version: mean = 0.05 Affective version: mean = 0.2 RESPONSE segment t-test (assuming unequal variance) t(16)=2.57; p = .01 mean values sf SC (BVP could not be taken reliably) Preliminary evaluation suggests that an animated character expressing emotions and empathy may undo some of the user’s frustration. Agents Adapting to User Emotion assumes real-time recognition of user emotions user’s action user model evidence ti node user’s traits Dynamic Decision Network (simplified) ti+1 agent’s actions user model user’s traits learning emotional state emotional state bodily expressions sensors learning bodily expressions evidence nodes sensors U QUESTION: Given user’s state at ti, which agent action will maximize agent’s expected utility at ti+1, in terms of, e.g., user’s learning and emotion? Dynamics of User Emotions ti user personality user goals ti+1 agreeableness succeed by myself provide help extraversion agent’s action have fun reproach user’s emotional state at ti+1 reproach neg valence pos valence joy Ref.: Conati, C. 2002. Probabilistic assessment of user’s emotions in educational games. Applied Artificial Intelligence 16(7-8):555–575. joy shame shame user’s emotional state at ti do nothing arousal bodily expressions eyebrows position skin conductivity heart rate sensors vision based recognizer EMG down(frowning) GSR high BVP high Outline revisited designing and evaluating life-like characters The mind of life-like agents Emotion, social role awareness, attitude change Demo - Casino scenario Implementation and character behavior scripting Evaluating life-like characters Using biosignals to detect user emotions Experimental study with character-based quiz game Book project - character scripting languages and applications Book Project character scripting languages and applications Wide dissemination of life-like character technology requires Book will offer state-of-the-art on XMLbased markup languages and tools standardized ways to represent the behavior of agents Scripting languages for face animation, H. Prendinger, M. Ishizuka (Eds.) body animation and gestures, emotion Life-like Characters. Tools, Affective Functions and Applications expression, synthetic speech, Springer Hardcover interaction with environment,… (in preparation) Characters are already used in a wide variety of applications Book contains some of the most successful character-based applications Synopsis chapters on character design useful as Standard/Reference Book State-of-the-Art in Life-like Agents Course Book for HCI, HAI, multimedia, life-like agent applications, scripting languages,… Conclusion Social Computing Designing life-like characters as social actors Human-computer interaction as social interaction Believability-enhancing agent features Emotion, personality, social role awareness, attitude change, familarity change Casino demo Future avenues – “smart” environments (character & annotated environments) Evaluating life-like characters as social actors Experimental study using user’s biosignals Life-like characters’ affective response may undo some of the user’s negative feeling Future avenues – real-time adaptivity of agent behavior to user’s emotion, decision-theoretic approach to agent behavior