Transcript Natural Interaction with Virtual Characters and Virtual Environments

Immersive Virtual Characters for
Educating Medical Communication Skills
T. Bernard, C. Oxendine, D. S. Lind, P. Wagner
Dept of Surgical Oncology (Medical College of Georgia)
K. Johnsen, A. Raij, R. Dickerson, R. Wells, B. Lok
Dept of Comp and Info Science and Eng (College of Eng)
M. Cohen, A. Stevens
Dept of Surgery (College of Med)
J. Cendan, M. Duerson, R. Pauly
Dept of Community Health and Family Med (College of Med)
R. Ferdig
College of Education
Previous Work: Technology for
Medical Communication Skills
Traditional approaches [Bearman 2003]
– Narrative
– Problem Solving
Human Patient Simulator [Meurs 1997]
– Mechanical simulation
– Motors and actuators for I/O
JUST VR [Ponder 2002]
– Immersive approach (Stereo Projection)
– Trains students to react to emergency
situations
– Uses human instructor to control system
Virtual Standardized Patient [Hubal
2000]
– Commercial Desktop application by RTI
– Speech Recognition
– Natural Language Engine
Project Description
Simulate a standardized
patient encounter
Allow repeated interaction with
an Immersive, interactive
virtual patient in a constrained
scenario
Virtual patient, DIANA
Virtual instructor, VIC
To address issues with SPs
– Experience diversity
– Quality Control
– Feedback
Communication skills
No physical diagnosis
Interpersonal Simulator
Play Video
Things to look for: interaction modalities
System
Low Cost
– < $8,000(USD)
COTS
Components
Potential:
– Every Hospital
Natural Interaction Input
No Keyboard, No Mouse
Speech Recognition
– Dragon Naturally Speaking 8
Pro
– Accuracy 90+% with 10
minutes training
– 70% match to database
Track Communication Cues
– Non-Verbal
Track head gaze
Track left hand
Track body lean
– Verbal
Inflection
Jargon
Gesture Recognition
– Pointing, handshake
Natural Interaction Output
DIANA and VIC look
at user
Life-size characters
Animation
– Hand gestures
– Head movement
Perspective-Correct
Rendering
Why this works
– Does not rely on
complete sentences
– Constrained scenario
– Students trained on
specific questions
Eight Studies
2004
–
–
–
–
April: Project initiated
August: Prototype (n=7) UF
October: Experts (n=3) UF
December: Pilot Test (n=10) UF
2005
–
–
–
–
June: Two Institutions (n=16) UF/MCG
July: VP vs SP (n=16, n=8) UF/MCG
October: Cultural Bias (n=16) MCG
October: Class Integration (n=33) UF
n = 101
Testing Centers
– Harrell Center at UF
– Medical College of Georgia
– Diana was in Exam Room #3 (video)
Eight Studies
VP ≈? SP
How is experiencing an interpersonal scenario with a virtual
person similar to – and different from – experiencing an
interpersonal scenario with a real person?
Clearly different
– But… in what important ways?
Patient-doctor interview provides a constrained scenario
The study asks:
– Are post-encounter impressions similar?
– Are empathy and other emotions and attitudes similarly expressed?
– Which social constructs are followed?
These questions must be explored to:
– Determine the extent to which interpersonal scenarios can be simulated with
virtual humans
– Identify how component technologies need to improve to enable effective
interpersonal virtual human systems
Overall Performance
Similar Experiences
–
–
–
–
Same % of participants asked key questions
Out of 12: 6.3 ± 1.7, 5.5 ± 2.1 (α = 0.37)
Same % of participants passed the scenario
SP = 50%, VP = 36%
But…
–
–
–
–
Some critical items were not asked at the same frequency
Sexual activity (SP = 0.88, VP = 0.44)
Nausea (SP=0.88, VP=0.25)
Related to virtual patient expressiveness
Behavioral Measures
Empathy
– Empathetic moment – “I’m scared, can you help
me?”
– Expressed the same % and same # of times (SP
= 2.2, VP = 1.3)
But…
– Appears less genuine (very robotic)
– Conversation flow is “rapid-fire”
Confirmatory phrases statistically different (SP = 20, VP
= 3.5)
Lessons Learned
Overall experiences similar
– Questions asked
– Global measures
Education goals met
– Students rated educational merits similarly
– Students rated difficulty similarly
Global measures of realism do not work
– Battery of specific measures more accurate
Practice tool in addition to SPs
Diana rated a 6.36(.85) on a 1 to 10 scale (Average
is a 7.47 for real SP’s)
VR works out of the lab
Papers
Computer Science
– Raij,et al., “Interpersonal Scenarios: Virtual≈Real?”, IEEE Virtual Reality 2006
– Johnsen, et al., “Evolving an Immersive Medical Communication Skills Trainer”, Journal
on Presence: Teleoperators and Virtual Environments
– Dickerson, et al., “Virtual Patients: Assessment of Synthesized Versus Recorded
Speech”, Medicine Meets Virtual Reality 14
– Johnsen, et al., “Experiences in Using Immersive Virtual Characters to Educate Medical
Communication Skills”, IEEE Virtual Reality 2005
– Dickerson, et. al., “Evaluating a Script-Based Approach to Simulating Patient-Doctor
Interaction”, SCS 2005 Int’l Conf on HCI Adv for Modeling and Sim
Medicine
– Stevens, et al., “The Use of Virtual Patients to Teach Medical Students History Taking
and Communication Skills“, American Journal of Surgery
– Lind and Lok, “The Role of Virtual Patients in Medical Education: Teaching Tool Versus
Technological Trend”, FOCUS on Surgical Education
– Stevens, et al., “Implementing a Virtual Patient into the Medical School Curriculum at
the University of Florida”, Southern Group on Education Affairs 2006 (pres.)
– Cohen, et al., “Do Health Professions Students Respond Empathetically to a Virtual
Patient?”, Southern Group on Education Affairs 2006 (pres.)
– Bernard, et al. “A Multiinstitutional Pilot Study to Evaluate the Use of Virtual Patients to
Teach Health Professions Students History-Taking and Communication Skills”, Society
of Medical Simulation 2006
– Stevens, et al., “The Use of Virtual Patients to Teach Medical Communication Skills”,
Assoc of Surg Educ 2005, Southern Group on Educational Affairs 2005 (pres.)
Current Work
Incorporation into the
classroom
UF Essentials of Patient
Care
– n=34 students this year
– n=135 next year
MCG Patient
Communication
– n=200 each year
Study
– Real Speech vs. Synethsized
Speech
– Immersive vs. Non-immersive
Do people carry real world biases into
the virtual world?
Do people carry real world biases into
the virtual world?
The Use of a Virtual Scenario to Teach
Communication Skills for Geriatric Patients
Patient Diversity
Do students treat
different virtual patients
similarly?
Measures
Eye gaze
Body Lean
Vocal Inflection
Time
Interruptions
Communication Skills
Fundamental to clinical practice.
Patient & physician satisfaction.
Patient understanding & adherence.
Healthcare outcomes.
Malpractice litigation.
Learned skill & experiential
learning alone is insufficient.
Virtual Objective Structured Exam
(VOSCE)
NBME
Curriculum
integration
UF
1st year Medical Students
MCG
1st year Medical Students
n = 135
n = 200
EPC courses
ECM courses
(Semester 1)
(Semester 1)
Virtual Patient Experience
(Early 1st semester)
EPC courses
(Semesters 2-4)
ECM courses
(Semesters 2-4)
Virtual Patient Teams
T. Bernard, C. Oxendine, D. S. Lind, P. Wagner
Dept of Surgical Oncology (Medical College of Georgia)
K. Johnsen, A. Raij, R. Dickerson, R. Wells, B. Lok
Dept of Comp and Info Science and Eng (College of Eng)
M. Cohen, A. Stevens
Dept of Surgery (College of Med)
J. Cendan, M. Duerson, R. Pauly
Dept of Community Health and Family Med (College of Med)
R. Ferdig
College of Education