Transcript assessment-of-communication-performance-in

ASSESSMENT OF COMMUNICATION PERFORMANCE
IN SIMULATED VOLCANIC CRISES
Jackie Dohaney, E. Brogt, T.M. Wilson, B. Kennedy
Postdoctoral Fellow, Geoscience Education
[email protected]
Why teach and assess
Communication Skills?
1. We typically don’t teach communication
skills
2. These are often adhoc, embedded,
implicit (hidden) learning outcomes
3. Effective communication is central to
Disaster Risk Reduction principles
ASSESSING COMMUNICATION
Project Outcome:
Build and assess curricula that improves students’
communication skills.
Research Objective:
Test & create evidence-based measures of
communication.
1) ‘Unpack’ Communication Performance
2) Variables/Proxies of Communication
3) Qualitative assessment of communication
(via student pre-post interviews)
4) Compare proxies to interviews.
ASSESSING COMMUNICATION
1) ‘Unpack’ Communication Performance
How can we measure communication?
 Effectiveness of communication is highly
contextual, and receiver-dependent.
 Anecdotal, holistic, inherited and conflicting
views on teaching communication.
-> Little educational research is dedicated to
the rigorous assessment of communication.
Different perspectives
ASSESSING COMMUNICATION
1) ‘Unpack’ Communication Performance
-> Definition of “Best Practices” in Crisis Communication
7 C’s of Communication
Comprehensible – simple, clear, jargon-free
Contextualized – diversity, cultures, differences
Captivating – engaging, relevant
Credible – open, frank, acknowledges uncertainty
Consistent – backed by evidence, confirmable
Courteous – compassionate, empathetic, respectful
(Addresses ) Concerns – empowers action/response
Vivienne Bryner (University of Otago, NZ)
http://www.youtube.com/watch?v=grhrLT8tfjg
ASSESSING COMMUNICATION
Curricula: Role-plays & Simulation
To acquire and perfect communication skills,
research suggests students need to practice them
in authentic scenarios. (Cox et al. 2012).
Role-playing, modelling, coaching, rehearsal,
reinforcement and feedback, enable individuals
to improve their communication competence.
(Richmond et al. 2013).
ASSESSING COMMUNICATION
2) Variables that contribute to communication?
Student
Interviews
Actual
performance
i.e., confidence
ASSESSING COMMUNICATION
Variable: Communication Experience
Measure: Communication Experiences Survey
(Validation Process; Dohaney et al 2015)
Experiences:
Oral presentations
Poster presentations
Debates & Speeches
In the classroom (Teaching)
Group Discussions & Meetings
Professional Media
Estimated number of
experiences
None
Few (1-10)
Some (11-20)
Many (21-30)
A lot (>30)
PLEASE COMPLETE THE SURVEY
Setting/style:
- At conferences, in your department, during your education
- Provided, received or self-evaluative feedback for
communication?
ASSESSING COMMUNICATION
Variable: Communication Efficacy
Measure: Self-perceived communication competence
SPCC Instrument (McCroskey and McCroskey 1988)
Statements:
1. Present a talk to a group of strangers.
2. Talk with an acquaintance.
3. Talk/Discuss at a large meeting of friends.
4. Talk in a small group of strangers.
5. Talk with a close friend.
6. Talk/Discuss at a large meeting of acquaintances.
7. Talk with a stranger.
8. Present a talk to a group of friends.
9. Talk in a small group of acquaintances.
10. Talk/Discuss at a large meeting of strangers.
11. Talk in a small group of friends.
12. Present a talk to a group of acquaintances.
Likert Choices:
Very strong ability
Strong ability
Average ability
Poor ability
Very poor ability
Can look at gender, age, nationality,
curricula-type, etc to assess any factors that
might influence pre- scores and “changes”
(post- pre).
ASSESSING COMMUNICATION
Perceptions of Crisis Communication:
Content
 Language & Terms
 Amount of content
 Level of detail
 Equity & Diversity
Delivery
 Visual Aids
 Media-type
 Tone
 Intent of message/Purpose
Different Audiences
 Scientists, Emergency Managers
 The public and communities
ASSESSING COMMUNICATION
Variable: Science Communication Best Practices
Measure: Perceptions of Science Communication
Best Practices (Development process; Dohaney et al 2015/16)
Examples of Statements:
1. In a crisis, scientific information presented
to the PUBLIC should appear open and
completely transparent.
Likert Choices:
Stongly Agree
Agree
Neutral
Disagree
Strongly Disagree
2. In a crisis, scientific information presented to the
PUBLIC should be comprehensive (including all
scientific facts discussed).
3. In a crisis, when communicating to the PUBLIC, it is
appropriate to show your emotions
ASSESSING COMMUNICATION
Perceptions: Better Communicators
ASSESSING COMMUNICATION
Perceptions: Scientific Responsibilities
ASSESSING COMMUNICATION
3) Qualitative assessment of communication (via
student pre-post interviews)
-> Direct Measure of Communication Performance
Instrument- Presentation Skills Protocol for
Scientists (2PS)
Rubric to guide coding
of pre- and
post-interviews
ASSESSING COMMUNICATION
Pre-Interview Script:
Question: How do scientists actually know that a
volcano is going to erupt?
“ That goes back to the monitoring that we are
always doing.
Our scientists know when something is happening,
that is different from what we usually see. That’s
when we will investigate further.
And ya know, break it down into what that actually
means.”
ASSESSING COMMUNICATION
Post-interview script:
Question: How do scientists actually know that a
volcano is going to erupt?
“So we monitor the volcanoes constantly. And we have
different methods of doing this. Scientists know what to
look out for.
We monitor it so frequently that, anything out of character,
of the background level, so that we know something is a
different kind of activity.
So we will look into that further. And from there we can
figure out if that is a normal process, or an increase in
activity”
ASSESSING COMMUNICATION
Future Work:
(4) Compare proxies to interviews.
(5) Incorporating the qualitative coding into a
robust rubric for assessment of communication
performances that is appropriate to crisis
contexts
Excellent
Average
Poor
Thank You! Any Questions?
Jackie Dohaney
Postdoctoral Fellow, Geoscience Education
[email protected]
Perceptions (please see me!)
Communication Experiences Survey
http://canterbury.qualtrics.com/SE/?SID=SV_9Xrc0QfnzZnyN6d
SPCC Study Participants
Student participants (n=37) were recruited from third and fourth
year physical volcanology and hazards management courses
which hosted the volcanic hazards simulation as part of their
curricula.
Two iterations of the simulation were tested; One was embedded
at the end of a 7-day field course (took place in January, 2012;
n=19) the other was embedded within a lecture-based course
(took place in August 2012; n=18).
They were mixed cohorts of American study-abroad students and
New Zealand students who attended the University of
Canterbury. Students ranged in gender (female (13) and male
(24)), nationality (New Zealand (24), United States of America
(11), Netherlands (1) and India (1)), and age (aged 19 – 22 (20)
and  23 years old (17)).