Transcript Meaning beyond numbers

Conflicts of Interest


Kim Walker – No conflicts of interest to disclose
Ann Dohn – No conflicts of interest to disclose
Department of Graduate Medical Education (GME)
SES032
Meaning beyond numbers:
The power of qualitative inquiry for
program assessment
Kim Walker, PhD
Education Specialist
Ann Dohn, MA
DIO & GME Director
Session Outcomes
 Describe the use and strengths of qualitative inquiry in program
assessment.
 Describe and perform the basic steps of qualitative inquiry
including: deciding on a data collection tool, coding data for
emergent themes, member checking and interviewer corroboration
to establish validity.
 Use emergent themes and supporting qualitative and quantitative
data to provide constructive and actionable feedback to program
leadership.
 Compare and contrast the pros and cons of three software tools
designed for qualitative inquiry.
Department of Graduate Medical Education (GME)
Yes, Numbers Matter!
Hypothetical Institutional Report Card
Department of Graduate Medical Education (GME)
Qualitative Inquiry: Pros and Cons
• Qualitative data = “what” “why” & “how” of the resident and
faculty perceptions and experiences that drive these
ratings
• Explore topics in more depth and detail than quantitative
research
• Qualitative data, while meaning-FULL, can also be
unwieldy and difficult to come to any conclusion outside of
“these residents nowadays are sure an entitled bunch of
needy trophy kids!”
• Cannot generalize your findings to a broader population
Department of Graduate Medical Education (GME)
Qualitative Data Mining
Department of Graduate Medical Education (GME)
Data Source: Surveys
• Larger populations where opinions
matter and research subjects are likely
to respond to closed and open
questions
• Important to minimize survey length
and questions to only latent constructs
of interest
Department of Graduate Medical Education (GME)
Data Source: Observations
Department of Graduate Medical Education (GME)
Data Source: Interviews
Interactive collection of participant perspectives.
Types of interviews:
1.
Structured (only pre-set
questions)
2.
Semi-structured
3.
In-depth and unstructured
Uses:
1.
To clarify meaning
2.
Discuss progress/results
3.
Support exploratory work
Department of Graduate Medical Education (GME)
Data Source: Focus Groups
• Combines interviewing and participant
observation
• Uses group interaction to generate data
and gain first hand insights
Department of Graduate Medical Education (GME)
Focus Groups vs. In-depth Interviews
Use focus groups when…
Use in-depth interviews
when…
• Group interaction may
stimulate responses
• Topic amenable to less input
from individual respondents
• Logistically feasible to
assemble target respondents
in one location
• Quick turn around needed,
funds limited
• Group/peer pressure would
inhibit responses
• Subject matter is too sensitive
for group discussion
• Topic warrants greater depth
of individual respondents
Department of Graduate Medical Education (GME)
Recap: Sources of Qualitative Data
•
•
•
•
•
•
•
•
Surveys (GME or Program Specific)
Program Assessments (faculty and residents)
Observations (resident shadowing; rounding)
Interviews (individual or small groups)
Focus groups (larger consensus groups)
Document reviews (Internal reviews)
Case studies (holistic, multiple data sources)
Internet? Message boards? Residency chat rooms?
What is the word on the “streets” about our
programs?
Department of Graduate Medical Education (GME)
Data Collected: Now What?
Department of Graduate Medical Education (GME)
Data Coding: Approaching your data
General  Specific
Deductive
Inductive
Specific  General
Observation
Abductive
Department of Graduate Medical Education (GME)
Plausible
explanation and
recommendations
Coding Qualitative Data
Codes are:
Labels that assign symbolic meaning to the descriptive or
inferential information compiled during the study
Prompts or triggers for deeper reflection
Developed using multiple approaches
Deductive – Provisional “start list” from conceptual framework,
preliminary codebook
Inductive – Codes emerge progressively, no preliminary
codebook
Abductive – Based on early plausible explanations,
combination of deductive and inductive
Department of Graduate Medical Education (GME)
Tools for coding: Manual
Department of Graduate Medical Education (GME)
Data Coding: software supported
Department of Graduate Medical Education (GME)
From Text to Themes
• Thorough, well documented
analysis
• Increases opportunities for
replication
• Enhances credibility
• Clarify if broad categories (or
domains) are theoretically or
logically categorized
• How findings relate to your
theoretical framework
Department of Graduate Medical Education (GME)
Transcribed text
Organized Codes
Identifying categories
Interpret findings
Formulate relevant
themes
Highlight key findings
Mining the Data Activity:
hypothetical data set
Department of Graduate Medical Education (GME)
Mining the Data Activity:
Step 1 – Read each excerpt and highlight key words that
capture the essence of the response.
Department of Graduate Medical Education (GME)
Mining the Data Activity:
Step 2 – Summarize in few descriptive terms in “Individual
Margin Coding Notes”
Interns disconnected
Needs: structure,
organization, attention
to details
Higher standards
Department of Graduate Medical Education (GME)
Mining the Data Activity:
Step 3 – Refine coding notes into themes identified across all
responses/excerpts.
Interns disconnected
Needs: structure,
organization, attention
to details
Higher standards
Department of Graduate Medical Education (GME)
Structured,
detailed
curriculum
Mining the Data Activity:
Step 4 – Focus on key overall themes from the entire
qualitative assessment.
Department of Graduate Medical Education (GME)
Promoting Qualitative Research Validity
Strategy
Description
Extended fieldwork
To provide for both discovery and
validation, collect data over extended
period of time
Low-inference
descriptors
Use of codes/categorizes as close to
participants actual words
Triangulation
Cross-checking information and
conclusions through the use of multiple
procedures or sources: Data, Methods,
Investigator, Theory
Department of Graduate Medical Education (GME)
Promoting Qualitative Research Validity
Strategy
Description
Participant
Feedback
Researcher’s interpretations and
conclusions should be discussed with
other participants
Peer Review
Researcher’s interpretations and
conclusions should be discussed with
other people (skeptical, disinterested
peers, devil’s advocate)
Negative case
sampling
Locating and examining cases that
disconfirm the researcher’s
expectations and tentative
explanations
Department of Graduate Medical Education (GME)
What the numbers say: Not Good!
Department of Graduate Medical Education (GME)
GME Plan
1. Qualitative Analysis of:
Resident Focus Group
Three years Program and House Staff Evaluations
Internal Review Report
2. Summarize Findings – Focus and Prioritize on top emergent
issues
3. Meeting to present to program leadership
4. Follow up: monitor subsequent year surveys; follow up focus
group?
Department of Graduate Medical Education (GME)
From Analysis to
Recommendations
Program Case Study
Areas of Concern
“Background Noise”
Department of Graduate Medical Education (GME)
From Analysis to Recommendations
Department of Graduate Medical Education (GME)
From Analysis to Recommendations
Program Case Study:
• Focused on top three emergent
themes/categories of resident dissatisfaction
with learning experience, supported by
quantitative and qualitative data
1. Lack of mentoring (research and career)
2. Minimal faculty engagement/
camaraderie
3. Service over education (lack of support
staff / need for additional residents)
Department of Graduate Medical Education (GME)
Delivering the Golden Nuggets
 Provide concise, data driven findings and recommendations
 Who is receiving the information? Are they in a position to take
action?
 Define action plans, set goals and timelines
 Check back in with leadership
 Follow up with monitoring subsequent surveys; focus
group/interviewees
Department of Graduate Medical Education (GME)
From Analysis to Recommendations
Program Case Study:
• Recommendations
1. Lack of mentoring (research and career):
 Implementation of Mentoring Program
2. Minimal faculty engagement/camaraderie
 Focus on increase faculty attendance at educational
sessions (attendance tracked)
 Planned offsite retreat for faculty and residents
3. Service over education (lack of support staff / need for
additional residents)
 Hire additional support staff to handle “scut” work
 Program expansion for additional residents
• Next Step: Follow up focus group review and 2014 survey
analysis
Department of Graduate Medical Education (GME)
Concise, Data-Driven Summary
Applying quantitative metrics to
qualitative data
Department of Graduate Medical Education (GME)
Qualitative Software Tools
Pkg 1
Current Version
Pkg 2
10
7
Pkg 3
Web-based
Supported OS
Win XP SP2 Win XP SP2 or later; WinXP or Mac OS X
or later
Mac OS*
10.5 Modern Browser
Text Coding
PDF Coding
Video Coding
Audio Coding
Image Coding
YES
YES
YES
YES
YES
YES
YES
YES
YES
YES
Solid program Good for multimedia;
all round
Spanish version;
*Mac OS in 2014
YES
NO
YES
YES
NO
Web-based flexibility for
collaboration; Mac OS
Department of Graduate Medical Education (GME)
Resources
Department of Graduate Medical Education (GME)
Questions
Please feel free to contact us with any questions @ 650-723-5948
[email protected] or [email protected]
Department of Graduate Medical Education (GME)