Transcript Feb 2012 Open Response Surveys

Design and Analysis
of Open Response Surveys:
Lessons Learned
Dr. Joan Burtner
Associate Professor of Industrial Engineering
Mercer University
Design and Analysis of Open
Response Surveys

Designing Survey

Administering Survey

Conducting Survey Analysis

Reporting Survey Results
IIE/SHS FEB 2012
Mercer University School of Engineering
Slide 2
Open Response Surveys

Ask participants to respond to a question such as
“What can we do to improve patient care?

Excellent method for collecting “Voice of the
Customer” data within a Quality Mgt. System

Allows analysts to interpret qualitative
characteristics (tone, frustration, pride)

Opportunity to discover unanticipated responses
IIE/SHS FEB 2012
Mercer University School of Engineering
Slide 3
Designing and Administering the
Survey

Purpose of survey

Delivery method (paper, electronic, phone, in-person, etc.)

Length of survey, allowable response time

Incentives for participation

Anonymity, confidentiality

Who will collate and/or analyze data?

How will data be analyzed?
IIE/SHS FEB 2012
Mercer University School of Engineering
Slide 4
Design of Survey Questions

Structure, specificity of written questions
◦ What can we do to improve the process?
◦ What is the one thing that will make your job easier?
◦ What is the “one idea” that will improve the
registration process?

Text-based questionnaires vs. oral interviews
◦ Limited, prescribed, uniform questions
◦ Non-uniform follow on questions in addition to
prescribed questions
IIE/SHS FEB 2012
Mercer University School of Engineering
Slide 5
Example 1: On-line Survey Question
Please describe challenges that make it difficult to deliver
patient care.

Over 700 responses submitted anonymously
over a two week period

Data downloaded into an Excel file

Many rows of data contained no text, just time
and date of submission

Two 2-person teams reviewed the data and
developed preliminary codes
IIE/SHS FEB 2012
Mercer University School of Engineering
Slide 6
Example 1: Initial Coding

Coding done electronically by labeling each
response in Excel

Initial decision to allow only one major code for
each entry to simplify analysis.

Twenty-four categories emerged between the
two teams.

Overlap and clarification of labels resulted in the
elimination of four categories.
IIE/SHS FEB 2012
Mercer University School of Engineering
Slide 7
Example 1: Revised Coding

Difficult to gain consensus among all four
coders on the “one” main concept for each
response

Realization that multiple codes increased validity
of work

Excel spreadsheet expanded to four columns to
accommodate multiple codes for each response
IIE/SHS FEB 2012
Mercer University School of Engineering
Slide 8
Code - Process Change?

Patients should not be transferred to med surg during
change of shift. There needs to be a window of time that
patients cannot be transferred. It is very unfair to the
patient to be transferred at that time as going off nurses
are trying to get caught up and give report; and coming
on nurses are trying to get report. For example; no
transfers between 6:00 and 07:15pm
IIE/SHS FEB 2012
Mercer University School of Engineering
Slide 9
Code - Work Load? Policy?

If a patient is computer literate, why do we not have
some of the admission forms on line so that patients can
fill out the questions at home? They would certainly
have access to their medications & we might have a
more exact list of what they take. They would be in a
more comfortable & less stressful environment when
filling out the forms & the older folks would most likely
have a family member to assist in remembering when
they had previous procedures, etc.
IIE/SHS FEB 2012
Mercer University School of Engineering
Slide 10
Audience Participation Exercise

Form teams of two or three

Consider the following responses
(handout)

How would you code the data?

Do multiple codes increase validity?
IIE/SHS FEB 2012
Mercer University School of Engineering
Slide 11
Example 2: Nursing Interruptions

Research agreement with a local hospital

Students observed nurses at work

Data collected orally and transcribed by
observers

Anecdotal reports from nurses collected
periodically as time permitted
IIE/SHS FEB 2012
Mercer University School of Engineering
Slide 12
Example 2: Nursing Interruptions
Data Excerpt 1
April 15th
Comments by Nurse - Interruptions
Helping with code lift or code response
Family
Telephone calls
Doctors
When technician needs help
Comments by Observer 1
Tech was using chart station when Nurse was trying to chart
Having room in another section causes a lot of extra walking
This could be a system failure interruption
IIE/SHS FEB 2012
Mercer University School of Engineering
Slide 13
Example 2: Nursing Interruptions
Data Excerpt 2
April 16th
Comments by Nurses - Interruptions
Family
Lab Calls asking about results or talking about results
Doctors calling
Patients – getting called by other patients and helping other patients
Unexpected events – Patients getting out of bed, codes
Helping nurses, covering nurses when they are out for lunch
Comments by Observer 2
Quiet day with not many interruptions
Nurse didn’t have much to do
IIE/SHS FEB 2012
Mercer University School of Engineering
Slide 14
Coding Nursing Interruptions

Nature of the interruption
◦ Phone call , call light, patient’s family,
healthcare professional, etc.

Urgency of the interruption
◦ Avoidable, justifiable, etc.

Consequence of the interruption
◦ Delay, error, etc.
IIE/SHS FEB 2012
Mercer University School of Engineering
Slide 15
Example 3: On-line Survey Question
Please list and describe ideas that will contribute to cost
efficiency within our system.

Over one hundred responses submitted

Data downloaded into an Excel file

Single person reviewed the data and categorized
each response

Fifty-eight usable responses

Data organized into a pivot table
IIE/SHS FEB 2012
Mercer University School of Engineering
Slide 16
Reporting Analysis Results

Example 1
◦ Ten-page written report
◦ Power-Point presentation to five-person committee
◦ Follow-up submission: frequency distribution models
overall and departmentalized

Example 2
◦ In-person feedback to nurse manager
◦ Brief report with Pareto diagrams

Example 3
◦ Electronic feedback to client
◦ Pivot table demonstration
IIE/SHS FEB 2012
Mercer University School of Engineering
Slide 17
Code Frequency - All Locations
IIE/SHS FEB 2012
Mercer University School of Engineering
Slide 18
Frequency - Aberdeen
IIE/SHS FEB 2012
Mercer University School of Engineering
Slide 19
Frequency - Dallas
IIE/SHS FEB 2012
Mercer University School of Engineering
Slide 20
Frequency - Greenville
IIE/SHS FEB 2012
Mercer University School of Engineering
Slide 21
Example 3: Pivot Table 1
IIE/SHS FEB 2012
Mercer University School of Engineering
Slide 22
Example 3: Pivot Table 2
IIE/SHS FEB 2012
Mercer University School of Engineering
Slide 23
Example 3: Pivot Table 3
IIE/SHS FEB 2012
Mercer University School of Engineering
Slide 24
Lessons Learned 1

Nurses are willing to spend time completing
surveys.

Many are eager to have their ideas heard.

It is important to give feedback, when possible,
with respect to how the survey results may
change the process.
IIE/SHS FEB 2012
Mercer University School of Engineering
Slide 25
Lessons Learned 2

The process of analyzing and quantifying openended responses is very time-consuming.

It is very difficult to gain consensus on some
responses.

System-wide data should also be reported at
the unit level if possible.

In the case of “one best idea” it is difficult to
prioritize.
IIE/SHS FEB 2012
Mercer University School of Engineering
Slide 26
Lessons Learned 3

Due to the somewhat subjective nature of the
coding and frequency tabulation, it may be
difficult to use open-ended survey responses to
measure the effectiveness of process changes.

However, changes in the culture and attitudes of
the respondents may be apparent.
IIE/SHS FEB 2012
Mercer University School of Engineering
Slide 27
Questions / Contact Information

Dr. Joan Burtner
[email protected]

Associate Professor of Industrial Engineering
and Industrial Management

(478) 301- 4127

Mercer University School of Engineering
1400 Coleman Avenue
Macon, GA 31207
IIE/SHS FEB 2012
Mercer University School of Engineering
Slide 28