Transcript View Presentation

Tune-up Time
Using Focus Groups & Cognitive
Interviews to Re-Evaluate
an Established Survey
Carol Cosenza
Center for Survey Research
University of Massachusetts Boston
Acknowledgments
• Thanks to Jack Fowler (CSR) and Paul
Cleary (Yale)
• Work on this project was supported by a
cooperative agreement from the Agency
for Healthcare Research and Quality
(#U18HS016978).
• Focus Groups
• Usually done before writing the survey
• Learn about ideas/concepts/terminology
• Formulate hypotheses
• Cognitive Interviews
• After survey questions written
• How do Rs handle cognitive tasks (Comprehension, Retrieval of
Information, Judgment, and Formation of Answers)
The Survey
Experience of
Care and
Health
Outcomes
Background
• For those who have received mental health or substance
abuse services through a health plan
• Consumer Assessment of Healthcare Providers and
Systems (CAHPS®)
• Family of surveys (CAHPS Dental, CAHPS Hospice, Clinician & Group)
• Original goal to develop standardized patient surveys used to
compare results across health plans
• Now focus on providers and used nationally to report on quality of
care
• Consumers/patients report on and evaluate their
experiences
• Access to care
• Communication with providers
Creating the ECHO® Survey
• Work started in 2000 and continued for 6+ years
• Included stakeholder meetings, focus groups, cognitive
interviews, and field testing
• Combination of 2 surveys
• Mental Health Statistics Program's Consumer Survey (MHSIP)
• Consumer Assessment of Behavioral Health Services (CABHS)
• Endorsed by the National Quality Forum July 2007
• Survey content
• Not based on a specific provider encounter
• Focuses on “counseling and treatment” & overall ratings
• Includes Qs about administrative services (of health plan)
What you’re thinking now
• Did she say the survey was written
15 years ago?
• Why is a mental health survey called
“ECHO”?
• If you were paying VERY close attention:
Do respondents know what “counseling
and treatment” means?
Things have changed
in the last 15 years
Insurance Changes
• Structure
• Accountable Care Organizations (ACO)
• Patient Centered Medical Homes (PCMH)
• Affordable Care Act
• Not allowed to exclude pre-existing conditions
• Kids can stay on parent’s account until 26
• Overall fewer uninsured
• Change in how clients interact with insurance
companies
• Methods of communication
• Choices and availability
Changes in Mental Health Care
• Mental Health Parity
• Coverage for mental illness must be comparable to
that of physical ailments
• ACA
• Mental health and substance abuse treatment as one
of the 10 “essential health benefits”
• Antidepressants and other medications
• More common
• More prescribed by non-mental health professionals
• Advertised direct-to-consumer
• Societal changes
Our Task
Add some questions about mental health
care (from ECHO) into a general provider
survey (CG-CAHPS)
• We have questions already written
• We know CG-CAHPS works
• We want to know if adding these Qs is OK
Some questions included A LOT
of definitions (that were ignored)
In the last 6 months (not counting emergency
rooms or crisis centers), how many times did you
go to an office, clinic, or other treatment program
to get counseling, treatment or medicine for
yourself?
None
1 to 10
11 to 20
21 or more
1
Some questions included
A LOT of separate phrases
2
In the last 6 months (not counting emergency
3
rooms or crisis centers), how many times did you
go to an office, clinic, or other treatment program 4
to get counseling, treatment or medicine for 5
6
yourself?
None
1 to 10
11 to 20
21 or more
The next questions ask about your counseling or treatment.
Do not include counseling or treatment during an overnight
hospital stay or from a self-help group.
Professional counseling can include care from
psychologists, psychiatrists, social workers, nurse
practitioners, or other professional counselors.
18. In the last 6 months, did you call someone to get
professional counseling on the phone for yourself?
The next questions ask about your counseling or treatment.
Do not include counseling or treatment during an overnight
hospital stay or from a self-help group.
Professional counseling can include care from
psychologists, psychiatrists, social workers, nurse
practitioners, or other professional counselors.
18. In the last 6 months, did you call someone to get
professional counseling on the phone for yourself?
Multiple Providers
• We found a lot of Rs saw more than one
provider for mental health care
• Talked to a counselor/therapist
• Got medicine from someone else
• Involvement with Primary Care doctors
• Different experiences – hard to summarize
• Making appointments
• Wait time
• Overall rating
Some phrases not understood
• “treatment”
• Some questions used “counseling or treatment” –
others used “counseling, treatment, or medicine”
• Rs didn’t know what to include
• Especially problematic in in context of provider survey
• “crisis center”
• “professional counseling on the phone”
• getting an appointment for counseling or
treatment “as soon as you needed”
• Standing appointments
• Some thought this was double-barreled
Overall Findings
It didn’t work
Time to Pull out the Toolbox
– what can we do now?
Expert Review
• Interest in access to care
• How easy to get services – counseling and medicines
• ECHO is PLAN-based (not provider-based)
• Harder for Rs to summarize
• How mental health services are provided – and
how patients use those services – has changed
Cognitive Interviews – Round 2
• Split out getting counseling from getting
medicines
• Most Rs thought getting care “as soon as
needed” was about emergency care
• Difference between how hard it is to get
something (appointments & medicines)
and how often you were able to get it as
soon as you needed
Overall Findings – Round 2
It still needs work
Need to look at the
entire instrument
Evaluation
Options
• Are the questions (still) consistently understood?
• Do the Qs accurately describe patients’
experiences?
• Are concepts asked about still relevant?
• Any new concepts that should be asked about?
What we talked about - Access
•
•
•
•
Getting first appointment
Getting other appointments
Provider continuity
How this differs when talking about getting
counseling and getting medicines
What we talked about –
New ways of interacting
• Web-based
• Face-to-Face (Skype-like)
• Live chat
• Messaging
• Texting
• Phone
What we talked about Insurance
• Most had no idea about topics that were asked
about
• Limits on number of visits/ “using up” benefits
• Referrals
• Paperwork and calling customer service line
• Most paperwork seems to be done through
providers office
• When survey was written, health plans were
trying to manage/limit mental health services
Other things talked about –
What to call it
• “Mental health” not as stigmatizing as it
once was
• No one calls it (or likes) “behavioral health”
• “Treatment” needs a modifier – especially
if asking about it in context of other health
care
• “Counseling or mental health treatment”
might work
Other things talked about
• Emergency care
• Getting care right away works in other CAHPS
instruments – still not sure why it’s not working here
• Involvement in care
• Other treatment options
• Mandatory treatment
• This came up when talking about the right to refuse
treatment
What did we learn?
• Cognitive interviews showed we had
problems
• Focus groups gave us a big picture about
how mental health care has changed
• We live in a different world in 2016 than
we did in 2000
• New technology
• New ways of interacting with ALL health care
Where are we?
• Reframe Survey
• Move towards a provider-based instrument asking Qs
about experience with one provider
• Need more/different Qs about access
• Re-evaluate terms that may no longer be relevant
• Crisis Center
• Call someone for care on phone
• Perhaps add Qs about new modes of mental
health care
What’s next?
• Meet with stakeholders
• Modify survey
• What we learned from cognitive interviews
and focus groups
• Make more consistent with other CAHPS
surveys
• Cognitive interviews with the revised
instrument
Thank you