Transcript Evidence-based practice: Challenges and evolving remedies

Evidence-based practice and policy in
California: Choices ahead
Eileen Gambrill
School of Social Welfare
University of California at Berkeley
Presented July 14, 2005 Leadership Symposium on Evidence-based
practice in the human services, Sacramento, California
My goals:
1.
2.
To describe the philosophy and process of evidence
based practice and policy as presented in original
sources (e.g. Sackett et al, 1997, 2000; Gray, 1997,
2001).
This is a new educational and practice paradigm for
closing the gaps between research and practice in
order to maximize opportunities to help clients and
avoid harm. It attends to ethical, evidentiary and
application concerns. It has not “been around for
decades” as suggested in Guide for Child Welfare
Administrators on Evidence Based Practice.(2005)
Chadwick Center, Children’s Hospital of San Diego,
p. 5.
Describe the origins of evidence-based practice.
Gambrill (2005)
My goals, cont.:
3. Suggest choices ahead for the state of California
regarding what view of evidence-based practice and
policy to use.
• Will we take advantage of the new and evolving paradigm of
EBP described by its originators that:
– is systemic; it has implications for educators and
researchers as well as for administrators, line staff and
clients
– attends to ethical issues such as informed consent
– attends to application issues
– offers a way to handle uncertainty in making decisions in
an honest way (sharing ignorance as well as knowledge;
there is a willingness to say “I don’t know)
Gambrill (2005)
My goals, cont.
• Will we define this narrowly as basing decisions
on practice related research; using practice
guidelines.
This view leaves out the role of clinical expertise,
attention to client values and preferences, and
application problems.
• Will we define this as business as usual: use the
name but not the substance.
Gambrill (2005)
Ethical Obligations
•
•
•
•
Beneficence
Avoid Harm
Informed Consent
Autonomy; Self-determination
Gambrill (2005)
Evidence-based Practice and Care
EBP “is the integration of best research evidence with clinical
expertise and [client] values (Sackett, Straus, Richardson,
Rosenberg & Haynes, 2000, p.1).
It is the conscientious, explicit and judicious use of current
best evidence in making decision about the care of individual
[clients]” (Sackett, et al.,1997, p.2).
Evidence-based health care refers to “use of best current
knowledge in decision making about groups and populations”
(Gray, 2001).
The origin of the philosophy and process of EBP is not the
Institute of Medicine (IOM). (See for example Sackett et al.,
1997, 2002; Gray, 1997, 2001).
Gambrill (2005)
A Model for Evidence-Based Decisions
Client Characteristics and Circumstances
Clinical Expertise
Client preferences and actions
Research Evidence
Haynes, R., Devereaux, P. & Guyatt, G. (2002). Editorial: Clinical expertise in the era of evidenceBased medicine and patient choice. ACP Journal Club, 136: A-11-14.
Gambrill (2005)
Evidence-based Practice, cont.
Clinical expertise includes use of effective relationship skills and
the experience of individual helpers to rapidly identify each
client’s unique circumstances, characteristics and “their
individual risks and benefits of potential interventions and their
personal values and expectations” (p.1). It is drawn on to
integrate information from these varied sources (Haynes,
Devereaux & Guyatt, 2002).
Client values refer to “unique preferences, concerns and
expectations each [client] brings to an…encounter and which
must be integrated into…decisions if they are to serve the
[client]” (p.1).
Gambrill (2005)
The process of Evidence-based Practice:
1.
2.
3.
4.
5.
Convert information needs related to practice decisions
into answerable questions
Track down, with maximum efficiency, the best evidence
with which to answer them
Critically appraise that evidence for its validity, impact
(size of effect) and applicability (usefulness in practice)
Apply the results of this appraisal to practice/policy
decisions. This involves deciding whether evidence
found (if any) applies to the decision at hand (e.g., is a
client similar to those studied, is there access to services
described) and considering client values and preferences
in making decisions as well as other application
concerns
Evaluate our effectiveness and efficiency in carrying out
steps 1-4 and seek ways to improve them in the future
(Sackett et al., 2000, pp.3-4).
Different Kinds of Questions
• Effectiveness: In elderly clients who are depressed,
what method is most effective in reducing depression?
• Prevention: In poor, inner-city children, are head start
programs effective in decreasing school drop out?
• Risk/prognosis: In children who are abused, are
actuarial or consensus risk assessment measures most
predictive of future abuse?
• Description/Assessment: In families in which there is
parent-child conflict, is self report or observation most
accurate in describing interaction patterns?
Gambrill (2005)
Different Kinds of Questions, cont.
• Other kinds of questions include harm, cost and selfdevelopment.
• Different questions require different kinds of research
methods to critically appraise proposed assumptions.
These differences are reflected in the use of different
“quality filters” to search for and appraise related
research (e.g. see Gibbs, 2003; Sackett et al, 2000).
Gambrill (2005)
Critical Thinking Values: Integral to EBP
• Courage: Critically appraise claims regardless of negative
reactions
• Curiosity: An interest in deep understanding and learning
• Intellectual empathy: Accurately understanding and
presenting the views of others
• Humility: Awareness of the limits of knowledge including
our own; lack of arrogance (e.g. promoting bogus claims
of effectiveness)
• Integrity: Honoring the same standards of evidence to
which we hold others
• Persistence: Willingness to struggle with confusion and
unsettled questions
Based on R. Paul (1993). Critical Thinking: What every person needs to survive in a
Rapidly changing world (3rd Ed.) Foundation for Critical Thinking.www.critical thinking.org
Gambrill (2005)
Examples of Innovative Technology
• Evidence cart (Strauss and Sackett, 1998)
• Knowledge manager (resource person)
(Gray, 1998)
• Decision aids (e.g. computer based, client
informed choice form)
• Systematic Reviews: Cochrane and
Campbell databases
Gambrill (2005)
Evidence-based practice:
• Is an alternative to authority-based practice.
Appeals to:
A famous person
“If Freud said it,it must be true.”
Popularity
“Eighty percent of social workers
use…I’m going to use it too.”
Tradition
“That’s the way we have always
done it.”
Consensus
“We all believe that ____”
Gambrill (2005)
Evidence-based practice
• It is as much about the ethical obligations of educators and
researchers to be honest brokers of knowledge and
ignorance as it is about the obligations of practitioners and
administrators.
• Many kinds of “evidence” may be considered in deciding
what outcomes to pursue and what services to offer
including:1) client preferences, 2) evidentiary status of
different services, 3) resources available, 4) cost, 5) legal
regulations, 6) codes of ethics
Gambrill (2005)
Implications of Evidence-Based Practice
• Move away from authority-based decision making
(appeal to tradition, consensus, popularity, status).
• Honor ethical obligations to clients such as informed
consent.
• Make practices and policies and their outcomes
transparent.
• Attend to application problems: Encourage a systemic
approach to improving services.
• Maximize knowledge flow: Honest brokering of
knowledge: Sharing ignorance and uncertainty as well
as knowledge.
Gambrill (2005)
Origins of Evidence-Based Decision Making*
1.
Study of variations in service delivery and clinical
practice.
Identification of gaps that show that people were not
acting systematically or promptly on research findings.
2.
•
Failure to start services that did more good than harm at
reasonable cost.
Failure to stop services shown to be of little value.
•
3. Economic pressures.
4. Key individuals (e.g., David Sackett)
5. The knowledge revolution:
•
•
•
•
Gambrill (2005)
Increased recognition of harmful side-effects of health care
Increased recognition of misplaced faith in adequacy of peer
review
Flaws in books and editorials
The evolution of the systematic review and web based updates
Origins of Evidence-Based Decision
Making*, cont.
6. The web revolution.
7. Appeal to clinicians (breaking down the division between
research and practice-emphasizing that clinicians have to
use their judgment and scientific training to interpret and
integrate guidelines).
8. The value of EBP to clients.
*Source: Gray, J.A.M.(2001). Evidence-based medicine for professionals. In A. Edwards
and G. Elwyn (Eds.), Evidence-based patient choice: Inevitable or impossible?
(pp. 19-33). New York: Oxford.
Gambrill (2005)
The Key Choice: How to view evidencebased practice
•
•
Whether to draw on the broad philosophy and
evolving process of EBP described by its originators
(e.g. Sackett et al, 1997, 2000) as a way to handle the
uncertainty in making decisions in an informed,
honest manner (sharing ignorance as well as
knowledge).
This choice has implications for researchers and
educators as well as for clients, clinicians and
administrators. It will affect all of the following
choices
Gambrill (2005)
Choices, cont.
1. Who will select the questions on which research
efforts are focused?
–
–
–
An elite? (a state or national board)
Administrators?
Clients and line staff (They know what questions arise
most often in everyday practice; lets gather this
information).
2. What criteria will be used to select services?
–
–
–
–
–
–
Gambrill (2005)
Consensus (e.g. opinions of a state advisory body)
Popularity
What is available
Evidentiary (shown via critical tests to do more good than harm)
Cost/benefit ratio
Ethical concerns
Choices, cont.
3. How transparent (honest) to be regarding the
evidentiary status of services, (for example
acknowledging lack of evidence for services
offered), with clients, staff, funding bodies,
consumer advocates.
Whether to involve clients as informed
participants; clearly describe the evidentiary status
of services provided including those offered by
agencies from which services are purchased.
Gambrill (2005)
What Hierarchy of Evidence To Use:
The one suggested in the Guide for Child Welfare
Administrators: (GCWA)
l. Well-supported, proven efficacious practice
2 Supported and probable efficacious practice.
3. Supported and acceptable practice.
4. Promising and acceptable practice.
5. Innovative or novel practice.
6. Experimental or concerning practice.
Gambrill (2005)
Concerns with the GCWA hierarchy
1. Use of justificationary language that encourages
confirmation biases and wishful thinking:
• Use of the word ‘proven’
• Repeated use of words such as ‘support’
• The word ‘harm’ is not mentioned.
2. Hides the fact that most services are of unknown
effectiveness.
3. Use of vague terms such as ‘probable’, ‘efficacious’,
4. In Appendix A we find that ‘clinical-anecdotal
literature’ and ‘generally accepted in clinical
practice’ are included as indicators of level 1
evidence.
Gambrill (2005)
A More Transparent Hierarchy (Gray, 2000)
1. Services that have been critically tested and
found to help clients.
2. Services that have been critically tested and
found to harm people.
3. Service methods are of unknown effectiveness
and are not in a research study or are in poor
quality research.
4. Services of unknown effect in a good quality
research program.
Gambrill (2005)
Choices, cont.
4. How rigorous to be in assessing the accuracy of
claims.
Consider differences between systematic and narrative
(traditional) reviews. In a systematic review:
• There is a search for all literature related to a question,
in all languages, in both published and unpublished
sources, including hand searches of journals.
• The search process is clearly described including the
databases reviewed. Authors describe where they searched,
how they searched.
• Rigorous criteria are used to appraise what is found and
these are clearly described. (See Cochrane and Campbell
Collaboration reviews).
Gambrill (2005)
Choices, cont. (systematic reviews)
Systematic reviews differ greatly from incomplete, nonrigorous, non-transparent reviews. Traditional narrative
reviews do not control for as many biases and thus
overestimate positive effects of services. They are
misleading in their conclusions. Compare claims of
effectiveness made by the developers of Multisystemic
Therapy and conclusions of Littell’s (2005) in Children and
Youth Services Review.
Gambrill (2005)
Choices, cont.
5. Whether to blow the whistle on pseudoscience,
quackery, fraud and related propaganda tactics
such as bogus claims of what is done and ‘what is
known’
Consider this: “The practice of child welfare has long
been based on a strong…” (Guide for Child Welfare
Administrators on Evidence-based Practice ,2005. p. 5)
This is highly misleading if by ‘strong’ we mean based on
high quality research and quality of services offered.
Consider the poor quality of parent training offered
(Gambrill and Goldman, 2005). Bogus claims are not
benign. They have resulted in harming in the name of
helping and interfere with further exploration; they
mislead rather than inform.
Gambrill (2005)
Choices, cont.
6. Whether to purchase services from agencies that
offer ineffective or harmful services. Whether to:
• Stop using ineffective and harmful services
• Start using effective services
7. How systemic to be.
• Whether to provide the resources and training needed for
staff to maximize the quality of services offered such as
access to needed databases/knowledge manager
• Whether to help all staff to acquire skills in evidence-based
practice as well as effective self development skills
drawing on evidence informed educational formats (See
critiques of traditional continuing education formats such
as
Thomson
O’Brien
et
al,
2003,
Cochrane
Library)
Gambrill (2005)
Choices, cont.
What style of EBP to use:Will all staff be trained
in all five steps of EBP:
1.
2.
3.
4.
5.
Posing well structured questions
Searching efficiently and effectively
Critically appraising related research
Integrating relevant information
Evaluating what happens?
– Or, will staff depend on others for steps 2 and 3?
– The process of EBP requires all staff to be skilled
in carrying out steps 1,4 and 5
Gambrill (2005)
Choices, cont. (How systemic to be)
– Whether to implement needed organizational
changes; e.g. see Evidence-based Health Care
(Gray, 2001):
• Hire knowledge managers (see Gray, 1998)
• Arrange access to relevant databases
• Set up a system for identifying errors so avoidable
ones can be minimized and educate all staff that
errors are typically systemic in cause
• Collect questions that arise in everyday practice from
line staff and clients. High frequency questions
should guide research efforts (e.g. systematic
reviews, new studies)
Gambrill (2005)
Choices, cont. (How systemic to be)
• Arrange facilitating incentive systems (e.g.
reward staff for blowing the whistle on harmful
and ineffective practices)
• Administrators and supervisors model critical
appraisal of claims; they raise questions
regarding current practices and policies and
welcome such questions from others; they avoid
propaganda strategies such as ad hominem
arguments
Gambrill (2005)
Questionable excuses for not taking a systemic view
• Critical appraisal and search skills are too hard for you
and your staff to learn
Not true; learning key questions to raise regarding
different kinds of research including research reviews is
easy, especially with the help of user friendly books such
as How to Read a Paper (Greenhalgh, 2001). Don’t let
the researchers and academics fool you; learning such
skills is not that difficult. Len Gibbs teaches them to
undergraduate students.
Gambrill (2005)
Questionable excuses, cont.
Regarding research reviews
• Is the question addressed clear and relevant?
• Do the authors clearly describe how they searched, where they
searched, and what criteria they used to appraise studies?
• Was a thorough search conducted using relevant databases?
• Did the search cover unpublished as well as published work?
• Were rigorous criteria used to review research?
For effectiveness questions:
•
•
•
•
•
•
Was the sample size adequate?
Was there a comparison group?
Were subjects randomly distributed to different groups?
Were ratings of outcomes blind?
Was there a follow up period?
Was there an intention to treat analysis?
Gambrill (2005)
You can:
– Keep a copy of Carl Sagan’s Baloney Detection
Kit available
– Be familiar with Hugh Rank’s cues to propaganda
pitches: Hi, Trust me, You need, Hurry, Buy
– Read Thouless’ Straight and Crooked Thinking
(1974) describing informal fallacies such as
appeal to popularity and emotional reasoning
Gambrill (2005)
Questionable excuses, cont.
• We researchers and academics don’t have time to
prepare systematic reviews
Masters degree students in the evidence based social
work program at Oxford complete Cochrane Reviews
in one year. True, many will take longer and require
considerable resources. But others will not.
• We don’t need a systemic approach
Services are unlikely to improve in quality in a
fragmented approach to change.
Gambrill (2005)
Indicators of choices made
• Increase in the percentage of systematic compared to
narrative reviews prepared related to questions of
direct concern to line staff and clients
• Increase in the percentage of clients involved as
informed participants
• Increase in the percentage of line staff, clients and
administrators who have acquired skills in posing
well-structured questions related to decisions that
must be made and in critically appraising research.
Gambrill (2005)
Indicators of choices made, cont.
• Increase in the percentage of all involved parties who
can spot human service propaganda such as inflated
claims about “what we know”
• Increase in the percentage of all involved parties who
ask probing questions regarding the effectiveness of
services and who avoid propaganda methods in related
discussions (such as distorting the views of others)
• Increase in the percentage of services purchased that
have been critically tested and found to be effective
• Decrease in the use of harmful and ineffective services
Gambrill (2005)
Obstacles (examples)
Arrogance
•
Preference for authority-based practices and policies
(examples):
• A reluctance to be transparent
• Inflated claims of effectiveness
• Use of the term “evidence-based” to refer to
business-as-usual
Prevalence of pseudoscience and propaganda and
underestimating vulnerability to their influence (e.g. see
Science and Pseudoscience in Clinical Psychology,
2003).
Gambrill (2005)
Obstacles, cont.
A justification approach to knowledge in which we
search for data to confirm our views.(Use of word
‘proven’.)
The symbiotic relationship between clients’ wishes
to be helped and professionals’ desire to help.
Gambrill (2005)
In Summary
The philosophy and practice of EBP as described
by its originators has implications for:
• Professional education
– Formats used and skills emphasized.
• Research
– Who decides on questions pursued
– How research is conducted and reported, for
example rigorous (likely to answer questions
posed), clear descriptions of limitations, accurate
rather than inflated claims of “what we know”,
involvement of consumers in critiquing research.
Gambrill (2005)
In Summary, cont.
•
Practice
–
–
–
–
–
Gambrill(2005)
Clients are involved as informed participants
regarding the evidentiary status of services
Staff are accurately informed regarding the
evidentiary status of services
Staff are skilled in carrying out the process of EBP
(e.g. posing well structured questions that help them
to draw on relevant research findings)
Staff evaluate the effects of their services
Services purchased from agencies have been shown
via critical tests to help clients
In Summary, cont.
• Administration and management
– Needed tools are provided (such as access to relevant
databases)
– Needed training is provided (e.g. in posing wellstructured questions)
– Needed feedback is provided (including supervisory)
to maintain and enhance skills
– A system is in place to harvest errors that occur so
avoidable ones can be minimized
– A user friendly complaint system is in effect and data
gathered are used to improve services
– Incentive systems are in place that support EBP
Gambrill(2005)
The Road Ahead
• California has an opportunity to take a leadership
role:
– Involve line staff and clients in identifying vital
questions and outcomes and involving them as
informed participants.
– Prepare systematic reviews related to key questions
of direct concern to line staff and clients using
Cochrane and Campbell protocols.
– Avoid human service propaganda such as bogus
claims of effectiveness.
– Create a workforce which is skilled in the process
of EBP.
Gambrill (2005)
The Road Ahead, cont.
– Create organizational cultures that support
EBP (e.g. provide access to needed databases)
– Maximize opportunities to honor ethical
obligations
• To help clients
• To avoid harm
• To involve clients as informed participants
• To maximize self-determination/autonomy
Gambrill (2005)
The Road Ahead, cont.
• Many challenges, especially a preference
for authority-based practice (The “trust me”
approach)
• Many exciting advances, especially in
options for integrating research and practice
at the line staff level and honoring ethical
obligations to clients.
Gambrill (2005)