Transcript EVIDEM - BioMedCom

EVIDEM
A practical framework to
support healthcare decisionmaking,
assess quality of evidence, and
define explicit data needs
Evidence and Value: Impact on DEcisionMaking A practical framework applying Multi-Criteria Decision
Analysis to support healthcare decisionmaking
Mireille M Goetghebeur PhD, Monika Wagner PhD, Hanane Khoury PhD,
Randy Levitt PhD, Lonny J Erickson PhD, Donna Rindress PhD
BioMedCom Consultants inc
Montreal, Canada
BioMedCom
Background & Hypotheses
 Healthcare decisionmaking is a complex process that requires
simultaneous integration of numerous disparate types of evidence
and value judgments
 There is a need, nationally and internationally for a transparent
access to evidence and values on which decisions are based for
coverage/use of healthcare interventions.
 Ad-hoc assessment of a healthcare intervention without breakdown
into components may result in loss of information and possibly
biased valuation
 Healthcare decisionmaking could be facilitated by structuring
evidence and value judgments on which it is based into a practical
and transparent architecture
 Transparency will enhance understanding of healthcare decisions
BioMedCom
Objectives
 Breakdown components of healthcare decisionmaking into practical
instruments structuring and quantifying assessment of healthcare
interventions to facilitate decisionmaking
 Build an iceberg architecture to provide multiple layers of
transparent access to components of decisionmaking
Decision
Extrinsic value
Intrinsic value of treatment
Value Matrix
Multi Criteria Decision Analysis
(MCDA)
Ad hoc assessment
Burden of disease
Type of medical service
Improvement of efficacy
Cost-effectiveness
Budget impact
Synthesized evidence
Quality of evidence
For each component of Value Matrix
Fully referenced
Quality Matrix
International standards
Decisionmaking body requirements
Criteria of quality
Healthcare savings
Equity
Etc…
Evidence
Full text source documents (published, registries, proprietary)
 Ultimately, optimize health by best use of healthcare interventions
BioMedCom
Conceptual framework
Components of decisionmaking
Decisionmaking process
Scientific judgment
Value judgment
Intrinsic
value
of product
Quality of
of
evidence
Validity of efficacy trial
Number of patients
Low
Etc…
Improvement over existing
treatment
Efficacy
Small
None
Etc…
Lower
Extrinsic
value
of product
Type of population treated
(equity)
Vulnerable vs productive
patients
Low priority?
High
High priority?
Large
Major
 Based on international standards
 Comprehensive analyses can be
systematized
 Not highly dependent on evaluator
perspective
Higher
 Similar value system
 Requires value judgment
 Dependent on evaluator
perspective
No shared value system
Dependent on
region/institution priorities
1Tunis
SR. Reflections on science, judgment and value in evidence-based decision making: a conversation with David Eddy.
Health Affairs. 2007 (26): w500-w515;
2Eddy DM. Clinical decision-making: from theory to practice-Anatomy of a decision. JAMA 1990 (263):441-443.
BioMedCom
EVIDEM – What?
Practical iceberg architecture to support decisionmaking
Decision
Extrinsic value
Intrinsic value of treatment
Value Matrix
Multi Criteria Decision Analysis
(MCDA)
Synthesized evidence
Quality of evidence
Key info for each component of
Value Matrix
Fully referenced
Quality Matrix
Criteria of quality
International standards
Decisionmaking body requirements
Evidence
Full text source documents (published, registries, proprietary)
BioMedCom
BioMedCom
EVIDEM – Who?
Communication tool to connect stakeholders
T
r
a
n
s
p
a
r
e
n
t
c
o
m
m
u
n
i
c
a
t
i
o
n
t
o
o
l
Decision
Extrinsic value
Intrinsic value of treatment
Decisionmakers
Value Matrix
Multi Criteria Decision Analysis
(MCDA)
Synthesized evidence
Quality of evidence
Key info for each component of
Value Matrix
Fully referenced
Quality Matrix
Criteria of quality
International standards
Decisionmaking body requirements
Evidence
EVIDEM investigators
& experts
Data producers
Full text source documents (published, registries, proprietary)
BioMedCom
EVIDEM - Why?
Sharing of value judgments and structured multilevel access to data
More transparent and
understandable
Can be explored more easily
Limit obstruction of
thinking process
by large amount of data
Decision
Extrinsic value
Intrinsic value of treatment
Value Matrix
Multi Criteria Decision Analysis
(MCDA)
Transparent
multilayer
access to
evidence
Focus on value
Sharing of weights & scores
Synthesized evidence
Quality of evidence
Key info for each component of
Value Matrix
Fully referenced
Quality Matrix
Criteria of quality
International standards
Decisionmaking body requirements
Transparent
access
to quality of data
and methods
Evidence
Full text source documents (published, registries, proprietary)
BioMedCom
EVIDEM – How?
Integrated process – example “Improvement of efficacy”
Decision
Extrinsic value
Value Matrix (MCDA)
Value Matrix component T3: Improvement of medical service - efficacy
Cluster Components
T3
Improvement
of medical
0
service –
1
efficacy
2
3
Score
Scoring example
For information only
0: Lower efficacy than comparators
1: Same efficacy as comparators
2: Some improvement in efficacy
3: Major improvement in efficacy,
larger eligible population
Lower efficacy
Major improvement in efficacy
Quality Matrix: quality of clinical evidence
Synthesized clinical evidence
EFFICACY DATA
Metaanalysis/types of trials
Inclusion criteria:
Exclusion criteria:
Primary outcome
Treatment Y
0=no data, 1=major issues, 2=a number of issues 3= some non critical issues
Evidence assessed
Manufacturer dossier
Criteria of quality
Comp. 1
Comp. 2
Etc.
New treatment clinical data
Effectiveness data
Comparator product data
 Patient characteristics
 Subgroup analyses
 Patient disposition:
POPULATION OF PATIENTS ELIGIBLE FOR TREATMENTS
Adherence to
requirements
3
0
1
Component
Clinical data
References
Dossier and
literature
Relevance and
validity
2
0
2
Comprehensiveness
and accuracy
3
0
2
Questions
2
Comments
Score
How comprehensive is the
clinical program? Are results
across trials consistent?
Are standard elements
relevant and valid?
(COMPLETE INSTRUMENT
BELOW -1 per comparator)
Element
Comment
Target population relevant (age, gender, disease stage, comorbidities, inclusion
criteria/exclusion criteria etc)? Does it correspond to the actual population in which the
drug is envisioned to be used?
Evidence - Full text sources/links
Registries
Published
trials
MetaAnalyses
(Cochrane)
Proprietary
data
Does the choice of comparators reflect standard of care?
Is time horizon long enough to capture all meaningful differences in key outcomes
between the intervention and comparator?
Are outcomes selected relevant? Are assumptions for outcomes selection valid?
Is the design appropriate (sample size, randomization, blinding)?
Is the analysis appropriate? Are analyses comprehensive [ITT] with statistics?
Disaggregated data for clinical outcomes & safety? Patient disposition provided?
Are all standard categories of side effects reported?
Conclusions supported by results?
BioMedCom
EVIDEM tools – MCDA Value Matrix
15 components defining intrinsic value of treatment
Cluster
Components of value assessment (alphabetical by cluster)
Weight
1-5
Synthesized
information*
Score
0-3
Value
Quality of dossier and evidence
Q1
Adherence to requirements of decisionmaking body (dossier)
Q2
Comprehensiveness and accuracy of evidence presented (dossier)
Q3
Relevance and validity of evidence (literature and dossier)
Quality Matrix
Disease impact
D1
Disease severity (risk of death & disability, acuteness)
D2
Size of population affected by disease
Treatment
T1
Current clinical guidelines on treatment or products of same class
T2
Current treatment limitations
T3
Improvement of medical service –
efficacy
T4
Improvement of medical service – safety & tolerability
T5
Improvement of medical service – patient reported outcomes,
convenience & adherence
T6
Public health interest (prevention & risk reduction)
T7
Type of medical service (symptom relief, prolonging life, cure etc.)
Independent
of product
Structured
summary of
evidence for
treatment
Scale anchors
and guidelines
Value
score
(linear
model)
Economics
E1
Budget impact of reimbursing treatment on drug plans
E2
Cost-effectiveness of treatment
E3
Savings with treatment (impact on healthcare spending excluding
treatment cost)
Total
*Available information from public domain and manufacturer dossier
Aggregated value score (% of maximum score)
Components identified from literature review and current decisionmaking processes, and selected to fulfill MCDA guidelines on clustering,
completeness, redundancy, operationality & mutual independence (Baltussen R, Niessen L. Priority setting of health interventions: the need for
multi-criteria decision analysis. Cost Eff Resour Alloc. 2006;4:14.; National Economic Research Associated. Multi-criteria analysis manual 2005.
http://www.communities.gov.uk/pub/252/MulticriteriaanalysismanualPDF1380Kb_id1142252.pdf Accessed Jan2007)
BioMedCom
EVIDEM tools - Quality Matrix
12 types of evidence, 3 criteria of quality
Matrix component
Definition
Adherence to
requirements*
(dossier)
Completeness
& accuracy
(dossier)
Relevance &
validity
(literature &
dossier)
Score (0-3)
1
Disease information
2
3
4
Treatment patterns
Impact of treatment in
therapy
Epidemiology
5
Treatment characteristics
6
Clinical data
Efficacy and safety data from clinical
trials (published, unpublished, metaanalyses, reviews) and from documents
submitted to regulatory bodies
7
Patient reported outcomes
8
Effectiveness data
9
Comparator treatment data
10
Price
information/justification
Economic evaluation
Patient reported outcomes for treatment including quality of life,
satisfaction, convenience etc
Effectiveness data (naturalistic/real life trials, effectiveness
estimates, registry requirements, etc)
Data on efficacy, safety, patient reported outcomes, treatment
characteristics for comparator treatments, as applicable
Price of treatment and comparators; price justification
11
Disease description, disease progression/duration,
pathophysiology, clinical presentation, severity of disease etc
Treatment patterns & current practices, treatment guidelines
Expected role of treatment, impact on healthcare system, place of
treatment in therapy
Epidemiology of disease treated by intervention (prevalence,
incidence, trends, sub-population etc, as applicable) and risk
factors
Indication, pharmacology, pharmacokinetics, interactions, contraindications, warnings, dosing and administration & concomitant
therapies
Quality
score
Quality
score
Quality
score
Economic evaluation; impact of treatment on healthcare utilization
and associated costs; impact on society and associated cost, as
applicable
Drug plan budget impact analyses if treatment reimbursed
12
Budget impact
Total
Aggregated quality score (% of maximum score)
*Requirements established by the decisionmaking body ; Definition of effectiveness trial based on the AHRQ criteria
Structured based on criteria defining quality and on requirements from more than 20 decisionmaking bodies worldwide (INTERFACE database. 2000-2008.
www.biomedcom.org); Assessment for each component based on instruments derived from quality standards (e.g., CONSORT, CHEC, STROBE, Siegel et al,BioMedCom
Mauskopf et al) as well as guidelines and processes from decisionmaking bodies
Pilot study in Canadian context
Proof of concept using historical cases
 Historical cases: 10 medicines (cardiovascular diseases, endocrinology,
infectious diseases neurology, oncology, ophthalmology,) using data from
literature review and manufacturer dossiers submitted to the Canadian
Common Drug Review (CDR) and Québec Conseil du Médicament (CM)
 QM scores: EVIDEM investigators;
 VM weights and scores, and feedback on process: Canadian Value Panel
composed of representative stakeholders across Canada (decisionmakers,
specialists, generalists, nurses, pharmacists, health
economists/epidemiologists)
 Feasibility: Algorithm developed to operationalize each cell of the matrices;
applicable to any therapeutic areas and jurisdiction
 Practicality: 30 min on average to apply VM by stakeholders; about 250 hrs
to build the structured package of fully traceable information (quality,
synthesized format and full text access) and value scores
 Feedback from panelist: Value of EVIDEM in structuring evidence, assessing
strengths and weaknesses systematically, and sharing values and value
scores
 Limitations of pilot: limited access for panelists to underlying source data
(electronic interface to be developed); extrinsic components not covered
(to be explored)
BioMedCom
Pilot study - Value Matrix
Value assessment example for clusters 1 & 2
 Anchors defined from a societal perspective, i.e., optimizing health at
the societal level
 Score options translate to 0% (score of 0), 33% (score of 1), 66%
(score of 2) and 100% (score of 3)
Cluster
Components of value assessment
(alphabetical by cluster)
Scale anchors
% Max Value score – Mean (SD)
Quality of dossier and evidence
Q1
Adherence to requirements of
decisionmaking body
0=Low adherence
3=High adherence
56 (17)
Q2
Comprehensiveness and accuracy of
evidence presented
0=Many gaps / inaccurate
3=Comprehensive and accurate
72 (14)
Q3
Relevance and validity of evidence
0=Low relevance/validity
3=High relevance/validity
78 (17)
Disease impact
D1
Disease severity (risk of death & disability,
acuteness)
0=Not severe (minor inconvenience)
3=Very severe
100 (0)
D2
Size of population affected by disease
0=Extremely rare disease
3=Common disease
22 (27)
BioMedCom
Pilot study - Value Matrix
Value assessment example for cluster 3
Cluster
% Max Value score – Mean (SD)
Components of value assessment
(alphabetical by cluster)
Scale anchors
Treatment
T1
Current clinical guidelines on treatment or
products of same class*
0=Not recommended
3= Strong first-line recommendation
NA=not available
67 (21)
T2
Current treatment limitations
0=No or very minor limitations
3=Major limitations
89 (17)
T3
Improvement of medical service – efficacy
0=Lower efficacy than comparator treatments presented
3=Major improvement in efficacy vs comparator
treatments presented
72 (25)
T4
Improvement of medical service – safety &
tolerability
0=Lower safety / tolerability than comparator treatments
presented
3=Major improvement in safety / tolerability vs
comparator treatments presented
61 (14)
T5
Improvement of medical service – patient
reported outcomes (PRO), convenience &
adherence
0=Worse PRO / lower convenience / lower adherence vs
comparator treatments presented
3=Major improvement
50 (18)
T6
Public health interest (prevention & risk
reduction)
0=No risk reduction compared to no intervention
3=Major risk reduction
NA=not applicable
T7
Type of medical service (symptom relief,
prolonging life, cure etc.)
0=Minor service
3=Major service
NA†
72 (14)
BioMedCom
Pilot study - Value Matrix
Value assessment example for cluster 4
Cluster
Components of value assessment
(alphabetical by cluster)
Scale anchors
% Max Value score – Mean (SD)
Economics
E1
Budget impact of reimbursing
treatment on drug plans
0=Strong impact on drug plan budget (significant additional expenditures)
3=Substantial savings for drug plans
61 (25)
E2
Cost-effectiveness of treatment
0=Not cost-effective
3=Highly cost-effective
89 (17)
E3
Savings with treatment (impact on
healthcare spending excluding
treatment cost)
0=Substantial additional healthcare spending, excluding product cost
3=Substantial savings in healthcare spending, excluding product cost
67 (21)
BioMedCom
Potential applications
Transparent access to evidence and values
Collaboration axis
Explicit data needs
Define explicit needs of
decisionmakers
Establish reasonable
requirements
Application axis
Retrospective
Validate process
in various
jurisdictions
Explore context of
past decisions
Generate data on
quality of evidence
Application axis
Prospective
EVIDEM
framework
Adapt framework
to existing
decisionmaking
processes
Basis to develop
extrinsic value
components
Collaboration axis
Research planning
Planning tool for researchers and
intervention developers to meet
explicit needs
Develop methodology to generate
data tailored to critical data needs
BioMedCom
Future developments
Collaborative studies and iterative processes
to explore the value of EVIDEM in context
Interactive web based architecture
integrating evidence and value for various
healthcare interventions
Expected outcome of a systematized and
shareable approach for data access and value
assessment is to optimize resources,
decisions and health
BioMedCom
Acknowledgments
EVIDEM Canadian Value panel:
Jean-Francois Bussières BPharm MSc MBA FCSHP, Director, Pharmaceutical Practice Research Unit, Dept of Pharmacy, CHU
Sainte-Justine Research Center; Clinical Associate Professor, Faculty of Pharmacy, University of Montréal.
Benoit Cossette BPharm MSc, Pharmacist, Coordinator, Drug Disease Management Program, Fleurimont Hospital, Sherbrooke
University Hospital Centre.
Doug Coyle PhD, Associate Professor, Faculty of Medicine, Epidemiology and Community Medicine, University of Ottawa.
Cheri Deal MD PhD FRCPC, Associate Professor and Program Director, Endocrine Service, CHU Sainte-Justine Research Center,
University of Montréal.
Roland Grad MD CM MSc CCFP FCFP, Associate Professor, Department of Family Medicine, McGill University.
Christine Lee MD, FRCPC Assistant Professor, Director of the Microbiology Residency Program, Department of Pathology and
Molecular Medicine, McMaster University.
Mitchell Levine MD FRCPC FISPE, Professor, Director of Centre for Evaluation of Medicines Department of Clinical Epidemiology &
Biostatistics, Department of Medicine, McMaster University
Diane Lowden RN MSc, Clinical Nurse Specialist, Montreal Neurological Institute & Hospital, McGill University.
John Mancini MD FRCP, Professor/Program Director Continuing Medical Education, Department of Cardiology, University of British
Columbia.
Paul Oh MD FRCPC, Medical Director of the Cardiac Rehab and Secondary Prevention Program, Department of Medicine, Clinical
Pharmacology and Toxicology, University of Toronto.
Genevieve Tousignant N MScN, Clinical Nurse Specialist, Montreal Neurological Institute & Hospital, McGill University.
Wendy Ungar PhD, Senior Scientist, Child Health Evaluative Sciences, The Hospital for Sick Children; Associate Professor, Health
Policy, Management, and Evaluation, University of Toronto.
Marie-Claude Vanier BPharm MSc, Clinical Associate Professor, Faculty of Pharmacy, University of Montréal; Pharmacist UMF-GMF
Cité de la Santé de Laval.
Funded by an unrestricted research grant from Pfizer Canada
BioMedCom
EVIDEM
EVIDEM tools are open access
We welcome input and collaborations
Please contact the EVIDEM Group @
www.evidem.org
[email protected]
Thank you
BioMedCom