Transcript Etched01” Title Slide

Slide 1
© 2003 By Default!
Development of an
Automated Nephrotoxicity
Pharmacosurveillance
System
Michael E. Matheny, MD MS MPH
Division of General Internal Medicine
Vanderbilt University, Nashville, TN
Geriatrics Research, Education & Clinical Care
Veteran’s Administration, Nashville, TN
A Free sample background from www.powerpointbackgrounds.com
Slide 2
© 2003 By Default!
Research Plan

Background & Significance

Objectives

Work Accomplished

Work Proposed

Training Activities & Objectives
A Free sample background from www.powerpointbackgrounds.com
Slide 3
© 2003 By Default!
Background
Surveillance Rationale

Phase 3 Trials insufficient to ensure
adequate safety of medications and devices
– Low frequency events are not detected
– Protected populations (pregnant women,
children) and more ill populations not
represented
– Complications delayed by a number of years
cannot be detected
A Free sample background from www.powerpointbackgrounds.com
Slide 4
© 2003 By Default!
Background
Current FDA Post-Marketing Surveillance

Combination of mandatory and voluntary
adverse event reporting
– Mandatory reporting by manufacturers and health
facilities
– Voluntary MedWatch / MAUDE reports by providers and
patients
2004 Drug-Related Adverse Event Reports
Total
422,889
Manufacturer & Facility Reports
401,396
MedWatch
21,493
A Free sample background from www.powerpointbackgrounds.com
Slide 5
© 2003 By Default!
Background
Current Post-Marketing Surveillance

‘Phase 4’ Trials
– Poor Compliance
• As of March 2006 report, 797 of 1231 (65%) agreedupon trials had yet to be started
– Barriers
• Lack of manufacturer incentives
– Expensive
– Drug already on the market
• Lack of regulatory enforcement
A Free sample background from www.powerpointbackgrounds.com
Slide 6
© 2003 By Default!
Background
FDA Response

Legislation

Complementary Data Sources
– Incorporate Phase 4 Trial costs into Approval Process
– Increase quality of adverse event reporting
– Commissioned IOM report “The Future of Drug Safety”
– Public Hearing in 2006

Aggregate Existing Clinical Registry Data
 Promote Use of Routine Medical Data
A Free sample background from www.powerpointbackgrounds.com
Slide 7
© 2003 By Default!
Background
Transition

Transition slide from surveillance to domain
A Free sample background from www.powerpointbackgrounds.com
Slide 8
© 2003 By Default!
Background
Acute Kidney Injury Statistics

1-7% of all hospital admissions

5-20% during an ICU stay

Crude Mortality Estimates

National trend of increasing incidence
– ~15% isolated AKI
– ~25% general inpatient
– ~40% in ICU
A Free sample background from www.powerpointbackgrounds.com
Slide 9
© 2003 By Default!
Background
Causes of Acute Kidney Injury

Acute Medical Conditions
– Rhabdomyolysis: 15-20%
– Sepsis: 40-50%
– Major Surgery: 10-20%
– Congestive Heart Failure: 20%
– Myocardial Infarction
– Any condition that generates a real or apparent
decreased intravascular volume
A Free sample background from www.powerpointbackgrounds.com
Slide 10
© 2003 By Default!
Background
Causes of Acute Kidney Injury

Medications
– Nephrotoxic medications cause 15-25% of AKI
– Aminoglycosides: 8-26%
• Single Daily Dose: ↓ risk
– Amphotericin: 50-80%
• Lipid Formulation: ↓ risk
– NSAID
– ACE Inhibitor
– Immunosuppressive Agents

Radiocontrast Dye
– Separate category, cause 10-15% of AKI
A Free sample background from www.powerpointbackgrounds.com
Slide 11
© 2003 By Default!
Objectives

Validation of an Automated Data Collection
System for Acute Kidney Injury Monitoring

Retrospective Evaluation of Acute Kidney
Injury among Hospitalized Veterans

Pilot an Acute Kidney Injury Automated
Surveillance System among Hospitalized
Veterans
A Free sample background from www.powerpointbackgrounds.com
Slide 12
© 2003 By Default!
Prior Work
A Free sample background from www.powerpointbackgrounds.com
Slide 13
© 2003 By Default!
Methodology & Application

Adaptation of Statistical Process Control Monitoring Methodologies
 Design and Implementation of a Web-Based Monitoring Application
(DELTA)
Matheny ME, Ohno-Machado L, Resnic FS. Monitoring Device Safety in
Interventional Cardiology. J Am Med Inform Assoc. 2006;13(2):180-7.
A Free sample background from www.powerpointbackgrounds.com
Slide 14
© 2003 By Default!
Methodology Validation
TIMI Randomized Trials
Method
SPC
BUS
LR-SPC
TP
8
0
8
FP
4
0
2
FN
0
8
0
TN
23
27
25
Sensitivity
100%
0%
100%
Specificity
85%
100%
93%
Matheny ME, Morrow DA, Ohno-Machado L, Cannon CP, Sabatine MS, Resnic FS. Med
Decis Making. 2007. Submitted
A Free sample background from www.powerpointbackgrounds.com
Slide 15
© 2003 By Default!
SPC vs BUS Sensitivity Analysis
Simulated PCI Data
20
15
# Alerts
10
5
61
58
55
52
50
Observed 47
Rate (% )
44
1787
1787
851
851
405
405
193
193
92
92
42
41
42
0
Matheny ME. Deelopment of Statistical Methodologies and Risk Models to Perform Real-Time
Safety Monitoring in Interventional Cardiology. [Master of Science Thesis] Cambridge:
Massachusetts
Institute of Technology; 2006.
A Free sample
background from www.powerpointbackgrounds.com
Slide 16
© 2003 By Default!
Clinical Event Detection
Vascular Closure Device – Retroperitoneal Hemorrhage


After manual review, access above inguinal ligament was
significant and VCD no longer.
Implemented Fellow Access Education Intervention
Matheny ME. Arora N, Ohno-Machado L, Resnic FS. Rare Adverse Event Monitoring of
Medical Devices with the Use of an Automated Surveillance Tool. AMIA Annu Symp Proc
2007.
Accepted
A Free sample
background
from www.powerpointbackgrounds.com
Slide 17
© 2003 By Default!
Process of Care Evaluation (SPRT)
Post-Procedural Mortality Among 18 BWH Operators


Found 1 clear Outlier, after case review, most were
“compassionate care”
Institution was less than 1.5 OR of national expectations
Matheny ME, Ohno-Machado L, Resnic FS. Risk-Adjusted SPRT Control Chart Methods for
Monitoring Operator and Institutional Mortality Rates in Interventional Cardiology. Am
J.from
Accepted
A Free sampleHeart
background
www.powerpointbackgrounds.com
Slide 18
© 2003 By Default!
Other Relevant Prior Work
Risk Prediction Modeling

LR External Validation and Model Development
(PCI Mortality)
• Matheny ME, Ohno-Machado L, Resnic FS. J Biomed Inform
2005;38:367-375

Evaluation of Support Vector Machine Modeling for
use in PCI Clinical Registry Data
• Matheny ME, Resnic FS, Arora N, Ohno-Machado L. J. Biomed
Inform. Accepted

Literature Review - Development and Evaluation of
Critical Care Mortality Models
• Ohno-Machado L, Resnic FS, Matheny ME. Prognosis in Clinical Care.
Annu Rev Biomed Eng. 2006;8:567-99

Book Chapter - Statistical and Machine Learning
Risk Predication Modeling
• Matheny ME, Ohno-Machado L. Clinical Decision Support: The
Road Ahead (Ed Greenes RA) Elsevier 2006.
A Free sample background from www.powerpointbackgrounds.com
Slide 19
© 2003 By Default!
Proposed Work
A Free sample background from www.powerpointbackgrounds.com
Slide 20
© 2003 By Default!
Conceptual Model
A Free sample background from www.powerpointbackgrounds.com
Slide 21
© 2003 By Default!
Validation of an Automated Data Collection
System for Acute Kidney Injury

Hypotheses:
– Clinical data that does not require free text NLP,
such as medication orders and laboratory tests,
will achieve at least 85% specificity and
sensitivity.
– Clinical data that requires free text NLP, such as
clinical diagnoses, will achieve at least 75%
specificity and sensitivity.
A Free sample background from www.powerpointbackgrounds.com
Slide 22
© 2003 By Default!
Development of ADCS
Data Sources

All records in 2005-2006 for patients
hospitalized in Nashville TVHS during 2006
A Free sample background from www.powerpointbackgrounds.com
Slide 23
© 2003 By Default!
Data Element Identification

Literature Review of AKI Risk Factors

General Patient Demographics

Manual chart review from randomly selected
Nashville TVHS Patients hospitalized during
2006 to identify locations and data quality for
each identified data element
– univariate and multivaraite risk factors
A Free sample background from www.powerpointbackgrounds.com
Slide 24
© 2003 By Default!
Data Element Selection

Complete capture of all relevant variables
may be infeasible from routine clinical data.

Review of data element list with location and
quality data with mentorship team

Elements will be selected based on a
combination of
– strength of association with AKI
– data collection quality
– algorithm complexity required for data extraction
A Free sample background from www.powerpointbackgrounds.com
Slide 25
© 2003 By Default!
Clinical Data Rule Development

Criteria must be developed for each clinical diagnosis

Example: status post nephrectomy

Example: systemic inflammatory response syndrome
– Identification: Direct documentation
– Possible locations: operative note, problem list, outpatient progress
note, inpatient progress note, inpatient discharge summary, or CT
scan of the abdomen.
– Requires: free text processing of multiple sources, could use Perl
word matching instead of concept-based indexing
– Identification: Direct documentation or by clinical criteria
– Clinical Criteria: abnormal HR, Temp, RR or spO2, or WBC.
– Locations of direct mention: problem list, inpatient progress note,
inpatient discharge summary
– Requires: use of inpatient lab and vital sign data AND/OR
processed free text notes
A Free sample background from www.powerpointbackgrounds.com
Slide 26
© 2003 By Default!
Primary Outcome

Acute Kidney Injury by RIFLE Criteria
– Risk
• Creatinine x 1.5 above baseline
• Or urine output < 0.5 ml/kg/hr x 6 hours
– Injury
• Creatinine x 2 above baseline
• Or urine output < 0.5 ml/kg/hour x 12 hours
– Failure
• Creatinine x 3 above baseline
• Or anuria or < 0.3 ml/kg/hour x 12 hours
– UOP not well captured, that criteria frequently ignored in
the literature
– Elevation needs to last at least 24 hours
A Free sample background from www.powerpointbackgrounds.com
Slide 27
© 2003 By Default!
Rule Validation

An annotated VA corpus with AKI-related data
elements does not exist, and must be created

Power Calculations:

Requires manual review of 400 charts for:
– Assumption: incididence of clinical data range from 0.2 to 0.8
– Alpha=beta=0.05
– One sided test of single proportion for diagnostic test sample (Flahault)
• 85% sensitivity/specificity with lower CI of 70%
• 75% sensitivity/specificity with lower CI of 58%

60% training sample, 40% testing sample

Random sampling from Nashville TVHS ’06-’07
A Free sample background from www.powerpointbackgrounds.com
Slide 28
© 2003 By Default!
Initial Estimate of Data Source Needs







Demographics
Labs
Inpatient and outpatient Orders
Vital sign data (inpatient/outpatient)
Medications
Problem List
Physician authored Free Text Notes: outpatient and
inpatient

* Radio-contrast Dye: If not in orders, will need
radiology ordering, if not there, will need radiology
free text
A Free sample background from www.powerpointbackgrounds.com
Slide 29
© 2003 By Default!
Retrospective Evaluation of Acute Kidney
Injury among Hospitalized VISN9 Veterans

Hypotheses:
– Concurrent use of diuretics, NSAIDs, ace inhibitors
and/or calcium-channel blockers, which are known to
cause pre-renal acute kidney injury, will result in
significant synergistic renal injury.
– Use of loop diuretics among hospitalized patients with
acute kidney injury will be associated with higher rates of
incomplete recovery of renal function.
– Variation in the risk-adjusted incidence of acute kidney
injury due to the process of clinical care can be detected
on both an institutional and medical specialty level
between VISN9 hospitals
A Free sample background from www.powerpointbackgrounds.com
Slide 30
© 2003 By Default!
Retrospective Evaluation
Data Sources

Retrospective Patient population: All
VISN9 patients hospitalized during 2006 –
2008

Will need 2005 (maybe 2004) patient data
for past medical history and baseline
creatinines in order to analyze
hospitalizations starting in 2006
A Free sample background from www.powerpointbackgrounds.com
Slide 31
© 2003 By Default!
Synergistic Pre-Renal Acute Kidney
Injury

Outcomes of Interest: Pair-wise interaction between
medications known to cause pre-renal AKI

Populations: 2 separate populations

SAS genmod - evaluate AKI (Risk, Injury, Failure)
– Admitted to hospital with AKI
– Develop AKI during hospital stay (>48 hours after admission)
– For the interaction terms of
•
•
•
•
•
•

loop diuretics * CCB
loop diuretics * ACE
loop diuretics * NSAIDS
CCB * ACE
CCB * NSAID
ACE * NSAID
after full risk adjustment of these factors:
• Age, gender, race, insurance status, medical co-morbidities, acute
clinical conditions, co-existing medications
A Free sample background from www.powerpointbackgrounds.com
Slide 32
© 2003 By Default!
Loop Diuretic Use Among AKI Patients

Outcomes of Interest: >3 month baseline
creatinine after inpatient AKI

Evaluate the association of failure to return to prehospitalization baseline and use of loop diuretics
after the diagnosis of AKI in the hospital

after full risk adjustment of these factors:
• Age, gender, race, insurance status, medical co-morbidities,
acute clinical conditions, co-existing medications

Additional Data Requirements: Additional time
required to allow follow-up creatinine monitoring
post-discharge
A Free sample background from www.powerpointbackgrounds.com
Slide 33
© 2003 By Default!
Evaluation of Variations in Clinical Care

Develop multivariable risk prediction model
for development of AKI among VISN 9
patients (separate manuscript)

Evaluate risk-adjusted incidence of AKI by
institution and clinical specialty to determine
if outliers exist using VISN9 (overall) data as
the baseline (uses above risk prediction
model).
A Free sample background from www.powerpointbackgrounds.com
Slide 34
© 2003 By Default!
Retrospective Evaluation of Acute Kidney
Injury among Hospitalized VISN9 Veterans

Knowledge Discovery (not hypothesis driven):
– Characterize the risk for acute kidney injury with the use
of the following medications:
•
•
•
•
•
•
NSAIDs
ACE Inhibitors
ARBs
Radiocontrast Dye
Aminoglycosides
Amphotericin B
– after full risk adjustment of these factors:
• Age, gender, race, insurance status, medical co-morbidities,
acute clinical conditions, co-existing medications
A Free sample background from www.powerpointbackgrounds.com
Slide 35
© 2003 By Default!
Pilot Evaluation of an Inpatient Acute Kidney
Injury Surveillance System

Hypotheses
– The methodology and application will detect a
simulated AKI event rate elevation among
retrospective routinely collected clinical data.
– A prospective automated outcomes surveillance
system evaluating acute kidney injury among
hospitalized veterans will allow discovery of
medication-related nephrotoxicity when
pharmacy formularies are changed or new
medications are introduced.
A Free sample background from www.powerpointbackgrounds.com
Slide 36
© 2003 By Default!
Surveillance System
Simulated AKI event rate elevation

Use the retrospective ’06-’08 VISN9 data, and
simulate an AKI event rate elevation in a
medication not known to be nephrotoxic with a
simulated outbreak algorithm frequently used in
biosurveillance
– Allows a sensitivity analysis to detect how significant
the elevation must be (how many cases in what
period of time) before the system will alert.
– Good prelim data for a merit application
A Free sample background from www.powerpointbackgrounds.com
Slide 37
© 2003 By Default!
Prospective Pilot of an Inpatient Acute Kidney
Injury Surveillance System

Having some trouble with this hypothesis.

What I really want to do is to setup the
prospective system after doing the simulated
detection, but there’s no way to know what, if
anything, it will detect as time goes no.

How to frame that?
A Free sample background from www.powerpointbackgrounds.com
Slide 38
© 2003 By Default!
Training Plans
Year 1

Formal Classes

Seminar

Ongoing Funded Grants

Conferences

Journal Club
– Database Development
– Perl course / regular expression parsing
– updates in acute kidney injury
– POEM (Nashville, TN) - Attend grant meetings dealing with clinical
concept tagging project
– DELTA-MASSDAQ (Boston, MA) – Attend grant meetings for
statistical and application development
– AMIA
– NKF
– VA HSR&D
A Free sample background from www.powerpointbackgrounds.com
Slide 39
© 2003 By Default!
Training Plans
Year 2

Formal Classes

Ongoing Funded Grants

Conferences

Journal Club
– Statistical Process Control
– Methods for Confounding Adjustment in Prospective Cohorts
– POEM (Nashville, TN) - Attend grant meetings dealing with clinical
concept tagging project
– DELTA-MASSDAQ (Boston, MA) – Attend grant meetings for
statistical and application development
– AMIA
– NKF
– VA HSR&D
A Free sample background from www.powerpointbackgrounds.com
Slide 40
© 2003 By Default!
Training Plans
Year 3

Seminar

Conferences

Journal Club
A Free sample background from www.powerpointbackgrounds.com
– Grant Writing
– AMIA
– NKF
– VA HSR&D
Slide 41
© 2003 By Default!
Training Plans
Year 4

Conferences

Journal Club

Submit a HSR&D IIR Merit Grant Application
– AMIA
– NKF
– VA HSR&D
A Free sample background from www.powerpointbackgrounds.com
Slide 42
© 2003 By Default!
The End
Michael Matheny, MD MS MPH
[email protected]
Tennessee Valley Medical Center - Nashville
Geriatric Research, Education and Clinical Care
Room 4-B110
1310 24th Ave. S.
Nashville, TN 37212
A Free sample background from www.powerpointbackgrounds.com