Transcript AHRQ_06_poster

Use of REMIND Artificial Intelligence Software for Rapid Assessment of Adherence
to Disease Specific Management Guidelines in Acute Coronary Syndromes
Ali F. Sonel, MD, C. Bernie Good, MD MPH, Harsha Rao, MD, Alanna Macioce, BS, Lauren J. Wall, BS, Radu Stefan Niculescu, PhD,
Sahtyakama Sandilya, PhD, Phan Giang, PhD, Sriram Krishnan, PhD, Prasad Aloni, MS, MBA, Bharat Rao, PhD
ABSTRACT
Introduction: Manual extraction of data for Quality Improvement is tedious,
requiring significant individual training and careful attention to the HIPAA
Privacy Rule. Automated chart abstraction is an alternative approach that saves
time and costs. We compared manual chart abstraction from an electronic
medical record (VA CPRS EMR System) to automated extraction using the
REMIND artificial intelligence software in 327 consecutive patients admitted
with unstable angina or non-ST elevation myocardial infarction.
Methods: All patient features required by ACC/AHA guidelines for determining
eligibility for class I recommendations to use aspirin, beta-blockers, heparin,
glycoprotein IIb/IIIa receptor antagonists, and ACE inhibitors were extracted by
both methods. Manual extraction was carried out by well-trained, qualified chart
abstractors with prior experience in manual chart abstraction. When both
extraction results were identical, the result was assumed correct. Disagreements
were manually adjudicated based on pre-determined definitions.
Results: Manual extraction and data entry required 176 hours compared to 4
hours using the Siemens REMIND software. A total of 5232 data elements were
identified, with agreement in 4385 (84%) and disagreement in 847 (16%),
involving 2.5-35% of patients for various parameters. REMIND was found to be
correct in 642 disagreements (76%) and manual extraction was correct in the
remaining 24% (205). Based on adjudication, REMIND identified adherence
compared very favorably to manual extracted as well as adjudicated guideline
adherence for aspirin (83% vs. 88% vs. 85%), beta blockers (78% vs.82% vs.
81%), heparin (53% vs. 51% vs. 54%), glycoprotein IIb/IIIa receptor antagonists
(35% vs. 38% vs. 40%) and ACE inhibitors (69% vs. 78% vs. 76%).
Conclusions: REMIND can assess disease specific management guideline
adherence at least as accurately as manual chart abstraction. Use of REMIND for
Quality Improvement and research can result in significant savings, better
resource utilization, and may improve data extraction quality.
BACKGROUND
•Research and quality improvement projects
involve large amounts of data collection through
review of medical records
•Manual data collection requires a significant
amount of training and is time consuming
•Automated data extraction methods could save
time and improve resource utilization
•Little is known about the accuracy of automated
systems for record extraction
SPECIFIC AIMS
•Compare the accuracy of data collection in a large
and complex medical record set using manual
extraction and REMIND automated extraction tool
•Compare the level of adherence to ACC/AHA
guideline recommendations for treatment of non-ST
elevation acute coronary syndromes (ACS) using
manual extraction and REMIND automated data
extraction tool
Center for Health Equity Research and Promotion, VA Pittsburgh Healthcare System and the Cardiovascular Institute, University of Pittsburgh Pittsburgh, PA, Siemens Medical Solutions, USA, Malvern, PA
METHODS
Patient Population
•327 patients admitted with high-risk non-ST-segment elevation
myocardial infarction were included in the study
Data Collection
•Records were extracted from VA CPRS Electronic Medical Record
System
•Manual extraction of predefined variables was performed by a trained
abstractor with expertise in ACS data abstraction for research purposes
•An artificial intelligence model developed by Siemens, the REMIND
automated data extraction tool, was used to extract the same information
electronically
•Medical information required to determine eligibility and the presence of
absence of contraindications for Class I treatment recommendations in
the ACC/AHA guidelines was collected for the following medications:
• Aspirin in all patients
• Beta-blockers in all patients
• Heparin in all patients
• Angiotensin converting enzyme (ACE) inhibitors or angiotensin
receptor blockers (ARB) in patients with diabetes mellitus, congestive
heart failure, left ventricular dysfunction or hypertension
• Glycoprotein IIb/IIIa receptor antagonists in patients in whom an
early invasive management strategy is planned
Data Analysis
•We compared the results of the two methods for accuracy
•When both extraction methods were in agreement, the result was
assumed to be correct. When extracted results differed, disagreements
were manually adjudicated based on pre-determined definitions, using the
source documents of each extraction method
• Accuracy was defined as the number of patients where there was
agreement with adjudication as to whether the patient was compliant or
not, divided by the total number of patients in the study
•Compliance is defined as the number of patients eligible and not
contraindicated to that medication, who actually received the medication,
divided by the number of patients who are eligible and have no
contraindication to that medication.
RESULTS
• Complete data extraction required 176 hours of manual extraction, compared to 4.5 hours
with REMIND automated extraction
Table 3: Accuracy* of Compliance Assessment with REMIND Compared to Manual Extraction
Table 1: Determination of Contraindications and Eligible Patients for Processes of Care
Patients with Contraindications for
Processes of Care (%)
ACCURACY (%) N=327
Ideal Patients for
Processes of Care (%)
TREATMENT
(N=327)
REMIND
MANUAL
ADJUDICATED
REMIND
MANUAL
ADJUDICATED
Aspirin
82 (25%)
77 (24%)
86 (26%)
245 (75%)
250 (77%)
241 (74%)
TREATMENT
REMIND
MANUAL
Aspirin
319 (98%)
314 (96%)
Beta Blockers
319 (98%)
316 (97%)
Heparin
315 (96%)
296 (91%)
ACE Inhibitors/ARB
300 (92%)
310 (95%)
300 (92%)
290 (89%)
Beta Blockers
229 (70%)
186 (57%)
233 (71%)
98 (30%)
141 (43%)
94 (29%)
Heparin
82 (25%)
77 (24%)
86 (26%)
245 (75%)
250 (77%)
241 (74%)
Glycoprotein IIb/IIIa Receptor
Antagonists
ACE
Inhibitors/ARB
142 (43%)
128 (39%)
152 (46%)
146 (45%)
125 (38%)
125 (38%)
*Accuracy defined as true positives plus true negatives divided by the total number of patients
Glycoprotein
IIb/IIIa Receptor
Antagonists
99 (30%)
93 (28%)
106 (32%)
103 (32%)
79 (24%)
86 (26%)
•REMIND can determine ACC/AHA guideline adherence for non-ST-elevation
acute coronary syndromes at least as accurately as manual chart abstraction.
Table 2: Assessment of Compliance with Guideline Recommended Therapies
COMPLIANCE BY
REMIND
EXTRACTION (%)
COMPLIANCE BY
MANUAL
EXTRACTION (%)
COMPLIANCE
FOLLOWING
ADJUDICATION (%)
Aspirin
202/245 (83%)
220/250 (88%)
206/241 (85%)
Beta Blockers
76/98 (78%)
116/141 (82%)
76/94 (81%)
Heparin
131/245 (53%)
127/250 (51%)
131/241 (54%)
ACE Inhibitors/ARB
101/146 (69%)
98/125 (78%)
95/125 (76%)
Glycoprotein IIb/IIIa
Receptor Antagonists
36/103 (35%)
30/79 (38%)
34/86 (40%)
TREATMENT
CONCLUSION
IMPLICATIONS
•Use of REMIND for quality improvement and research related applications in
facilities with electronic medical records can result in significant savings and
better resource utilization.
•Use of REMIND can enable evaluation of very large sets of medical
information that would otherwise be impractical by manual extraction