Transcript Pediatric Research Seminar September 27, 2007

Medical Record Reviews –
The Rules of the Road
David H. Rubin, MD
Department of Pediatrics
INTRODUCTION
• Any study that uses pre-recorded patient
focused data as the primary source of
information in a research study





Physician, nurses notes
Ambulance call reports
Diagnostic tests
Clinic, administrative, government records
Computerized databases
WHY SELECT THIS DESIGN?
• Allows research to address issues that cannot
be addressed with prospective studies







Effect of harmful exposures (no randomization
possible)
Effect of potentially beneficial exposures
Occurrence of rare events
Studies of patterns of disease or behavior
Quality assurance studies
Studies where cases may be shared (trauma
database)
Pilot studies for prospective studies
DATA QUALITY
• “Free form” quality of medical records may
increase missing and/or erroneous data
• Handwriting may be illegible or
uninterruptible
• May miss examining potential cases
• Computer vs paper records
• Data abstraction techniques require
standardization
MISSING OR CONFLICTING
DATA
• Missing information may lead to nonresponse
bias

Subjects with missing information may be very
different from subjects without any missing
information
• Missing values management


Case deletion – bias and reduced sample size are
problems
Case insertion – use what you have and compare
missing/non missing cases for differences
SAMPLE SIZE
• Usually determined based on the
summary measure and the size/width
of the confidence interval desired


An interval with a greater CI (eg 99% CI v
95% CI) is wider and more likely includes the
true population value
The width of the CI depends on sample size
SAMPLING
• Select all cases within a given time frame
• For nonconsecutive sampling it is best to choose
probability sampling


Provides equal opportunity for each eligible case to
be selected
Use random number generator
• Triage level
• Incidental sampling – choosing most easily
accessible cases
• Systematic sampling – choosing every xth case
RELIABILITY
• Very important
• Any differences in data extraction by 2 different
people?
• Kappa



Value ranges from -1 (perfect disagreement) to 1
(perfect agreement)
K = [observed agreement (%) – expected agreement
(%) / [100% - expected agreement (%)]
Try to achieve kappa of 0.6 or better (60%
agreement)
MINIMUM REQUIREMENTS FOR
MEDICAL RECORD REVIEWS
(Lowenstein, 2005)
1)
2)
3)
4)
Explicit protocols for case selection/exclusion
Abstractor training
Precise definitions of key variables
Use of standardized abstraction and coding
forms
5) Monitoring of abstractor performance
6) Blinding of abstractors to study hypothesis and
patient groups
7) Testing of interrater reliability
QUALITY OF MEDICAL RECORD
REVIEWS
(Badcock, 2005)
• Observational study of medical record reviews
published in several emergency medicine
journals
• 107 articles analyzed
• Clear aim reported in 93%
• Standard abstraction forms: 51%
• Interrater reliability: 25%
• Ethics approval: 68%
• Sample size/power: 10%
MEDICAL RECORD STUDIES IN
EM RESEARCH
(Worster, 2005)
• From 1993 -1998 assessed medical
record review studies in 6 EM journals
• 79 (14%) medical review studies in 563
original research articles
• Adherence to methodological criteria



Sampling method: 99%
Abstractor blinding to hypothesis: 4%
Interobserver agreement for the 12 criteria
ranged from 57% - 95%
ACCURACY IN HOSPITAL
DISCHARGE SUMMARIES
(Callen 2009)
• Retrospective analysis of 966 handwritten
and 842 electronically generated
discharge summaries in Australia
• 12.1% of handwritten and 13.3% of
electronic summaries contained errors
• Medication omission was biggest problem
• NO difference in training level
MISSING INFORMATION IN ED
CHARTS
(Richason 2009)
• Examined case reports in 4 ED journals
from 2000-2005 and used 11 reporting
standards
• 1,316 case reports identified
• Poor reporting of



Co-morbidities
Outcomes
Concurrent medications
METHODOLOGY FOR RETROSPECTIVE
REVIEWS IN CHILD PSYCHIATRY
•
•
•
•
Conceive question
Literature review
Proposal methods
Create data abstraction instrument and
manual
• Sample size
• Obtain IRB approval
• Pilot study
SAMPLE SIZE
(Gearing 2006)
• Estimate 10 charts per variable (Sackett,
1991)
• Others estimate 5-7 charts/variable
• Convenience sampling – select cases over
specific time period
• Quota sampling – predetermined number
sampled
• Systematic sampling – every “nth” case
chosen
PRACTICAL ISSUES
• Check all possible CPT codes for
diagnosis or procedure code


Febrile seizure may have been coded as
seizure
Gastroenteritis may have been coded as viral
syndrome
• Pilot your Data Abstraction Form
• Create detailed “Codebook” for your study

Especially critical if > 1 researcher on study
REFERENCES
• Worster A, Haines T. Advanced statistics: understanding
medical record review (MRR) studies. Acad Emerg Med
2004;11:187-192.
• Lowenstein SR. Medical record reviews in emergency
medicine: the blessing and the cure. Annals Emerg Med
April 2005;45(4):452-455.
• Babcock D et al. The quality of medical record review
studies in the international emergency medicine literature.
2005;45(4):444-447.
• Worster A. et al. Reassessing the methods of medical
record review studies 2005;45:448-451.
• Gearing et al. Methodology for Retrospective chart review
in child adolescent psychiatry. J Can Acad Child Adoles
Psychiatry 15:3:2006
REFERENCES
• Callen JJ, McIntosh. Accuracy of
medication in hospital discharge
summaries. Int J. Med Inform. 2009;Oct 1
• Richardson TP et al. Case reports
describing treatrments in the ED. BMC
Emerg Med.2009. June 15:9:10.