Transcript General Latent Variable Modeling Approaches to Measurement

Developing a Measure for
Recovery using Item Response
Theory
Frances M. Yang, Ph.D.
Instructor in Medicine, Assistant Scientist I
Harvard Medical School, Department of Medicine
Beth Israel Deaconess Medical Center,
Department of Internal Medicine
Institute for Aging Research, Hebrew SeniorLife
[email protected]
1
Collaborators &
Acknowledgements
Yih-Ing Hser
David Huang
Libo Li
Executive Committee Members
Richard N Jones, ScD
Doug Tommet, MS
UCLA CALDAR Emerging Investigator
Pilot Award
2
Overview
• Purpose
– We used Item Response Theory (IRT) to find the
“best” set of items to identify levels of recovery
from drug addictions
• Significance
– Recovery is difficult to measure
– Short, efficient, and sensitive recovery screening
tool is needed
• Innovative Approach
– Clinical and theoretical input from an
interdisciplinary team of experts.
– Modern methodology using IRT for balancing the
discrimination and severity levels of each item to
measure the underlying latent trait of recovery.
3
Consensus definition of recovery
that came out of the 2005
SAMHSA/CSAT
• “a process of change through which an
individual achieves abstinence and
improved health, wellness, and quality of
life.”
• This broad and vague definition is an
opportunity, as Hser and Anglin have
pointed out, to develop and operationalize
for research purposes.
4
Modern Measurement Methodology
• Item Response Theory
– 2-PL Model: Normal ogive assumption
• Differential Item Functioning
– Detecting measurement noninvariance in
items of a scale attributable to certain
characteristics
• Translational Research (T1)
– Bench to Bedside
– Identifying markers, tests, or items on the
basic level for clinical or research screening
6
Structure Research Algorithm for
Addiction Severity Index
Sections
Number of items
General Information
7 Items
Medical Status
11 Items
Employment/Support Status
24 items
Drug/Alcohol Use
27 items
Legal Status
30 items
Family History
3 items
Family/Social Relationships
26 items
Psychiatric Status
22 items
Total items
150 items
Total indicators
249 indicators
7
Recovery Assessment
Challenges
• Clinical:1
– Forms and protocols
– Single step, simple responses
– Screening, not diagnosing
– Tools/aids to assessing attention
– Collect symptoms not attributions
• Research:
– Develop a recovery screening tool that is:
• Short
• Simple
• Structured
• Accurate
– Demonstrate validity
8
Methods: Sample
• Treatment Utilization, and Effectiveness
(TUE) Study
• All respondents were recruited from the Los
Angeles area in 1992–1994. The settings
included:
– sexually transmitted disease (STD) clinics,
– hospital emergency rooms,
– and county jails
• Participants: N=865
9
Modified Delphi Method
•Expert Consensus Panel of 7 Members who
are experts in addictions research
•Identified domains based on items from the
Addiction Severity Index independently
•Agreement regarding narrowing of domains
for item response theory analysis
•Agreement was based on majority consensus
if four or more agreed.
Statistical Analysis
Psychometric Modeling Steps
1. Preprocessing:
•
•
•
Items were recoded into indicators (based on distributions
of responses and combining conditional items)
Multicollinearity checking
Items were removed if there was a lack of covariance with
other items.
2. Dimensionality Testing:
•
Parallel analysis
•
•
Exploratory factor analysis
Drop the indicators below 0.4 based on NIH Patient
Reported Outcomes Measurement Information
Systems
11
Statistical Analysis
Psychometric Modeling Steps
(Continued)
3. Dimensionality Assessment:
-Comparative Fit Indices > 0.95
-Root Mean Squared Error of
Approximation <0.05
4. Item response theory analysis
-Item parameter estimation
Software: Stata v. 11 and Mplus v. 6.0
12
Flow Diagram for Item Selection
STAGE
Item Pool from Source Instrument: Addiction Severity Index
I
A Priori Hypothesized Item Grouping
II
Health Indicator Set
Abstinence Indicator
Set
Inclusion of New Domains
III
Context
Environment
Family
History
Expert Consensus Panel
Support
network/
Family
Support
Citizenship Indicator
Set
Expert Consensus Panel
Social
Function
ing
Education/
Employment/
Income
Motivation
Abuse
Psychometric Data Analysis
Abstinence
IV
Citizenship
Mental
Health
Physical
Health
Selfsufficiency
Social
Relationships
Final Indicator Selection (Clinical Expert Panel)
V
13
Expert Consensus Form
14
Final Hypothesized Domains
Domains identified by experts from the ASI items
(# indicators):
• Abstinence (50 indicators)
• Citizenship (57 indicators)
• Social Relationships (9 indicators),
• Self-Sufficiency (24 items)
• Physical Health (33 items)
• Mental Health (15 items)
Sample Characteristics
(N=865)
Characteristic
Mean (SD) or
N (%)
Age (years old)
32 (8.9)
Female
294 (34%)
Race/Ethnicity
White (Not of Hispanic Origin)
169 (19.5%)
Black (Not of Hispanic Origin)
452 (52.3%)
American Indian/American Native
Asian or Pacific Islander
Mexican
14 (1.6%)
9 (1.0%)
175 (20.2%)
Puerto Rican
7 (0.8%)
Cuban
3 (0.4%)
16
Results: Recovery Item
Parameters
Abstinence:
Primary alcohol user
Alcohol use in lifetime
and used in the past 30
days
Alcohol substance is the
major problem
$ spent on alcohol
$ spent on drugs
Item
Item
Discrimination Difficulty
(a )
(b)
2.840
0.020
1.468
3.922
1.150
0.726
0.178
0.486 17
Item Characteristic Curves for
Indicators Assigned to Abstinence
Domain: Factor 1
Alcohol use in lifetime and used in
the past 30 days
Alcohol substance is the major
problem
EFA
CFI:
RMSEA:
0.973
0.124
$ spent on alcohol
$ spent on drugs
CFA
CFI:
RMSEA:
0.992
0.161
18
Item Information Curves for
Indicators Assigned to Abstinence
Domain: Factor 1
Alcohol use in lifetime and used in
the past 30 days
Alcohol substance is the major
problem
$ spent on alcohol
$ spent on drugs
19
Test Information Curves for
Indicators Assigned to Abstinence
Domain: Factor 1
20
Results: Recovery Item Parameters
Abstinence:
Alcohol and Drug User
Alcohol problems in the past 30 days
Drug problems in the past 30 days
Troubled by alcohol in the past 30
days
Troubled by drugs in the past 30 days
Importance of alcohol treatment
Importance of drug treatment
Item
Item
Discrimination Difficulty
(a )
(b)
2.054
1.585
2.597
1.088
3.402
1.378
9.218
0.868
-1.790
1.204
-2.001
0.693
21
Item Characteristic Curves for
Indicators Assigned to Abstinence
Domain: Factor 1
Alcohol problems in the past 30 days
Drug problems in the past 30 days
Troubled by drugs in the past 30
days
Importance of drug treatment
Troubled by alcohol in the past 30 days
Importance of alcohol treatment
EFA
CFI:
RMSEA:
0.973
0.124
CFA
CFI:
RMSEA:
0.981
0.149
22
Item Information Curves for
Indicators Assigned to Abstinence
Domain: Factor 2
Alcohol problems in the past 30 days
Troubled by alcohol in the past 30 days
Importance of alcohol treatment
Drug problems in the past 30 days
Troubled by drugs in the past 30
days
Importance of drug treatment
23
Test Information Curves for
Indicators Assigned to Abstinence
Domain: Factor 1
24
Results: Recovery Item Parameters
Item
Discrimination
(a )
Physical Health
Item
Difficulty
(b)
Have chronic medical problems
1.577
0.681
1.491
1.066
0.786
2.844
10.235
0.538
4.398
0.559
Take prescribed medications
Receive pension for physical disability
Troubled by medical problems in the
past 30 days
Importance of medical treatment
EFA
CFI:
RMSEA:
0.996
0.130
CFA
CFI:
RMSEA:
0.996
0.130
25
Item Characteristic Curves for Indicators
Assigned to Physical Health Domain
Have chronic medical problems
Receive pension for physical disability
Importance of medical treatment
Take prescribed medications
Troubled by medical problems
in the past 30 days
26
Item Information Curves for Indicators
Assigned to Physical Health Domain
Have chronic medical problems
Receive pension for physical disability
Importance of medical treatment
Take prescribed medications
Troubled by medical problems
in the past 30 days
27
Test Information Curve for Indicators Assigned
to Physical Health Domain
28
Results: Recovery Item Parameters
Mental Health
Item
Discrimination
(a )
Item
Difficulty
(b)
Suffer sexual abuse in life
0.487
2.726
Anxiety in the past 30 days
1.117
0.617
Hallucinations in the past 30 days
0.925
2.393
Trouble concentrating in lifetime
0.883
0.018
Violent behavior in lifetime
0.679
1.471
0.543
0.018
0.758
1.743
0.758
1.091
29
Importance of psychological treatment
Ever treated in a hospital/residential
environment for psychological or emotional
problems
Ever treated in an outpatient or private
patient for psychological or emotional
problems
Item Characteristic Curves for Indicators
Assigned to Mental Health Domain
Suffer sexual abuse in life
Hallucinations in the past 30 days
Violent behavior in lifetime
Ever treated in a
hospital/residential environment for
psychological or emotional
problems
Anxiety in the past 30 days
Trouble concentrating in lifetime
Importance of psychological
treatment
Ever treated in an outpatient or
private patient for psychological
or emotional problems
30
Item Information Curves for Indicators
Assigned to Mental Health Domain
Anxiety in the past 30 days
Trouble concentrating in lifetime
Importance of psychological
treatment
Ever treated in an outpatient or
private patient for psychological
or emotional problems
Suffer sexual abuse in life
Hallucinations in the past 30 days
Violent behavior in lifetime
Ever treated in a hospital/residential
environment for psychological or
emotional problems
EFA
CFI:
RMSEA:
0.996
0.020
CFA
CFI:
RMSEA:
0.822
0.099
31
Test Information Curve for Indicators Assigned
to Mental Health Domain
32
Summary
• We used item response theory (IRT) analysis
to identify 3 domains of recovery: Abstinence,
Physical Health, and Mental Health
• We chose the best fitting models for each
domain
• The final number of indicators was 23
indicators from 249 items.
33
Strengths and Limitations
Strengths
– Large and longitudinal sample
– Data-rich with item level information
Limitations
– Assumptions of model: trait(s) may be
discontinuous in population
– Best fit models do not meet standard criteria used
in psychometric literature.
– Limited generalizability
34
Conclusion
•We used item response theory (IRT) to
empirically derive indicators for the future
development of a standardized short recovery
assessment for both clinical and research studies.
•Based on IRT principles, we determined items
that reflect both discrimination between and
difficulty across varying levels of underlying
recovery levels for individuals in the TUE study.
35
36
37
38
Future Work
• Examine other item pools in a different
data set, more recent and generalizable.
• Resubmit R03 with Specific Aims to
examine differential item functioning in the
recovery scale due to gender differences
– And include validity test with independent
outcome measures.
– Longitudinal IRT
• Diagnostic validity relative to structured
clinical psychiatric diagnoses
39