Transcript Reporting Item Response Theory result Jeff Brookings Wittenberg

Reporting item response
theory results
Jeffrey B. Brookings
Wittenberg University
Presented at the SAMR/SWPA Symposium:
Handy tips for communicating and
reporting your findings
April 5, 2013
Ph.D. Comics, 2013
Item Response Theory
1. Mathematical models that probabilistically describe the
relation between a person’s response to an item and
his/her standing on a latent trait.
2. The Rasch model—a “one-parameter” model (difficulty)—
locates person ability and item difficulty on the same scale
(logits or log odds).
3. “…a person having a greater ability than another person
should have the greater probability of solving any item of
the type in question, and similarly, one item being more
difficult than another means that for any one person the
probability of solving the second item is the greater one.”
(Rasch, 1960, p. 117)
4. The purpose of Rasch analysis is to produce
unidimensional measures that cover a wide range of the
latent trait.
Reporting results from
a Rasch analysis
1. Item and scale descriptive statistics
2. PCA of standardized residuals following
extraction of the Rasch component (test for
unidimensionality)
3. Item “difficulty” estimates (in logits)
4. Item fit statistics
5. Item characteristic curves (ICCs)
6. Category response curves (CRRs)
7. Person/item map
8. Person/item separation reliability
The Psychosocial Risk Factor Survey
(Eichenauer, Feltz, Wilson, & Brookings, 2010)
• Assesses psychosocial risk factors for
cardiac disease
• 70 items, 5-point response scale: 0 “Strongly Agree” to 4 - “Strongly Disagree”
• Scales: Depression, Anxiety, Hostility, Social
Isolation, and Emotional Guardedness.
• Analyses: Responses to the 14 Depression
Scale items (340 patients from five cardiac
rehabilitation programs in the Midwest)
Rasch Item Statistics
PCA of Standardized Residuals
Total raw variance in observations
Raw variance explained by measures
Raw variance explained by persons
Raw variance explained by items
Raw unexplained variance (total)
Unexplained variance in 1st contrast
Unexplained variance in 2nd contrast
Unexplained variance in 3rd contrast
Unexplained variance in 4th contrast
Unexplained variance in 5th contrast
29.5 100.0%
15.5
52.6%
6.4
21.6%
9.2
31.0%
14.0 47.4%
1.9
6.4%
1.6
5.5%
1.3
4.4%
1.2
4.2%
1.2
4.0%
Figure 1. Outfit Plot for PRFS Depression Items
3
PRFS12
Measures
2
1
PRFS67
PRFS62
PRFS52PRFS32
PRFS7
0
PRFS57
PRFS22
PRFS27 PRFS37
PRFS17
PRFS2
PRFS42
-1
PRFS47
-2
0.1
1
Overfit
Outfit Mean-square (log-scaled)
Underfit
Item characteristic curve—with 95% CI—for item 27:
“My thoughts feel so scattered lately”
Item characteristic curve—with 95% CI—for item 12:
“I think more about ending my life lately”
Rasch Category Responses
Person/Item Map
Mean person
measure = -0.94
Mean item
measure = .00
Reliability
• Person separation reliability – Analogous
to Cronbach’s α; degree to which the scale
differentiates persons; range 0 – 1
– For PRFS Depression: .88
• Item separation reliability – Degree to
which item difficulties are differentiated;
range 0 – 1
– For PRFS Depression: .99
Summary of Rasch Analysis for
the PRFS Depression Scale
• Good evidence for unidimensionality
• Mean point-measure r = .626
• Acceptable person and item separation
reliabilities (.88 and .99, respectively)
• Some misalignment of persons and items
• One mis-fitting item: #12 (“I think more
about ending my life lately”)
Recommended Reading
Bond, T.G., & Fox, C.M. (2007). Appling the Rasch model:
Fundamental measurement in the human sciences
(2nd ed.). Mahwah, NJ: Erlbaum.