Transcript Item Response Theory Pattern Scoring

The ABC’s of
Pattern Scoring
Dr. Cornelia Orr
Vocabulary
• Measurement – Psychometrics is a
type of measurement
• Classical test theory
• Item Response Theory – IRT
(AKA logistic trait theory)
• 1, 2, & 3-parameter IRT models
• Pattern Scoring
Slide 2
General & Specialized
Measurement
• Assign numbers to
objects or events
• Ex. – hurricanes,
earthquakes, time,
stock market,
height, weight
Psychometrics
• Assigning numbers
to psychological
characteristics
• Ex. – achievement
personality, IQ,
opinion, interests
Slide 3
Different Theories of
Psychometrics
Classical Test Theory
• Item discrimination
values
• Item difficulty values
(p-values)
• Guessing (penalty)
Number correct scoring
Item Response Theory
a) Item discrimination
values
b) Item difficulty values
c) Guessing (pseudoguessing) values
Pattern scoring
Similar constructs – Different derivations
Slide 4
Different Methods
of Scoring
Number-Correct Scoring
•
Simple Mathematics
•
Raw scores (# of points)
–
Mean, SD, SEM, % correct
•
Number right scale
•
Score conversions
–
Scale scores, percentile
ranks, etc.
Pattern Scoring
• Complex Mathematics
• Maximum likelihood
estimates
– Item statistics, student’s
answer pattern, SEM
• Theta scale (mean=0,
standard dev=1)
• Score conversions
– Scale scores, percentile
ranks, etc.
Slide 5
Comparison: Number Correct
and Pattern Scoring
Similarities
• The relationship of
derived scores is the
same, e.g.,
– High correlation, (0.95)
of number right scores
and scale scores
– Scale score has the same
percentile rank for both
methods
Differences
• Methods of deriving
scores
• The number of scale
scores possible
– Number right = limited
to the number of items
– IRT = unlimited or is
limited by the scale
(ex. 100-500)
Slide 6
Choosing the Scoring Method
•
•
•
•
Which model?
Simple vs. Complex?
Best estimates?
Advantages/Disadvantages?
Ex. – Why do the same number correct get
different scale scores?
Ex. – Flat screen TV – how do they do that?
Slide 7
Advantages of IRT and
Pattern Scoring
• Better estimates of an examinee’s ability
– the score that is most likely, given the
student’s responses to the questions on the
test (maximum likelihood scoring)
• More information about students and
items are used
• More reliability than number right scoring
• Less measurement error (SEM)
Slide 8
Disadvantages of IRT
and Pattern Scoring
• Technical - Complex Mathematics –
– Difficult to understand
– Difficult to explain
• Not common – Not like my experience.
• Perceived as “Hocus Pocus”
Slide 9
0.8
0.6
D is c rimination=1
D iffic ulty =0.5
0.2
0.4
Ps eudo-Gues s . =0.13
0.0
Probability of Correct Response
1.0
Item Characteristic
Curve (ICC)
-4
-2
0
2
4
Ac hiev ement Index (Theta)
Slide 10
Examples
Effect Of Item Difficulty
No Type a
1 1 MC 0.0150
2 1 MC 0.0150
3 1 MC 0.0150
4 1 MC 0.0150
5 1 MC 0.0150
b
250.000
275.000
300.000
325.000
350.000
c
0.1
0.1
0.1
0.1
0.1
Response Patterns
(1=correct)
Pattern
12345
11100
01110
10101
10011
SEM
SS
43
43
43
43
300
305
305
310
Answering more difficult items
(b-parameter) can result in
higher scores.
Slide 11
Examples
5 Items (Effects of Item
Discrimination)
No Type a
b
1 MC 0.0050 300.000
2 MC 0.0100 300.000
3 MC 0.0150 300.000
4 MC 0.0200 300.000
5 MC 0.0250 300.000
c
0.2
0.2
0.2
0.2
0.2
Response patterns
(1=correct)
Pattern
12345
11001
11100
01110
00111
SEM
94
61
46
39
SS
280
270
300
305
Answering more discriminating
items (a-parameter) can result
in higher scores.
Slide 12