Transcript ONR MURI - University of Colorado Boulder

ACT-R models of training
Cleotilde Gonzalez and Brad Best
Dynamic Decision Making Laboratory (DDMLab)
www.cmu.edu/ddmlab
Carnegie Mellon University
DDMLab – September 27, 2006 - 1
Our Goals in the MURI Project
• Create computational models that will be used as
predictive tools for the different effects resulting
from the application of empirically-based training
principles
• The predictive training models will help:
o
o
o
manipulate a set of training, task and ACT-R
parameters
determine speed and accuracy as a result of the
parameter settings and the training principles
Easily generate predictions of training effects for
new manipulations
DDMLab – September 27, 2006 - 2
Agenda
• Prolonged work and the speed-accuracy tradeoff
o
The data entry task
o
ACT-R models of fatigue effects
• Repetition priming effect
o
Initial ACT-R models of repetition priming
o
Predictions to be verified in current data collection
• Training difficulty principle
o
The radar task
o
ACT-R models of consistent and varied mapping effects
DDMLab – September 27, 2006 - 3
Data entry is ubiquitous in human life
DDMLab – September 27, 2006 - 4
Opposing processes affect performance
•
From Healy et al., 2004: Prolonged work results in distinctive opposite effects
(facilitative and inhibitory)
•
Proportion of errors increases while response time decreases.
DDMLab – September 27, 2006 - 5
The 2x2 levels of ACT-R
http://act.psy.cmu.edu
(Anderson & Lebiere, 1998)
Declarative Memory
Procedural Memory
Chunks: declarative
Productions: If
facts
(cond) Then (action)
Symbolic
Activation of chunks
Conflict Resolution
(likelihood of
(likelihood of use)
retrieval)
SubSymbolic
DDMLab – September 27, 2006 - 6
ACT-R equations
http://act.psy.cmu.edu
(Anderson & Lebiere, 1998)
W  S
B  ln  t
Activation Ai  Bi 
Learning
j
ji
 A
j
d
j
i
Intentions
Memory
Goal
Retrieval
j
Latency
Ti  F  e Ai
Ui  Pi G  Ci   U
Succ i
Learning Pi 
Succ i  Faili
Utility
Productions
Visual
Manual
Motor
Vision
World
IF the goal is to categorize new stimulus
and visual holds stimulus info S, F, T
THEN start retrieval of chunk S, F, T
and start manual mouse movement
Size Fuel Turb Dec
Stimulus
Bi
Chunk
S 20 1
SSL
L 20 3
S13
Y
DDMLab – September 27, 2006 - 7
Chunk Activation
activation
(
)(
associative
source
activation* strength
base
= activation
+
+
mismatch
penalty
*
similarity
value
)
+ noise
A i  Bi   Wj  Sji   MPk  Simkl  N(0,s)
j
k
Activation makes chunks available to the degree that past experiences
indicate that they will be useful at the particular moment.
Base-level: general past usefulness
Associative Activation: relevance to the general context
Matching Penalty: relevance to the specific match required
Noise: stochastic is useful to avoid getting stuck in local minima
Higher activation = fewer errors and faster retrievals
DDMLab – September 27, 2006 - 8
Production compilation
• Basic idea:
o
o
Productions are combined to form a macro
production  faster execution
Rule learning:
• Retrievals may be eliminated in the process
• Practically: declarative  procedural transition
o
Production learning produces power-law speedup
• The power-law function does not appear in the compilation
mechanism, rather the power-law emerges from the
mechanism
DDMLab – September 27, 2006 - 9
ACT-R Models of Fatigue Effects in Data
Entry
• ACT-R Model 1: Prolonged work and the speedaccuracy tradeoff (Experiment 1 from Healy, Kole,
Buck-Gengler and Bourne, 2004)
• ACT-R Model 2: Speed-accuracy trade-off changes
for motoric and cognitive components (Experiment 2
from Healy, Kole, Buck-Gengler and Bourne, 2004)
• ACT-R Model 3: How do cognitive and motoric
stressors affect different response time components:
articulatory suppression and weight (Experiment 1 from
Kole, Healy ad Bourne, 2006)
DDMLab – September 27, 2006 - 10
ACT-R model of the data entry task
Encode next number
Initiation
time
Retrieve key location
All numbers encoded
Execution
time
Type next number
All numbers typed
Conclusion
time
Hit Enter
DDMLab – September 27, 2006 - 11
ACT-R Model 1
Error Proportion
Total RT
0.16
2.75
0.14
Error proportion
Observed
Predicted
0.1
0.08
0.06
0.04
Total Response Time (msec)
2.7
0.12
Oserved
2.65
Predicted
2.6
2.55
0.02
2.5
0
1
2
3
4
5
6
Block
R2=.89
7
8
9
10
1
2
3
4
5
6
7
8
9
10
Block
R2=.89
DDMLab – September 27, 2006 - 12
ACT-R model 1
• Speedup: Production
Encode next number
Initiation
time
Retrieve key location
All numbers encoded
Execution
time
Type next number
All numbers typed
Conclusion
time
Hit Enter
compilation:
o From Visual 
Retrieval (key loc) 
Motor
o To Visual  Motor
o faster access to key
loc
• Accuracy ↓: Degradation
of source activation (W):
A i  Bi   Wj  Sji   MPk  Simkl  N(0,s)
j
k
DDMLab – September 27, 2006 - 13
ACT-R Models of Fatigue Effects in Data
Entry
• ACT-R Model 1: Prolonged work and the speedaccuracy tradeoff (Experiment 1 from Healy, Kole,
Buck-Gengler and Bourne, 2004)
• ACT-R Model 2: Speed-accuracy trade-off changes
for motoric and cognitive components (Experiment 2
from Healy, Kole, Buck-Gengler and Bourne, 2004)
• ACT-R Model 3: How do cognitive and motoric
stressors affect different response time components:
articulatory suppression and weight (Experiment 1 from
Kole, Healy ad Bourne, 2006)
DDMLab – September 27, 2006 - 14
ACT-R Model Accuracy
0.94
0.92
Proportion Correct
Observed
Predicted
0.9
0.88
0.86
0.84
0.82
1
2
3
4
5
6
7
8
9
10
Block
R2=.68
DDMLab – September 27, 2006 - 15
ACT-R Model 2
Conclusion Time
Initiation Time
1.2
1.18
Initiation Time (sec)
1.16
1.14
1.12
1.1
1.08
1.06
Observed
1.04
Predicted
1.02
Conclusion Time (sec)
0.3
Observed
0.29
Predicted
0.28
0.27
0.26
0.25
0.24
1
1 2 3 4 5 6 7 8 9 10
1
2
3
4
5
6
Block
R2=.81
7
8
9
10
Block
R2=.85
DDMLab – September 27, 2006 - 16
ACT-R model 2
• Speedup: Production
compilation
Encode next number
o
Initiation
time
Retrieve key location
o
o
All numbers encoded
o
Execution
time
Type next number
All numbers typed
Conclusion
time
Hit Enter
From Visual  Retrieval
(key loc)  Motor
To Visual  Motor
faster access to key loc
Gradual decrease in goal
value (G)
Ui  Pi  G  Ci   U
• Accuracy ↓:
o
A gradual decrease in
source activation (W)
A i  Bi   Wj  Sji   MPk  Simkl  N(0,s)
j
k
DDMLab – September 27, 2006 - 17
ACT-R Models of Fatigue Effects in Data
Entry
• ACT-R Model 1: Prolonged work and the speedaccuracy tradeoff (Experiment 1 from Healy, Kole,
Buck-Gengler and Bourne, 2004)
• ACT-R Model 2: Speed-accuracy trade-off changes
for motoric and cognitive components (Experiment 2
from Healy, Kole, Buck-Gengler and Bourne, 2004)
• ACT-R Model 3: How do cognitive and motoric
stressors affect different response time components:
articulatory suppression and weight (Experiment 1 from
Kole, Healy ad Bourne, 2006)
DDMLab – September 27, 2006 - 18
ACT-R Model 3
Total response time
ACT-R Prediction
Human data
2.8
Suppression
2.7
Total Response Time (msec)
Silent
2.6
2.5
2.4
2.3
2.2
2.1
2
1
2
3
4
5
6
7
8
9
10
Block
DDMLab – September 27, 2006 - 19
ACT-R Model 3
Proportion correct
Human data
ACT-R Prediction
0.92
Proportion Correct
0.9
Suppression
0.88
Silent
0.86
0.84
0.82
0.8
0.78
1
2
3
4
5
6
7
8
9
10
Block
DDMLab – September 27, 2006 - 20
ACT-R model 3
• Same as Model 2:
Encode next number
suppression
Initiation
time
Retrieve key location
All numbers encoded
Execution
time
• With articulatory
• Not a ‘fit’ but a
prediction
Type next number
All numbers typed
Conclusion
time
Hit Enter
DDMLab – September 27, 2006 - 21
Summary of Act-R models of Fatigue
• The model provides detailed predictions of the speed and accuracy
tradeoff effect with prolonged work in the data entry task:
o
Decreased RT by production compilation
o
Increase Errors by gradual decrease in source activation
• Fatigue may affect cognitive and motor components differently:
o
strong effects of fatigue by gradual decrease in goal value
o
no effects on motor components
• The ACT-R model suggest that cognitive fatigue may arise from a
cognitive control and motivational processes (Jongman, 1998)
DDMLab – September 27, 2006 - 22
Agenda
• Prolonged work and the speed-accuracy tradeoff
o
The data entry task
o
ACT-R models of fatigue effects
• Repetition priming effect
o
Initial ACT-R models of repetition priming
o
Predictions to be verified in current data collection
• Training difficulty principle
o
The radar task
o
ACT-R models of consistent and varied mapping effects
DDMLab – September 27, 2006 - 23
Empirical test of model’s predictions
(Gonzalez, Fu, Healy, Kole, and Bourne, 2006)
• Effects of number of repetitions and delay on repetition priming
o
How performance deteriorates with different delays after training?
o
How re-training may help retention of skills?
0.8
no re-train
RT difference (Training-Test)
0.7
Re-train 1
Re-train 2
0.6
0.5
0.4
0.3
0.2
0.1
0
0
2
4
6
8
10
12
14
16
18
Number of days between end of training and testing
DDMLab – September 27, 2006 - 24
Agenda
• Prolonged work and the speed-accuracy tradeoff
o
The data entry task
o
ACT-R models of fatigue effects
• Repetition priming effect
o
Initial ACT-R models of repetition priming
o
Predictions to be verified in current data collection
• Training difficulty principle
o
The radar task
o
ACT-R models of consistent and varied mapping effects
DDMLab – September 27, 2006 - 25
The Radar Task
(From Gonzalez and Thomas, under review;
Gonzalez, Thomas, and Madhavan, under review)
Radar Grid
Threat Sensors
Weapon System Sensors
F
D
Quiet Airspace Report
Target Set
(Memory Set)
G
J
Guns
Ignore
Missiles
Total Block Score
DDMLab – September 27, 2006 - 26
The Radar Task
(From Gonzalez and Thomas, under review;
Gonzalez, Thomas, and Madhavan, under review)
DDMLab – September 27, 2006 - 27
Current data fits to human data:
effects of mapping and load
RADAR: Model Latency at Training
Human
Model
1400
Response Time (ms)
1200
1000
800
600
400
200
0
CM 1 - 1
CM 4 - 4
VM 1 - 1
VM 4 - 4
Condition
DDMLab – September 27, 2006 - 28
ACT-R Model of automaticity
Rehearse
Memory Set
Focus on
Next Frame
Attend to
Next Target
No More Targets
Target Different Type
as MS
VM:
Target Same Type
Try Retrieving
Not Target
(Retrieval Failure)
Respond:
Target Not Found
CM:
Target Same Type
Target Found
Target
(Retrieval Success)
Respond:
Target Found
DDMLab – September 27, 2006 - 29
Consistent Mapping Conditions
Rehearse
Memory Set
Focus on
Next Frame
Attend to
Next Target
No More Targets
Respond:
Target Not Found
Target Different Type
as MS
CM:
Target Same Type
Target Found
Respond:
Target Found
DDMLab – September 27, 2006 - 30
Varied Mapping Conditions
Rehearse
Memory Set
Focus on
Next Frame
Attend to
Next Target
No More Targets
VM:
Target Different Type
Target Same Type
as MS
Try Retrieving
Not Target
(Retrieval Failure)
Respond:
Target Not Found
Target
(Retrieval Success)
Respond:
Target Found
DDMLab – September 27, 2006 - 31
Current model issues
• The model depends on knowing when to retrieve
o
o
If the target is the same type as the memory set, in CM that
means it’s the target, but in VM it may be a distracter
When are participants aware of the condition they’re in?
• False alarm rates should indicate when participants think they’re in
CM, but actually are in VM
o
o
Adaptation should happen over time in VM false alarm rates
Latency may increase in VM due to fewer skipped retrievals
(participants may initially think they’re in VM and actually get
the target without doing a retrieval)
DDMLab – September 27, 2006 - 32
Summary of this year’s accomplishments
• Generation of new ACT-R models to demonstrate the Training
Difficulty hypothesis in the radar task
o
o
effects of consistent and varied mapping with extended
task practice
Initial tuning of the model with experimental data collected
• Enhancement of ACT-R models and creation of new models that
reproduce the speed-accuracy tradeoff effect for the data
entry task
• Enhancement of current ACT-R models of repetition priming
and depth of processing for the data entry task
• Initial development of a general model of ACT-R fatigue that
can be applied to any existing model
DDMLab – September 27, 2006 - 33
Plans for next year
• On the data entry task:
o
o
Report on the cognitive functions and mechanisms
corresponding the speed-accuracy trade off in prolonged work,
based on ACT-R/empirical results
Enhance and create the models corresponding to the repetition
priming and depth of processing effects
• Generate a predictive training tool for the data entry task in which
training, task, and ACT-R parameters can be manipulated to
produce speed and accuracy results
• On the Radar task:
o
o
Reproduce the effects of the difficulty of training hypothesis
Produce new predictions on the effects of stimulus-response
mappings
• Add learning to condition determination
DDMLab – September 27, 2006 - 34