Educational Data Mining
Download
Report
Transcript Educational Data Mining
Educational Data Mining:
Discovery with Models
Ryan S.J.d. Baker
PSLC/HCII
Carnegie Mellon University
Ken Koedinger
CMU Director of PSLC
Professor of Human-Computer Interaction & Psychology
Carnegie Mellon University
In this segment…
We will discuss Discovery with Models in
(some) detail
Last time…
We gave a very simple example of Discovery
with Models using Bayesian Knowledge
Tracing
Uses of Knowledge Tracing
Can be interpreted to learn about skills
Skills from the Algebra Tutor
skill
L0
T
0.01
0.01
ApplyExponentExpandExponentsevalradicalE
0.333
0.497
CalculateEliminateParensTypeinSkillElimi
0.979
0.001
CalculatenegativecoefficientTypeinSkillM
0.953
0.001
Changingaxisbounds
0.01
0.01
Changingaxisintervals
0.01
0.01
ChooseGraphicala
0.001
0.306
combineliketermssp
0.943
0.001
AddSubtractTypeinSkillIsolatepositiveIso
Which skills could probably be
removed from the tutor?
skill
L0
T
0.01
0.01
ApplyExponentExpandExponentsevalradicalE
0.333
0.497
CalculateEliminateParensTypeinSkillElimi
0.979
0.001
CalculatenegativecoefficientTypeinSkillM
0.953
0.001
Changingaxisbounds
0.01
0.01
Changingaxisintervals
0.01
0.01
ChooseGraphicala
0.001
0.306
combineliketermssp
0.943
0.001
AddSubtractTypeinSkillIsolatepositiveIso
Which skills could use better
instruction?
skill
L0
T
0.01
0.01
ApplyExponentExpandExponentsevalradicalE
0.333
0.497
CalculateEliminateParensTypeinSkillElimi
0.979
0.001
CalculatenegativecoefficientTypeinSkillM
0.953
0.001
Changingaxisbounds
0.01
0.01
Changingaxisintervals
0.01
0.01
ChooseGraphicala
0.001
0.306
combineliketermssp
0.943
0.001
AddSubtractTypeinSkillIsolatepositiveIso
Why do Discovery with Models?
We have a model of some construct of
interest or importance
Knowledge
Meta-Cognition
Motivation
Affect
Collaborative Behavior
Helping Acts, Insults
Etc.
Why do Discovery with Models?
We can now use that model to
Find outliers of interest by finding out where the
model makes extreme predictions
Inspect the model to learn what factors are
involved in predicting the construct
Find out the construct’s relationship to other
constructs of interest, by studying its
correlations/associations/causal relationships with
data/models on the other constructs
Study the construct across contexts or students,
by applying the model within data from those
contexts or students
And more…
Finding Outliers of Interest
Finding outliers of interest by finding out
where the model makes extreme predictions
As in the example from Bayesian Knowledge
Tracing
As in Ken’s example yesterday of finding upward
spikes in learning curves
Model Inspection
By looking at the features in the Gaming
Detector, Baker, Corbett, & Koedinger (2004, in
press) were able to see that
Students who game the system and have poor
learning
game the system on steps they don’t know
Students who game the system and have good
learning
game the system on steps they already know
Model Inspection: A tip
The simpler the model, the easier this is to do
Decision Trees and Linear/Step Regression:
Easy.
Model Inspection: A tip
The simpler the model, the easier this is to do
Decision Trees and Linear/Step Regression:
Easy.
Neural Networks and Support Vector
Machines: Fuhgeddaboudit!
Correlations to Other Constructs
Take Model of a Construct
And see whether it co-occurs with other
constructs of interest
Example
Detector of gaming the system (in fashion
associated with poorer learning) correlated
with questionnaire items assessing various
motivations and attitudes
(Baker et al, 2008)
Example
Detector of gaming the system (in fashion
associated with poorer learning) correlated
with questionnaire items assessing various
motivations and attitudes
(Baker et al, 2008)
Surprise: Nothing correlated very well
(correlations between gaming and some
attitudes statistically significant, but very
weak – r < 0.2)
Example
More on this in a minute…
Studying a Construct Across
Contexts
Often, but not always, involves:
Model Transfer
Model Transfer
Richard said that prediction assumes that the
Sample where the predictions are made
Is “the same as”
The sample where the prediction model was
made
Not entirely true
Model Transfer
It’s more that prediction assumes the differences
“aren’t important”
So how do we know that’s the case?
Model Transfer
You can use a classifier in contexts beyond where it was
trained, with proper validation
This can be really nice
you may only have to train on data from 100 students and 4
lessons
and then you can use your classifier in cases where there is data
from 1000 students and 35 lessons
Especially nice if you have some unlabeled data set with
nice properties
Additional data such as questionnaire data
(cf. Baker, 2007; Baker, Walonoski, Heffernan, Roll, Corbett, &
Koedinger, 2008)
Validate the Transfer
You should make sure your model is valid in
the new context
(cf. Roll et al, 2005; Baker et al, 2006)
Depending on the type of model, and what
features go into it, your model may or may
not be valid for data taken
From a different system
In a different context of use
With a different population
Validate the Transfer
For example
Will an off-task detector trained in schools
work in dorm rooms?
Validate the Transfer
For example
Will a gaming detector trained in a tutor
where
{gaming=systematic guessing, hint abuse}
Work in a tutor where
{gaming=point cartels}
Validate the Transfer
However
Will a gaming detector trained in a tutor unit
where
{gaming=systematic guessing, hint abuse}
Work in a different tutor unit where
{gaming=systematic guessing, hint abuse}?
Maybe…
Baker, Corbett, Koedinger, &
Roll (2006)
We tested whether
A gaming detector trained in a tutor unit where
{gaming=systematic guessing, hint abuse}
Would work in a different tutor unit where
{gaming=systematic guessing, hint abuse}
Scheme
Train on data from three lessons, test on a
fourth lesson
For all possible combinations of 4 lessons
(4 combinations)
Transfer lesson .vs. Training
lessons
Ability to distinguish students who game from nongaming students
Overall performance in training lessons: A’ = 0.85
Overall performance in test lessons:
A’ = 0.80
Difference is NOT significant, Z=1.17, p=0.24
(using Strube’s Adjusted Z)
So transfer is possible…
Of course 4 successes over 4 lessons from
the same tutor isn’t enough to conclude that
any model trained on 3 lessons will transfer
to any new lesson
What we can say is…
If…
If we posit that these four cases are
“successful transfer”, and assume they were
randomly sampled from lessons in the middle
school tutor…
Maximum Likelihood Estimation
How likely is it that models transfer to four lessons?
(result in Baker, Corbett, & Koedinger, 2006)
Probability of data
100%
80%
60%
40%
20%
0%
0%
10%
20%
30%
40%
50%
60%
70%
80%
Percent of lessons models would transfer to
90%
100%
Studying a Construct Across
Contexts
Using this detector
(Baker, 2007)
Research Question
Do students game the system because of state or trait
factors?
If trait factors are the main explanation, differences
between students will explain much of the variance in
gaming
If state factors are the main explanation, differences
between lessons could account for many (but not all)
state factors, and explain much of the variance in
gaming
So: is the student or the lesson a better predictor of
gaming?
Application of Detector
After validating its transfer
We applied the gaming detector across 35
lessons, used by 240 students, from a single
Cognitive Tutor
Giving us, for each student in each lesson, a
gaming frequency
Model
Linear Regression models
Gaming frequency = Lesson + a0
Gaming frequency = Student + a0
Model
Categorical variables transformed to a set of
binaries
i.e. Lesson = Scatterplot becomes
3DGeometry = 0
Percents = 0
Probability = 0
Scatterplot = 1
Boxplot = 0
Etc…
Metrics
r2
The correlation, squared
The proportion of variability in the data set
that is accounted for by a statistical model
r2
The correlation, squared
The proportion of variability in the data set
that is accounted for by a statistical model
r2
However, a limitation
The more variables you have, the more
variance you should be expected to predict,
just by chance
r2
We should expect
240 students
To predict gaming better than
35 lessons
Just by overfitting
So what can we do?
Our good friend BiC
Bayesian Information Criterion
(Raftery, 1995)
Makes trade-off between goodness of fit and
flexibility of fit (number of parameters)
Predictors
The Lesson
Gaming frequency = Lesson + a0
35 parameters
r2 = 0.55
BiC’ = -2370
Model is significantly better than chance would
predict given model size & data set size
The Student
Gaming frequency = Student + a0
240 parameters
r2 = 0.16
BiC’ = 1382
Model is worse than chance would predict given
model size & data set size!
Standard deviation bars, not standard error bars
In this talk…
Discovery with Models to
Find outliers of interest by finding out where the
model makes extreme predictions
Inspect the model to learn what factors are
involved in predicting the construct
Find out the construct’s relationship to other
constructs of interest, by studying its
correlations/associations/causal relationships with
data/models on the other constructs
Study the construct across contexts or students,
by applying the model within data from those
contexts or students
Necessarily…
Only a few examples given in this talk
An area of increasing importance
within EDM…
In the last 3 days we have discussed
(or at least mentioned)
5 broad areas of EDM
Prediction
Clustering
Relationship Mining
Discovery with Models
Distillation of Data for Human Judgment
Now it’s your turn
To use these techniques to answer important
questions about learners and learning
To improve these techniques, moving forward
To learn more
Baker, R.S.J.d. (under review) Data Mining in Education.
Under review for inclusion in the International
Encyclopedia of Education
Available upon request
Baker, R.S.J.d., Barnes, T., Beck, J.E. (2008)
Proceedings of the First International Conference on
Educational Data Mining
Romero, C., Ventura, S. (2007) Educational Data Mining:
A Survey from 1995 to 2005. Expert Systems with
Applications, 33 (1), 135-146.
END
values
a
b
c
d
e
f
g
h
i
j
k
0.1
0.31703
0.184794
0.292674
0.968429
0.599052
0.258772
0.288868
0.479694
0.845986
0.312878
0.325583
0.2
0.587882
0.818468
0.66771
0.286849
0.571331
0.878487
0.368984
0.156295
0.529126
0.009659
0.827527
0.3
0.069229
0.614344
0.016678
0.625279
0.07258
0.60644
0.376906
0.546482
0.780456
0.85199
0.99095
0.4
0.134072
0.761594
0.45686
0.075598
0.902216
0.349661
0.41452
0.377848
0.271817
0.808268
0.152187
0.5
0.773527
0.568502
0.212827
0.296644
0.606759
0.763751
0.337572
0.658086
0.527355
0.248425
0.306963
0.6
0.382031
0.954357
0.46915
0.793141
0.422994
0.00778
0.132219
0.218946
0.26634
0.204495
0.428783
0.7
0.499437
0.317859
0.56981
0.97822
0.926654
0.549637
0.241934
0.293575
0.910287
0.498185
0.803212
0.8
0.452056
0.133885
0.554752
0.771215
0.77231
0.867048
0.398835
0.310958
0.779538
0.75974
0.127566
0.9
0.013696
0.055595
0.887505
0.253549
0.529121
0.301857
0.846878
0.989624
0.480956
0.442541
0.614105
1
0.504806
0.462066
0.596407
0.986423
0.535024
0.475623
0.450906
0.07588
0.036826
0.995523
0.827306
Real data
Random numbers
values
a
b
c
d
e
f
g
h
i
j
k
0.1
0.31703
0.184794
0.292674
0.968429
0.599052
0.258772
0.288868
0.479694
0.845986
0.312878
0.325583
0.2
0.587882
0.818468
0.66771
0.286849
0.571331
0.878487
0.368984
0.156295
0.529126
0.009659
0.827527
0.3
0.069229
0.614344
0.016678
0.625279
0.07258
0.60644
0.376906
0.546482
0.780456
0.85199
0.99095
0.4
0.134072
0.761594
0.45686
0.075598
0.902216
0.349661
0.41452
0.377848
0.271817
0.808268
0.152187
0.5
0.773527
0.568502
0.212827
0.296644
0.606759
0.763751
0.337572
0.658086
0.527355
0.248425
0.306963
0.6
0.382031
0.954357
0.46915
0.793141
0.422994
0.00778
0.132219
0.218946
0.26634
0.204495
0.428783
0.7
0.499437
0.317859
0.56981
0.97822
0.926654
0.549637
0.241934
0.293575
0.910287
0.498185
0.803212
0.8
0.452056
0.133885
0.554752
0.771215
0.77231
0.867048
0.398835
0.310958
0.779538
0.75974
0.127566
0.9
0.013696
0.055595
0.887505
0.253549
0.529121
0.301857
0.846878
0.989624
0.480956
0.442541
0.614105
1
0.504806
0.462066
0.596407
0.986423
0.535024
0.475623
0.450906
0.07588
0.036826
0.995523
0.827306
num vars
1
2
3
4
5
6
7
8
9
10
r2
0.000
0.144
0.370
0.411
0.421
0.422
0.612
0.703
1
1
r2
Nine variables of random junk successfully got
an r2 of 1 on ten data points
And that’s what we call overfitting