Transcript Gobert for DRK-12 PI mtg 2010x

Science Assistments
Dr. Janice Gobert
Principal Investigator
[email protected]
Associate Professor,
Social Sciences Dept.
& Computer Science
Dept.
Co-Director,
Learning Sciences &
Technology Program
Funded by NSF-DRL# 0733286, NSF-DGE #0742503, NSF-DRL #1008649)
and by the U.S. Dept. of Education (R305A090170).
Science Assistments Team
Investigators
Dr. Janice Gobert (PI), Social Sciences & Policy Studies, WPI
Dr. Neil Heffernan, Computer Science, WPI
Dr. Ryan Baker, Social Sciences & Policy Studies, WPI
Dr. Joe Beck, Computer Science, WPI
Dr. Carolina Ruiz, Computer Science, WPI
Dr. Ken Koedinger, HCII, Carnegie-Mellon University
Graduate Students & Staff
Arnon Hershkovitz, Ph.D, Post-Doctoral Researcher
Ermal Toto, Ph.D. Student, LS&T/Software Engineer
Orlando Montalvo, Ph.D. Student, LS&T/Software Engineer
Michael Sao Pedro, Ph.D. Student, Computer Science
Juelaila Raziuddin, Ph.D. Student, LS&T
Adam Nakama, M.Sc. Student, LS&T
Matt Bachmann, M.Sc. Student, Computer Science
Mike Wixon, M.Sc. Student, LS&T
Cameron Betts, M.Sc. Student, LS&T
Project Overview
• Science Assistments is an
environment for
conducting performance
assessment of middle
school students’ inquiry in
Physical, Life, & Earth
Science.
• Our activities are based on
guided inquiry &
experimentation with
microworlds.
• We are auto-tutoring of
students’ inquiry based on
data mining and
knowledge engineering.
Hello! You are going to be a
scientist today and conduct
experiments in a virtual laboratory!
Students learn and are assessed while they do
inquiry with microworlds
• With microworlds, students:
- develop a hypothesis,
- design & conduct an experiment
- analyze data & warrant their claims, and
- communicate findings (NSES, 1996).
• And because we log all students’ actions, we can respond in real
time using our pedagogical agent, Rex
Educational Data Mining to auto-analyze log files
• Extending prior work on using logs to
characterize inquiry moves in microworlds
(Buckley, Gobert et al, 2010).
• Extending work of Baker et al (2008) by applying
text replays to label students’ inquiry moves.
Text Replay Tagging Software
Student
Clip
Replay
Clip Tags
EDM, cont’d
•
•
•
Once logs are labeled, use EDM to
determine what fine-grained logged features
correspond to specific inquiry skills.
Build detectors over feature sets, i.e.,
aggregates of logged actions.
Validate detectors (Sao Pedro et al,
Montalvo et al, 2010).
“Goodness” of our detectors for coding students’ inquiry processes
Testing Stated
Hypothesis
A’ = 0.86
Kappa = 0.46
Using CVS during
experimenting
A’ = 0.85
Kappa = 0.47
Data Table Use
A’ = 0.94
Kappa = 0.46
• A’ = probability of correct labeling given 2 examples (+ and – examples)
• Kappa = does the predictor do better than chance (chance level = 0; 1=
perfect)
Sao Pedro et al., 2010; Montalvo et al., 2010
Using Detectors to Predict Performance
Using our detectors as a basis for assessing authentic skill, we can
generate models that let us:
(1) Predict skill proficiency before a student starts a new activity
(2) Research the relationship between authentic skill honed in our
learning environment and other transfer measures of inquiry
Sao Pedro, Baker, Gobert, Montalvo, & Nakama (in prep.)
Results

Models accurately predict next attempt on microworld for
each inquiry skill
– Testing hypotheses (A’ = .79)
– Designing controlled experiments (cvs) (A’ = .74)
– Planning using the table tool (A’ = .71)

Authentic skill significantly correlates with performance
on transfer measures of inquiry
– Multiple-choice “testing hypotheses” assessment (r = .41)
– Multiple-choice “controlled experiments” assessment (r = .26)
– Authentic “controlled experiments” assessment (r = .38)
Papers/results cited here





Baker, R.S.J.d., de Carvalho, A. M. J. A. (2008) Labeling Student Behavior Faster
and More Precisely with Text Replays. Proceedings of the 1st International
Conference on Educational Data Mining, pp. 38-47.
Buckley, B. C., Gobert, J., Horwitz, P., & O’Dwyer, L. (2010). Looking inside the
black box: Assessing model-based learning and inquiry in BioLogica. Int.
Journal of Learning Technologies, 5(2), 166 - 190.
Montalvo, O., Baker, R.S.J.d., Sao Pedro, M.A., Nakama, A. & Gobert, J.D. (2010).
Identifying Students’ Inquiry Planning Using Machine Learning. Proceedings of
the 3rd International Conference on Educational Data Mining (pp. 141-150).
Sao Pedro, M.A., Baker, R.S.J.d, Montalvo, O., Nakama, A. & Gobert, J.D. (2010).
Using Text Replay Tagging to Produce Detectors of Systematic Experimentation
Behavior Pattern. Proceedings of the 3rd International Conference on
Educational Data Mining (pp. 181-190).
Sao Pedro, M., Baker, R.S.J.d, Gobert, J., Montalvo, O., & Nakama, A. (in prep).
Using Machine-Learned Detectors of Systematic Inquiry Behavior to Predict
Gains in Inquiry Skills.
For more information & papers


See our website,
www.scienceassistments.org
Or contact Janice Gobert:
[email protected].