Transcript Cognitive Tutors: Bringing advanced cognitive research to the

LearnLab: Bridging the Gap
Between Learning Science and
Educational Practice
Ken Koedinger
Human-Computer Interaction & Psychology, CMU
PI & CMU Director of LearnLab
1
Real World Impact of
Cognitive Science
Algebra Cognitive Tutor
• Based on ACT-R theory
& cognitive models of
student learning
• Used in 3000 schools
600,000 students
• Spin-off:
Koedinger, Anderson, Hadley, & Mark (1997).
Intelligent tutoring goes to school in the big city.
2
Cognitive Tutors:
Interactive Support for
Learning by Doing
Authentic problems
Feedback within complex solutions
Progress…
Personalized instruction
Challenging questions
… individualization
3
Success ingredients
• AI technology
• Cognitive Task Analysis
• Principles of instruction &
experimental methods
• Fast development &
use-driven iteration
4
Cognitive Task Analysis:
What is hard for Algebra students?
Story Problem
As a waiter, Ted gets $6 per hour. One night he made $66 in
tips and earned a total of $81.90. How many hours did Ted
work?
Word Problem
Starting with some number, if I multiply it by 6 and then add
66, I get 81.90. What number did I start with?
Equation
x * 6 + 66 = 81.90
Data contradicts common beliefs
of researchers and teachers
Expert Blind Spot!
100
90
80
70
60
50
40
30
20
10
0
% Correctly ranking equations as
hardest
Elementary
Teachers
Koedinger & Nathan (2004). The real story
behind story problems: Effects of
representations on quantitative reasoning.
The Journal of the Learning Sciences.
Middle
School
Teachers
High School
Teachers
Nathan & Koedinger (2000). An
investigation of teachers’ beliefs of
students’ algebra development.
Cognition and Instruction.
6
Cognitive Tutor Algebra course
yields significantly better
learning
Course includes text,
tutor, teacher
professional
development
~11 of 14 full-year
controlled studies
demonstrate
significantly better
student learning
Koedinger, Anderson, Hadley, & Mark (1997).
Intelligent tutoring goes to school in the big city.
7
Success? Yes
Done? No!
Why not?
• Student achievement still not ideal
• Field study results are imperfect
• Many design decisions with no research
base
• Use deployed technology to collect
data, make discoveries, & continually
improve
8
PSLC Vision
• Why?
Chasm between science & ed practice
• Purpose: Identify the conditions
that cause robust student learning
– Educational technology as instrument
– Science-practice collaboration structure
• Core Funding:
2004-2014
9
Do you know what you know?
What we
know about
our own
learning
What we do
not know
You can’t design for what you don’t know!
10
Transforming Education R&D
Ed tech
+ wide use = “Basic research at scale”
Algebra Cognitive Tutor
+
=
Chemistry Virtual Lab
• Fundamentally transform
English Grammar Tutor
– Applied research in education
– Generation of practicerelevant learning theory
Educational Games
11
Ed Tech => Data => Better learning
LearnLab Course
Committees
LearnLab Thrusts
How you can benefit from
LearnLab
• Research
– General principles to improve learning
• Methods
– Cognitive task analysis, in vivo studies
• Technology tools
• People
– Masters students & projects
13
What instructional
strategies work best?
• More assistance vs. more challenge
– Basics vs. understanding
– Education wars in reading, math, science…
• Research on many dimensions
–
–
–
–
–
–
–
–
Massed vs. distributed (Pashler)
Study vs. test (Roediger)
Examples vs. problem solving (Sweller,Renkl)
Direct instruction vs. discovery learning (Klahr)
Re-explain vs. ask for explanation (Chi, Renkl)
Immediate vs. delayed (Anderson vs. Bjork)
Concrete vs. abstract (Pavio vs. Kaminski)
…
Koedinger & Aleven (2007). Exploring the assistance dilemma
in experiments with Cognitive Tutors. Ed Psych Review.
14
Knowledge-Learning-Instruction
(KLI) Framework: What conditions
cause robust learning
LearnLab research thrusts
address KLI elements
• Cognitive Factors
– Charles Perfetti, David Klahr
• Metacognition & Motivation
– Vincent Aleven, Tim Nokes-Malach
• Social Communication
– Lauren Resnick, Carolyn Rose
Koedinger et al. (2012). The Knowledge-LearningInstruction (KLI) framework: Bridging the science-practice
chasm to enhance robust student learning. Cognitive Science.
• Computational Modeling &
Data Mining
– Geoff Gordon, Ken Koedinger
15
Results of ~200 in vivo experiments =>
Optimal instruction depends on knowledge goals
16
Cognitive Task Analysis
using DataShop’s
learning curve tools
Without decomposition, using
just a single “Geometry” KC,
no smooth learning curve.
But with decomposition,
12 KCs for area concepts,
a smoother learning curve.
Upshot: Can automate analysis
& produce better student models
17
How you can benefit from
LearnLab
• Research
– General principles to improve learning
• Methods
– Cognitive task analysis, in vivo studies
• Technologies
– Tutor authoring
– Language processing
– Educational Data Mining
• People: Masters students & projects
18
Questions?
19
Question for you
What do you need in a learning
science professional?
20
21
Extra slides
22
Cognitive Tutor Technology
• Cognitive Model: A system that can solve problems in
the various ways students can
3(2x - 5) = 9
If goal is solve a(bx+c) = d
Then rewrite as abx + ac = d
If goal is solve a(bx+c) = d
Then rewrite as abx + c = d
If goal is solve a(bx+c) = d
Then rewrite as bx+c = d/a
6x - 15 = 9
2x - 5 = 3
6x - 5 = 9
• Model Tracing: Follows student through their individual
approach to a problem -> context-sensitive instruction
23
Cognitive Tutor Technology
• Cognitive Model: A system that can solve problems in
the various ways students can
3(2x - 5) = 9
If goal is solve a(bx+c) = d
Then rewrite as abx + ac = d
If goal is solve a(bx+c) = d
Then rewrite as abx + c = d
Hint message: “Distribute a
across the parentheses.”
Known? = 85% chance
6x - 15 = 9
Bug message: “You need to
multiply c by a also.”
Known? = 45%
2x - 5 = 3
6x - 5 = 9
• Model Tracing: Follows student through their individual
approach to a problem -> context-sensitive instruction
• Knowledge Tracing: Assesses student's knowledge
growth -> individualized activity selection and pacing
24
Cognitive Task Analysis
Improves Instruction
• Studies: Traditional instruction vs. CTA-based
– Med school catheter insertion (Velmahos et al., 2004)
– Radar system troubleshooting (Schaafstal et al., 2000)
– Spreadsheet use (Merrill, 2002)
• Lee (2004) meta-analysis: 1.7 effect size!
25
Learning Curves
26
Inspect curves for individual
knowledge components (KCs)
Many curves show a
reasonable decline
Some do not =>
Opportunity to
improve model!
27
DataShop’s “leaderboard” ranks alternative models
100s of datasets from ed tech in math, science, & language
Best model finds 18 components of knowledge
(KCs) that best predict transfer
28
28
Data from a variety of educational
technologies & domains
Statistics Online Course
English Article Tutor
Algebra Cognitive Tutor
Numberline Game
29
Model discovery across domains
Koedinger, McLaughlin, &
Stamper (2012). Automated
student model improvement.
In Proceedings of Educational
Data Mining. [Conference best
paper.]
Variety of domains
& technologies
11 of 11 improved
models
30
Data reveals students’
achievement & motivations
We have used it to
• Predict future state test scores as well
or better than the tests themselves
• Assess dispositions like work ethic
• Assess motivation & engagement
• Assess & improve learning skills like
help seeking
…
31
LearnLab courses at
K12 & College Sites
• 6+ cyber-enabled courses:
Chemistry, Physics,
Algebra, Geometry,
Chinese, English
• Data collection
– Students do home/lab work
on tutors, vlab, OLI, …
– Log data, questionnaires,
tests  DataShop
Researchers
Schools
Learn
Lab
Chemistry virtual lab
Physics intelligent tutor
REAP vocabulary tutor
32
Bridging methodology:
in vivo experiments
Lab
In Vivo
experiment Experiment
Design
Research
Randomzd
Field Trial
Setting
Lab
School
School
School
Control condition
Yes
Yes
No
Yes
Scientific
Principle
Scientific
Principle
Instr.
Solution
Instr.
Solution
$/Short
$$/Medium
$$/Long
$$$$/Long
Focus on principle
vs. on solution
(Change N things)
Cost/Duration
33
Knowledge Components
• Definition: An acquired unit of cognitive
function or structure that can be inferred from
performance on a set of related tasks
• Includes:
– skills, concepts, schemas, metacognitive strategies,
malleable habits of mind, thinking & learning skills
• May also include:
– malleable motivational beliefs & dispositions
• Does not include:
– fixed cognitive architecture,
transient states of cognition or affect
• Components of “intellectual plasticity”
Koedinger et al. (2012). The Knowledge-LearningInstruction (KLI) framework: Bridging the sciencepractice chasm to enhance robust student learning.
34
General knowledge components,
sense-making, motivation, social
intelligence
Possible domain-general KCs
• Metacognitive strategy
– Novice KC: If I’m studying an example, try to remember
each step
– Desired KC: If I’m studying an example, try to explain how
each step follows from the previous
• Motivational belief
– Novice: I am no good at math
– Desired: I can get better at math by studying & practicing
• Social communicative strategy
– Novice: If an authority makes a claim, it is true
– Desired: If considering a claim, look for evidence for &
against it
35
What is Robust Learning?
• Achieved through:
– Conceptual understanding & sense-making
skills
– Refinement of initial understanding
– Development of procedural fluency with
basic skills
• Measured by:
– Transfer to novel tasks
– Retention over the long term, and/or
– Acceleration of future learning
36
Intelligence does not improve generically
KLI summary
• Learning occurs in
components (KCs)
• KCs vary in kind/cmplxty
– Require different kinds of
learning mechanisms
• Optimal instructional
choices are dependent
on KC complexity
Koedinger et al. (2012). The Knowledge-Learning-Instruction (KLI) framework: Bridging the
science-practice chasm to enhance robust student learning. Cognitive Science.
37
Conclusions
• Learning & education are complex
systems
• Lots of work for learning science!
• Use ed tech for “basic research at scale”
=> Bridge science-practice chasm
38