Transcript Teachable agents 2

Chan & Chou’s system
• Chan, T.-W., & Chou, C.-Y. (1997). Exploring the design of
computer supports for reciprocal tutoring. International Journal
of Artificial Intelligence and Education, 8, 1-29.
•
•
•
•
•
Task domain: Designing recursive Lisp functions
Reciprocal: Yes
Communication: Weird
Expert knowledge: Yes
Evaluation: Underpowered
User interface for tutee role
• Base case vs. recursive case
• Syntax handled by GUI
• Steps, but no immediate feedback; must submit/ask
User interface for tutor role
• Shows correct code & tutee’s code
• User must localize tutee’s bug by descending
through a “fault tree”
• If user tries to descend to wrong node, its
blocked by the system
• When a leaf is reach, user selects which hint
to give the tutee
• Points are taken off for giving too specific a
hint
Evaluation’s conditions
• 5 forms of single-user instruction
– User is tutor & agent is tutee (teachable agent)
– User is tutee & agent is tutor (tutoring system) 
most motivating? Especially if mostly tutee early, like
model scaffold fade theory.
– User is tutee & agent is tutor (2nd version of tutor)
– They switch roles periodically (reciprocal tutoring)
– User works without help (no agent)  worst gains
• 2 forms of two-user instruction
– User1 is tutor, user2 is tutee & agent guides tutor
– User1 is tutor & user2 is tutee (no agent)  gains
Evaluation results
• 5 students per condition  under powered
• Teachable agent is worst condition
– User is tutor & agent is tutee
– Users reported that it was very easy to walk down
the fault tree, but they didn’t learn much
• Caution
– Giving immediate feedback on tutoring actions
invites gaming and no learning
– Did this occur with PAL?
LECOBA
• Ramirez Uresti, J.A. and B. du Boulay (2004).
“Expertise, Motivation, and Teaching in Learning
by Teaching Systems, International Journal of
Artificial Intelligence in Education 14: 67-106.
• Task domain: Boolean Algebra
• Reciprocal: user decides who will solve problem
• Communication: Editing agent’s knowledge
• Evaluation: Yes
Editing the agent’s knowledge
• User can change order of rules & how they are
applied.
Evaluation
Motivated vs. free
Results
• Underpowered: 8 per cell
• No significant differences between conditions
Findings
• The teachable agent sometimes rejected the
user’s suggestions
– If the agent thinks it knows a rule & the user
suggests a different one, it will reject the user
– This irritated the users
• The teachable agent forgot sometimes
– This surprised and irritated the users
Schwartz, Chase, Chin et al.
• Pg 6 ff: Do students treat Betty as sentient &
take responsibility for teaching her?
– 5th graders using Gameshow
– Contestant is either Betty or user
• Code attributions of K as self vs. Betty
• When given opporutnity to prepare some
more, TA group did and Student group did not
How to do this study better?
• More coding of transcripts for computer talk
• Tutoring an agent vs. tutoring a person
– Wizard of Oz; menu based communitcation
– Turing test in detail
Physiological measures e.g., pupil dialation
Does TA reflect student
knowledge?
• High correlation between student answers to
all possible questions and Betty’s answers.
• Potential alternative to standard tests
Does the TA make a difference in
learning gains?
• Using Betty vs. using just a concept map editor
pg. 13 ff
• Students in Betty’s reasoning method in that
they became better at answering long
inference chain questions
• On simple short chain questions, no difference
• On long chain questions, Betty gets better
gradually.
• Intact classes
Does SRL Betty help learning?
• 5th grades on river ecosystem for 7 class
periods
• SRL Betty
– Mr. Davis prompts
– Betty refuses to take quiz until taught enough
• Betty
– Mr. Davis provided direct hints after quiz
• Intelligent Coach
– Same as Betty without the cover story
Results
• During training SRL Betty > Betty > Coach
• During transfer SRL Betty = Betty > Coach
What did they do differently?
• During training, SRL Betty forced students to
do more debugging of their maps, so much
more time on that than Betty and Coach
groups
• During transfer, SRL Betty group continued to
do more debugging.