Transcript Document

ITS Data Collection
Framework
Capturing data based on agent
communication standard
Olga Medvedeva,
Center for Pathology Informatics,
University of Pittsburgh
July 10, 2005
Educational Data Mining Workshop
20th AAAI-05 Conference
Outline
• Need for communication standard for Intelligent
Tutoring Systems
• Existing standard for multi-agent communication
• Implementation in SlideTutor
– Communication protocol
– Data collection
– Database query tool
– Lessons learned
• Comparison with recent standardization effort
• Advantages of using the the existing standard
July 10, 2005
Educational Data Mining Workshop
20th AAAI-05 Conference
Intelligent Learning Environment
Common Base
• Underlying theory
– Cognitive tutors (Anderson et al. 1995)
– Adaptive hypermedia (Brusilovsky et al.
1996)
– Constraint-based (Mitrovic et al. 2001)
• Modules
– Expert, Student, Interface, Pedagogic
• “Single-purpose” development approach
July 10, 2005
Educational Data Mining Workshop
20th AAAI-05 Conference
Keystone – communication
standard
• Previous efforts:
– Inter-tutor communication (Ritter, Koedinger 1996;
Brusilovsky et al. 1997) one-to-one translators,
strict channel, no real protocol
– Shared resources (Koedinger et al. 1999) – limited
use: lack of standard
– DORMIN protocol (developed at CMU) – used in
commercial product
• Our approach
– Multi-agent technology
– Use existing inter-agent communication standard
July 10, 2005
Educational Data Mining Workshop
20th AAAI-05 Conference
Foundation for Intelligent Physical
Agents (FIPA)
FIPA (www.fipa.org) - collection of standards for
inter-agent communication:
• Agent Management System – manages an agent
life-cycle, maintains a registry with unique Agent
Identifier (AID)
• Transport – describes message exchange protocol:
transport type and specific address for an agent
• Agent Communication Language (ACL) –
communication specifications
FIPA was officially accepted by the IEEE as one of its
standards committees on 8 June 2005
July 10, 2005
Educational Data Mining Workshop
20th AAAI-05 Conference
FIPA Design Principals
Envelope:
Sender (locator)
Receiver (locator)
Timestamp
Message (ACL):
Sender
(AID) (ACL):
Message
Receiver (AID)
Performative
(String)
Sender (AID)
Content:
( ACR)
Receiver
(AID)
Performative (String)
Reply-to(Message
ID)
Content: ( ACR)
Reply-to (Message ID)
July 10, 2005
• Forms abstract basis for
concrete architecture
• Sets minimum required
elements
• Permits introduction of
new elements
• Permits arbitrary
content language, uses
Abstract Content
Representation (ACR)
for ACL as key-value
pairs
Educational Data Mining Workshop
20th AAAI-05 Conference
FIPA ACL Message Structure
:sender – identity of the sender
:receiver – identity of the recipient
:content – the object of the action
:performative – the type of the communicative act
Optional:
:reply-with :replay-to :in-replay-to :replay-by– replay constraints
:language – encoding schema of the content of the message
:encoding – encoding identifier
:ontology – is used to give a meaning to symbols/concepts in the content
:protocol – gives additional context for the interpretation of the message
:conversation-id – identifies the ongoing sequence of communicative act, manages the
conversation strategies
July 10, 2005
Educational Data Mining Workshop
20th AAAI-05 Conference
FIPA Performatives
•
•
•
•
•
•
•
•
•
•
•
Accept-proposal
Agree
Cancel
Call-for-proposal
Confirm
Disconfirm
Failure
Inform
Inform-if
Inform-ref
Not-understood
July 10, 2005
•
•
•
•
•
•
•
•
•
•
•
Propagate
Propose
Proxy
Query-if
Query-ref
Refuse
Reject-proposal
Request
Request-when
Request-whenever
Subscribe
Educational Data Mining Workshop
20th AAAI-05 Conference
FIPA Implementation in Java
• Java Agent Services (JAS) (www.jcp.org)
defines a set of objects and service interfaces
to support the deployment and operation of
the agents.
• Contains interfaces for building messages,
directory services and a factory for message
transfer services.
• JAS is a base for multi-agent communication
in our system
July 10, 2005
Educational Data Mining Workshop
20th AAAI-05 Conference
SlideTutor Architecture
http://slidetutor.upmc.edu
SlideTutor - an agent-based model tracing ITS for visual classification problem solving in surgical
pathology
Reasoning GUI-Tutor Communications
Client GUI Download with
Java WebStart
WEB SERVER
Protocol Collection
Filter
Login Servlet
Java Webstart
Download Manager
Internet
Project DB
Tutor Servlet
Student
Modeling System
Pedagogic Model
Production
Rules
Jess
Pedagogic Production Rules
Domain
Ontology
Slide
Pedagogic
Ontology
Protégé-2000
WholeSlide
Images
Pedagogic Ontologies
Case DB
July 10, 2005
Educational Data Mining Workshop
20th AAAI-05 Conference
Reasoning GUI-Tutor
Probabilistic
Student Model
Image
Pump
Application
Download
Student
Files
Expert
Module
Viewer GUI-IDS
Dynamic Solution Graph
Internet
IMAGE DELIVERY SYSTEM
Viewer
Servlet
2 Student Interfaces
& 1 Authoring Interface
July 10, 2005
Educational Data Mining Workshop
20th AAAI-05 Conference
Generic Representation of
Problem-Solving Space
Student Model
Case Database
Student
Student
Model
Model
Student Model
State Data
State
Slide Slide
Slide
Representation
Representation
Representation
Case Data
CaseCase
DataData
Pedagogic Task
Structure
Visual
Classification
Task Structure
Domain
Behavior
Refiner
Domain
Task
Problem
Solving
Methods
Pedagogic Layer
Expert Model
July 10, 2005
Pedagogic
Behavior
Refiner
Problem
Solving
Methods
Dermatology
Knowledge Base
Domain
Model
Pedagogic
Task
Dynamic Solution Graph
Interface
Student
Educational Data Mining Workshop
20th AAAI-05 Conference
Pedagogic
Knowledge Base
Pedagogic
Model
Pedagogic Model
Collected Data
Protocol Agent
ProblemEvent
start problem AP_77
InterfaceEvent(s)
Client Agent
Tutor Agent
ClientEvent
TutorResponse
ProblemEvent
finish problem AP_77
July 10, 2005
Project
Database
Log
Files
• InterfaceEvent – lowlevel human-computer
interactions
• ClientEvent –
collection of
InterfaceEvents that
represents an
elementary subgoal,
understood by tutor
• TutorResponse –
system response to a
ClientEvent
Educational Data Mining Workshop
20th AAAI-05 Conference
Message Example
ClientEvent
Envelope:
Sender: Client_1
Receiver: PROTOCOL
TimeStamp = 1114444377783
Message:
Sender: Concept2
Receiver: PROTOCOL
Performative: X-Created
In-reply-with: 1114444378242
Content:
Type = Finding
Label = blister
Id = Concept2
ObjectDescription = Finding.blister.Concept2
Parent = null
Input:
name = text
value = blister
name = y
value = 11808
name = x
value = 38048
name = z
value = 0.03
InterfaceEventIDS = [1114444374333,
1114444375546, 1114444376304,
1114444376798, 1114444377444]
July 10, 2005
• Envelope indicates the locators
of client and protocol agents
• 4 required key-value pairs for a
message
• Performative defines a type of
communicative act
• List of preceding InterfaceEvent
Ids:
– click on Finding button
– Click on image
– Selecting 3 times down a tree of
findings
Educational Data Mining Workshop
20th AAAI-05 Conference
Message in Depth
ClientEvent
•
Envelope:
Sender: Client_1
Receiver: PROTOCOL
TimeStamp = 1114444377783
Message:
Sender: Concept2
Receiver: PROTOCOL
Performative: X-Created
In-reply-with: 1114444378242
Content:
Type = Finding
Label = blister
Id = Concept2
ObjectDescription = Finding.blister.Concept2
Parent = null
Input:
name = text
value = blister
name = y
value = 11808
name = x
value = 38048
name = z
value = 0.03
InterfaceEventIDS = [1114444374333,
1114444375546, 1114444376304,
1114444376798, 1114444377444]
July 10, 2005
•
•
Widget object (agent) description parameters
– Type (“Button”, “Finding”)
– Label (“Next”, “Blister”)
– Id – unique within a session
– ObjectDescription – combination of
Type+Label+Id (“Finding.blister.Concept2)
– Parent – list of all parent ObjectDescriptions
for hierarchical structures
Common for ITS user action triplet
– Action = Performative
– Selection = ObjectDescription+Parent
– Input = list form Content Input
Message encoded in XML is easy to translate
into other languages (RDF, KIF, SL, etc.)
Educational Data Mining Workshop
20th AAAI-05 Conference
TutorResponse Example
• Student performance data
Envelope
Sender: TutorEngine0
Receiver: PROTOCOL
TimeStamp: 1114444379378
Message:
Sender: TutorEngine0
Receiver: PROTOCOL
Performative: FAILURE
Conversation_ID: 1114444378242
Content:
ErrorCode = 15
NextStepType = Evidence
NextStepLabel = blister
NextStepID = Concept2
NextStepParent = null
NextStepAction = DELETE
Input:
name = Messages
value = "[TEXT:There is BLISTER present, but not where
you have pointed in the image. See if you can find
where. POINTERS:[PointTo:Concept2
IsPermanent:false Method:setFlash Args:[true]]]“
name= TutorAction
value = "PointTo:Concept2 IsPermanent:false
Method:setBackgroundColor Args:[RED]"
– Performative: FAILURE – user
took incorrect step
– ErrorCode = 15 – user incorrectly
located existing finding
– Input: - contains a description of
an error message to be
presented to user
• Tutor performance data
July 10, 2005
– Best possible next step – action
expert model would take in this
problem state
Educational Data Mining Workshop
20th AAAI-05 Conference
Database Schema
• High-level static tables similar to Mostow et al. 2002 contains
Experiment, CaseList, Student, etc.
• Low-level tables for captured events, including start/end of problem and
session closely follow the FIPA standard, generic with any number of
event parameters stored in corresponding Input tables
July 10, 2005
Educational Data Mining Workshop
20th AAAI-05 Conference
Web-Based Protocol Query Tool
• Allows the user to
obtain data sets specific
to a wide range of
constraints
• Outputs to HTML file
that can be transferred
to Excel
• Query can be saved
and viewed in SQL
• Interface, Client and
Tutor events data can
be joined in different
ways
July 10, 2005
Educational Data Mining Workshop
20th AAAI-05 Conference
Query Tool Results for
Identifying Blister
InterfaceEvents
ClientEvents
TutorResponses
July 10, 2005
Educational Data Mining Workshop
20th AAAI-05 Conference
Advantages of Event-Based Data
Representation
• Usability Perspective: InterfaceEvents linked to ClientEvents
(Saadawi et al. 2005)
– How many actions were performed
– How much time was required to achieve a particular subgoal, such
as identification of Blister
– How many InterfaceEvents were unrelated to any ClientEvent
• Student Performance over time: ClientEvents linked to
TutorResponses
– Number of hints requested
– Depth of hints
– Error frequency and distribution
• Tutor Performance: NextStep fields in TutorResponses
– Compare next student actions to those predicted by tutor
July 10, 2005
Educational Data Mining Workshop
20th AAAI-05 Conference
SlideTutor Data Sharing
Limitation
• This paper and presentation have been
approved by Institutional Review Board
(IRB)
• Researcher needs to sign a Limited Use
Agreement
• There might be one agreement with
consortiums
July 10, 2005
Educational Data Mining Workshop
20th AAAI-05 Conference
Lessons Learned
For the past year our data collection framework was
used in 4 small HCI studies and one large experiment
with a total of 50 students.
• Keep data clean: ended up maintaining ‘raw’ and
‘clean’ copies of database
• Granularity of captured data: capturing of detailed
data slows the system
• Separate database for assessment: no explicit
mapping of performance on tests and in the tutoring
system
July 10, 2005
Educational Data Mining Workshop
20th AAAI-05 Conference
Data Collection Framework
Advantages
• Advantages of relational database (Mostow et al. 2002)
– Eases the analysis of the enormous volume of complex data
• Generic framework that might be adapted to other model-tracing ITS
– Adapted in the extension of SlideTutor – ReportTutor that teaches
how to write the pathology reports
• Flexibility of FIPA-based communication protocol
– Flexibility to describe interaction events
– Extendable set of performatives
– Multiple messages in one envelope, unrestricted number of input
parameters
– Potential to reference ontologies within the message
– Can be easily reused in the Data Shop
July 10, 2005
Educational Data Mining Workshop
20th AAAI-05 Conference
Data Shop Project,
Pittsburgh Science of Learning Center
(http://www.learnlab.org )
• Logging and Analysis: Tools and reports to
aid PSLC researchers and course developers
– Log the activities of the experiments to a database
– Provide the reports and queries on that
experiments
• Goal: Standardize the messaging format
among tools, tutoring translators and agents
– Message types: tool_message, tutor_message,
curriculum_message, message
• Data Shop Tutor Logging v3 released in June
2005
July 10, 2005
Educational Data Mining Workshop
20th AAAI-05 Conference
Data Shop Tool Message and SlideTutor
Interface/Client events
tool_message
attempt_id
meta (0 or 1)
user_id
session_id
time
time_zone
problem_name (0 or 1)
semantic_event
id
semantic_event_id
name
trigger
event_descriptor (0+)
event_id
selection (0+)
id
type
action (0+)
id
input (0+)
id
step (0+)
probability
ui_event
id
July 10, 2005
(1+)
Educational Data Mining Workshop
20th AAAI-05 Conference
Data Shop Tutor Message
and SlideTutor TutorResponse
meta (0 or 1)
user_id
session_id
time
time_zone
event_descriptor (0+)
event_id
problem_name (0 or 1)
semantic_event
id
semantic_event_id
name
trigger
selection (0+)
id
type
action (0+)
id
input (0+)
id
step (0+)
probability
ui_event
id
action_evaluation (0+)
current_hint_number
total_hints_available
classification
tutor_advice (0+)
skill (0+)
probability
production (0+)
July 10, 2005
step_interpretation (0+)
name (1)
value (1)
custom_field (0+)
name (1)
value (1)
Educational Data Mining Workshop
20th AAAI-05 Conference
FIPA Advantages
• FIPA as a information exchange underlying standard
– Develop a set of performatives – a controlled vocabulary for
ITS communication
– Create sharable ontologies for domain knowledge, hint
content, error categories and use ‘:ontology’ FIPA parameter
to give a meaning to the message content
– Use ‘:protocol’ parameter to identify the translator and to
preserve the internal component structure
• Syntactically aligned systems
– Ease meta-analysis for tutors with the identical performatives
– Reuse data for simulations
– Shared services for real-time interoperability
• Identifying particular help-seeking behavior
• Calculating knowledge tracing probabilities
July 10, 2005
Educational Data Mining Workshop
20th AAAI-05 Conference
Acknowledgements
Grants:
• National Library of Medicine
• National Cancer Institute
People:
•
•
•
•
•
•
Rebecca Crowley
Girish Chavan
Eugene Tseytlin
Elizabeth Legowski
Katsura Fujita
Maria Bond
July 10, 2005
Educational Data Mining Workshop
20th AAAI-05 Conference
References
•
•
•
•
•
•
•
•
Anderson JR, Corbett AT, Koedinger KR, and Pelletier R. Cognitive Tutors: Lessons learned. Journal of the
Learning Sciences 4(2): 167-207, 1995
Brusilovsky, P., Kommers, P. & Streitz, N. (Eds.) (1996) Multimedia, Hypermedia, and Virtual Reality (LNCS
Vol. 1077). Berlin: Springer-Verlag, 1996
Mitrovic A, Mayo M, Suraweera, P and Martin, B. Constraint-Based Tutors: A Success Story. In Monostori,
L. and Vancza, J. (Eds). Proceedings of the 14th International Conference on Industrial & Engineering
Applications of Artificial Intelligence and Expert Systems, Budapest, Hungary, Springer, pp. 931-940, 2001
Ritter, S. and Koedinger, K. R. (1996). An architecture for plug-in tutor agents. Journal of Artificial
Intelligence in Education, 7, 315-347
Brusilovsky, P., Ritter, S., & Schwarz, E. Distributed intelligent tutoring on the Web, Proceedings of
AIEDâ97, the Eighth World Conference on Artificial Intelligence in Education. 1997
Koedinger KR, Suthers DD, & Forbus KD. Component-based construction of a science learning space: A
model and feasibility demonstration. International Journal of Artificial Intelligence in Education: 10, 392-31,
1999
Mostow J, Beck J, Chalasani R, Cuneo A, and Jia P. Viewing and Analyzing Multimodal Human-computer
Tutorial Dialogue: A Database Approach. Proceedings of the ITS 2002 Workshop on Empirical Methods for
Tutorial Dialogue Systems, 75-84
Saadawi G, Legowski E, Medvedeva O, Chavan G, and Crowley RS. A method for automated detection of
usability problems from client user interface events. Accepted to Proceedings of the American Medical
Informatics Association Symposium 2005
July 10, 2005
Educational Data Mining Workshop
20th AAAI-05 Conference