Transcript Knowledge representations for

Agile Technologies for
Personalizing Instruction
Faisal Ahmad, Sebastian de la Chica, Qianyi Gu, Shaw Ketels, Ifi Okoye
Tammy Sumner, Jim Martin, Alice Healy, Kirsten Butcher, Michael Wright
Digital Learning Sciences
University of Colorado at Boulder
University Corporation for Atmospheric Research
This work is supported in part by an ICS Generalization Grant, and NSF awards #0537194 and #0734875
Central Challenge
Enable
personalized
learning, while
still supporting
recognized
learning goals
Do it at scale

How People Learn
(NRC)

Extreme Diversity
(KnowledgeWorks)

Disrupting Class
(Christiansen)

N=1, R=G
(Prahalad)
2
www.DLESE.org
Strandmaps.NSDL.org
Curriculum Customization
CLICK Personalization Service
CLICK Personalization Service
 Automatically
identify potential
learner misconceptions by analyzing
student work
 Customize the selection and
presentation of learning resources
based on identified misconceptions
 High school plate tectonics
4
Guiding Principles

Personal and intentional




Build on learner understanding
Learner control
Learning goals organize and guide
Agile technologies




Domain independent: knowledge maps for
human cognition and machine reasoning
Automatic: NLP and ML
Embeddable: web services, not applications
Open: leverage existing web content
5
6
DEMO
7
Major CLICK Components

What should students know?


What do they already understand?


Compare student and domain maps
What learning activities would be useful?


Domain knowledge map
Select resources to address misconceptions and gaps
How to embed in learning environments?

Provide web service to application and portal developers
8
Detecting Potential Knowledge Gaps
Student Essay
Digital Library Resources
(1) Student Knowledge Model
(2) Domain Competency Model
(3) Knowledge
Trace
Alignment and
Comparison
9
Human-Centered Methodology
 Expert
studies to inform algorithms
(Ahmad et al 2007)




Domain knowledge map creation
Student essay to student knowledge map
Knowledge gap diagnosis
Personal instruction plan generation
 Expert
scoring of intermediate
results
 Mixed-method learning study
10
Algorithms

Concept extraction

(de la Chica 2008)
MEAD: multi-document summarization toolkit
(Radev et al 2004)


Custom sentence scoring features: standards,
gazetteer, hypertext, content word density
Eliminate redundancy, rank and choose top 5%
Student essays – lexical chains (de la Chica 2008)
 Knowledge gaps – NLP and graph structure
comparisons (Ahmad 2008)
 Personalized information retrieval –
concept matrix (Gu 2008)

11
CLICK Personalization Web Service
Misconception diagnoses and knowledge
map generation exposed via request types
(Ahmad 2008)

Submit or remove a concept map

Construct student map from essay

Construct domain map from URLs

Get student misconceptions

Get important concepts

Get related concepts
12
Mixed-Method Learning Study

32 undergraduates



16 – CLICK to revise essays on Earthquakes
and Plate Tectonics
16 – control Digital Library environment
Data collected

original essays, revised essays, detailed screen
capture “movies”, reflective questions, factual
knowledge tests
13
14
Essay Content Revisions
Deep vs. Shallow Essay
Revisions (% of Total)
80
Shallow revisions

Digital Library
(Control)
70
60
CLICK
(Experimental)
50
40
Deep revisions

30
20
10
0
% Shallow Revisions
Copying out of resource,
Paraphrasing, Integrated
copying, Integrated
paraphrasing, Concept deletion
Integrated sentence
paraphrasing to create new
sentence, Integrated resource
paraphrasing to create new
sentence, Inferencing,
Generation
% Deep Revisions
Shallow. CLICK< Control: F (1, 27) = 3.602, p = .068
(TREND)
Deep. CLICK > Control: F (1, 27) = 5.222, p = .030
(SIG EFFECT)
Codes based on Wiley and Voss
1999, Constructing arguments from
multiple sources
15
Types of Content Revisions
Percent Omissions Corrected
50
Omissions

45
40
35
Gaps in student content
knowledge such as missing
details and missing
concepts
Incorrect Statements
30

25
Coding still underway
20
15
10
5
0
Digital Library (Control)
CLICK (Experimental)
CLICK > Control: F (1, 27) = 6.490. P = 0.17
(SIG EFFECT)
16
Process Data
70
Exploration Episodes
Digital Library (Control)
60
CLICK (Experimental)

50
Exploring learning
resources and personalized
feedback
Essay Episodes
40

30
Revising or working with
essay
Switches
20

10

0
Exploration Episodes
Essay Episodes
Switches

Exploration. CLICK>Control: F (1, 27) = 6.076, p = .02 (SIG EFFECT)
Essay.
CLICK>Control: F (1, 27) = 6.815, p = .015 (SIG EFFECT)
Switches. CLICK>Control: F (1, 27) = 6.447, p = .017 (SIG EFFECT)
Moving between essay and
exploration
Integration of content
resources and developing
essay
Recognizing need for
outside knowledge source
17
Conclusions

Learning - Initial CLICK results promising



Encourages deep content revisions
Promotes integration between information
seeking and knowledge transformation
Students more likely to recognize that they
need new knowledge, a critical element of selfdirected learning
Algorithm Generalization: Promising
results for “near” domain
 Misconception prioritization and link
generation need further work

18
Further Reading

Ahmad, F., S. de la Chica, K. Butcher, T. Sumner, and J. Martin. (2007). Towards
automatic conceptual personalization tools. In Proceedings of the 7th ACM/IEEE-CS
Joint Conference on Digital Libraries (JCDL 2007): Vancouver, Canada (June 18-23),
pp. 452-461.

Butcher, K. and S. de la Chica. (in press). Supporting student learning with adaptive
technology: Personalized conceptual assessment and remediation. In M. Banich and
D. Caccamise (Eds.), Generalization of Knowledge: Multidisciplinary Perspectives.
London, England: Taylor and Francis.

de la Chica, S., F. Ahmad, J. Martin, and T Sumner. (2008). Pedagogically useful
extractive summaries for science education. 22nd Meeting of the International
Committee for Computational Linguistics (COLING 2008).

de la Chica, S., F. Ahmad, T. Sumner, J. Martin, and K. Butcher. (2008).
Computational foundations for personalizing instruction with digital libraries.
International Journal of Digital Libraries. To appear in the Special Issue on Digital
Libraries and Education.

Gu, Q., de la Chica, S., Ahmad, F., Khan, H., Sumner, T., Martin, J., Butcher, K.
(2008). Personalizing the Selection of Digital Library Resources to Support Intentional
Learning. Research and Advanced Technology for Digital Libraries, 12th European
Conference, ECDL 2008, Aarhus, Denmark, September 14-19. Lecture Notes in
Computer Science, pp. 244-255.
19
Examples of “Good” Concepts
Plate Tectonics
Weather and Climate
Good
standalone
concept
A gradual build-up of
mechanical stress in the crust,
primarily the result of tectonic
forces, provides the source of
energy for earthquakes;
sudden motion along a fault
releases it in the form of
seismic waves.
The shape and position of
waves in the polar jet
stream determine the
location and the intensity of
the mid-latitude cyclones.
Good concept
in context
Many places near this plate
boundary are at high risk for
earthquakes, including the San
Francisco area, the Pacific
Northwest, and Alaska, yet
fully half the nation's
earthquake hazard is in
Southern California.
This energy is used to heat
the Earth's surface and
lower atmosphere, melt and
evaporate water, and run
photosynthesis in plants.
20
21
Detecting Potential Knowledge Gaps
Student Essay
Digital Library Resources
(1) Student Knowledge Model
(2) Domain Competency Model
(3) Knowledge
Trace
Alignment and
Comparison
22