Transcript Slide - WordPress.com

Competency Based Learning in the
Web of Learning Data
Guillaume Durand, Nabil Belacel, Cyril Goutte
April 11th, 2016
LILE Workshop, Montreal, Canada
Outline
• Introduction
• Building (Web) Learning Path
• Constraints
• Heuristic and Graph theory
• Web to Web of Learning Data
• Beyond metadata, skills constraints
• Benefit of Educational data Mining technologies
• Limits
• Where to go next ?
• Semantic approaches, etc.…
2
Introduction
• Huge quantity of heterogeneous web data already
used for learning purpose
• How to enrich the learning experience?
3
Introduction
Enriching the web experience to get a Web
Learning experience.
Many many many many challenges to address to get there. We
picked two :
• Learning is a journey…..How to get it planned?
• Learning leads to knowledge acquisition …. How to link knowledge and
web data?
.
4
Building (Web) Learning Path
Constraints
Learning constraints:
Provide learners with meaningful
sequence of interdependent
learning material to reach a
knowledge level.
Web constraints:
Pick a consistent sequence among
billions of candidates.
5
Building (Web) Learning Path
Heuristic and Graph theory model (1)
• Let G = (V, E) be a directed graph V as
learning object set, E as competency
dependencies
• 𝐶𝑝𝑟𝑒 is a set of the competencies required by
vertex v
• 𝐶𝑝𝑜𝑠𝑡 is a set of competencies offered by
vertex v
• 𝐶𝑝𝑟𝑒 𝑣 ⊆ 𝐶𝑝𝑜𝑠𝑡 𝑢 ⇒ 𝐴𝑟𝑐 𝑢, 𝑣
• 𝐴𝑟𝑐 𝑢, 𝑣 ⇔ 𝐶𝑝𝑟𝑒 𝑣 ⊆ 𝐶𝑝𝑜𝑠𝑡 𝑢 ∪ 𝐶𝑙𝑒𝑎𝑟𝑛𝑒𝑟 𝑡
6
Building (Web) Learning Path
β6
Heuristic and Graph theory model (2)
v1
• Used a two steps approach to
search for learning path
• Solution space reduction
(Cliques)
• Heuristic / Binary integer
programming (BIP) solver
7
A65
E63,5
v2
T3,2,47 U50
↑ 6
↑ 3,5
↑ 0, 7
v3
L0,78,9
α8,9
I79
K08
↑ 8, 9
α: Fictitious LO with initial learner competency
β: Fictitious LO with targeted learner competency
LO list of gained competencies LO list of prerequisite competencies
Building (Web) Learning Path
Heuristic and Graph theory model (3)
Lesson learned:
Graph topology is key in algorithms
performance.
Trade off between accuracy and
performance.
8
Web to Web of Learning Data
Beyond metadata, skills constraints
• How to feed our graph based model
• Need to link learning material with competencies and skills
𝐶𝑝𝑟𝑒 (competencies required)
𝐶𝑝𝑜𝑠𝑡 (is a set of competencies offered)
?????
9
Web to Web of Learning Data
Benefit of Educational data Mining (EDM) technologies (1)
EDM (and Psychometrics ) people manipulate Q-matrices…
Q-matrix = mapping between skills and test items
10
Web to Web of Learning Data
Benefit of EDM technologies (2)
• Probabilistic diagnostic models (AFM(Cen), DINA(Junker),…) use
Q-matrix to make inferences on student success…
• Often heavy models…with computational challenges…
• Q-matrix is not always evaluated but used as ground truth...
• However, there is a race to improve/generate automatically Qmatrices and some people use matrix factorization (Desmarais,
Sun, Su, etc..) .
• Matrix factorization (popularized NetFlix prize), great for latent
factors like competencies and Q-matrix structure.
11
Web to Web of Learning Data
Benefit of EDM technologies (3)
Using Q-matrices approach we did two things:
• Developed a method to evaluate the strength of the mapping
between test items and the assumed tested skills.
• Used mapping (in future development maybe one day
automatically generated) to automatically name competencies
12
Web to Web of Learning Data
Benefit of EDM technologies (4)
Skills
students
Q-matrix evaluation.
𝑅 ≈𝑄×𝑆
where
• R is a result matrix (students x
items)
• Q is a Q Matrix (items x skills)
• S is the student skills mastery
profile (skills x students)
Used Weighted least squares to
cross-validate Q-matrices (injecting
missing values)
13
Items/questions
111110000011110
111110000011110
111110000011110
111111111111111
111111101111111
111110000011110
111111101111111
111110000011110
000001101100001
111111101111111
111111101111111
111110000011110
111111101111111
111111101111111
111110000011110
111111101111111
111111101111111
111111101111111
111111111111111
111111111111111
𝑅
=
00010110
00010010
00010010
01101010
01010011
00000010
11000010
00000010
01000000
01001011
01001010
00000011
01011010
01000010
10000010
01000010
01001010
01001110
11101010
01101010
𝑄
students
x
000000000000000
000001101110001
000010110100110
000000000001000
000000000000000
000000000010000
111110000011110
000000000000100
𝑆
Web to Web of Learning Data
Benefit of Educational data Mining
technologies (5)
Limited predictive quality RMSE~.30,
MAE~.20.
 Accuracy of expert description of skills ???
Opportunities
Cen, H., Koedinger, K., Junker, B. Learning Factors
Analysis - A General Method for Cognitive Model
Evaluation and Improvement. 8th International
Conference on Intelligent Tutoring Systems. 2006.
1. Convert a whole number to a fraction,
2. Separate a whole number from a
fraction,
3. Simplify before subtracting,
4. Find a common denominator,
5. Borrow from whole number part,
6. Column borrow to subtract the second
numerator from the 1rst,
7. Subtract numerators,
8. Reduce answers to simplest form.
14
Web to Web of Learning Data
Competency naming
Used a simple probalistic model to find a set of keywords that are
more relevant to skills
On PSLC Datashop dataset , retrieved unknown skill labels in close to
50% of cases.
15
Skill label
# items
Top 10 keywords
_identify-sr
52
phishing email scam social learned indicate legitimate engineering antiphishing indicators
_p2p
27
risks mitigate applications p2p protected law file-sharing copyright illegal
material
_print_quota03
12
quota printing andrew print semester consumed printouts longer unused cost
_vpn
11
vpn connect restricted libraries circumstances accessing need using university
resources
Web to Web of Learning Data
Limits and outcomes
Q-matrices models and evaluation are uni-relational…multi-relational
(skills/competencies required and offered)…
Not there yet to generate Q-matrices automatically….in the meantime
expert ones are not perfect…
Content of the learning material can bring interesting hints on the
competency potentially detected and/or can bring help to categorize
them.
16
Where to go next?
• Probabilistic approaches:
Competencies/topics are both latent features…..Latent Dirichlet
Allocation, latent semantic extraction, etc..
• EDM approaches:
Replace success/failure by new metrics on web learning ????
Integrate multi-relational dimension in new deterministic, probalistic
models (Update of AFM, Matrix Factorization).
• Competencies relationship extraction scalability
17
Thank you
Guillaume Durand, Nabil Belacel, Cyril Goutte
April 11th, 2016
LILE Workshop, Montreal, Canada
18