Transcript AAER Symposium Presentation – Nov. 2010

AAER Conference
Nov 04 2010
Symposium on
Limitations of Scientific Knowledge in Educational Research:
Bias, Non-Significant Findings & Knowledge Representation
Non-Significant Findings and a Research
Response in Self-Directed Learning
Jaya Kannan
Larry Lutsky
Yasmine Alwan
Lisa Bauer
Analyzing students’ articulation of learning goals:
How Problems in Research Led to
Solutions in Teaching
Jaya Kannan
Larry Lutsky
Yasmine Alwan
Lisa Bauer
Purpose of the presentation
 Share research work on LGs
 Discuss rubric development
 Examine what didn’t work and what was
learned
 Elicit feedback from participants
Brief History of Learning Goals
 Andragogy: theories of adult learning (Knowles)
 Learning goals individualize learning & contribute
to the development of self-directed learning
The Purpose Centered Model
 Relevance of Learning Goals
 Constructive Action teaching practices
 Seeing connections in goal setting in different
contexts
Method

Hypothesis


Procedures




Sample
Data collection
Method of analysis
Type of design
Rubric
Evolution of the Rubric
 Need to design a rubric
 Method of working – cyclical
 Establishing inter-rater reliability
 Scale items
 Operational definitions
 Math and writing – learning contexts
 Examples for each scale
Evolution of the rubric
Disagreements/clarification:
 Defining Specificity
 Defining complexity
 What constitutes measurability?
 Should we use a zero scale?
Evolution of the rubric
Resolutions
• Defining Specificity
– Break it up into content and syntax
• Defining complexity
– Establishing parameters for what is/is not a multilayer goal –case by case evaluation
• Should we use a zero scale?
– Only for measurability
• What constitutes measurability?
– Arrived at keywords
Rubric – early version
1
2
3
Specificity
Not specific – I want to learn
math
Moderately specific – I want
to understand mathematical
concepts
Highly specific – I want to learn
basic algebra. I want to get an “A”
in my course.
Complexity
Low complexity – I want to
learn how to multiply and
divide fractions.
Moderate complexity – I want
to learn about statistics so I
can analyze the data from my
CA
Highly complex – I want to learn
algebra so that I can apply the
concepts in my course and also
apply outside the class room.
Measurability
Low measurability – I want to
have more confidence in my
ability.
Moderate measurability – I
want to improve my math
skills
High measurability – I want to be
able to solve quadratic equations. I
want to be able to calculate the
mean and standard deviation. I
want to be able to determine if there
is subject-verb agreement
Rubric – final version
1= Low
2
3 = High
I want to learn basic algebra
Less broad subset
I want to solve algebraic equations
Narrow subset
Specificity –
Content
I want to learn math.
I want to understand
mathematical concepts.
Broad area superset
Specificity Syntax
I want to understand fractions
Describes an action
To improve my math skills by
completing math problems
Describes an action in detail
To learn sign numbers so I can
solve equations correctly
Describes more than one action
with detail
Complexity
I want to learn fractions
1 layer of goal
I want to learn how to
multiply and divide fractions
>=2 layers of goal
I want to learn about statistics so I
can analyze data from my CA
>= 2 layers of goal with
application
Measurability
I want to improve my math
skills
Any desirable change
Stated demonstration of
ability
I want to reduce the numbers
of errors I make while
solving math equations by the
end of the semester
Something quantifiable
By the end of the semester, I want
to reduce the number of errors I
make and complete more work
independently.
2 layers - Quantifiable, time frame
Results of data collected
Inter-Rater Agreement
(N = 4 raters)
Agreement Measure
•
•
•
Specificity
Complexity
Measurability
0.67
0.67
0.47
47.2%
58.3%
58.3%
86.1%
97.2%
94.4%
Mean Inter-Rater Difference
Percent of Pairs of Raters
with Exact Agreement
Percent of Pairs of Raters
within One Score Point of
Each Other
Results of data collected
Math Goals
N=22
9.00
8.00
7.00
6.00
5.00
4.00
3.00
2.00
1.00
0.00
7.75
7.02
6.14
7.00
Goal Rating
Mean
Overall Means-Writing
6.50
6.00
5.50
1st Goal
5.00
2nd Goal
4.50
4.00
1
(N=35)
2
( N=35)
<=4
3
5--10
>=11
# of Sessions
(N=8)
Goal number
Overall change in means over semesters
Change in first 2 means, divided by # of sessions
Overall Means - Math
Writing Goals
N=35
5.90
10
5.78
5.70
Goal Rating
5.80
5.66
5.60
5.50
5.50
5.40
5.30
1
(N=22)
2
3
(N=22)
Goal Number
8
6
4
1st Goal
2
2nd Goal
0
N=8)
<=4
5--10
# of Sessions
>=11
Results of data collected
Syntax Specificity
Writing
Writing
4
3.5
3
2.5
2
1.5
1
0.5
0
2.50
2.35
2.47
2.06
1.75
1.50
1.00
0.50
0.00
1
2
Goal number
3
1
Writing
2.50
2.31
2.00
1.97
1.71
Mean
1.50
1.00
0.50
0.00
1
2
Goal number
2
Goal number
3
Measurability -Writing
Complexity Means
Mean
2.19
2.09
2.00
Mean
Mean
Content Specificity
3
0.90
0.80
0.70
0.60
0.50
0.40
0.30
0.20
0.10
0.00
0.78
0.61
1
0.62
2
Goal number
3
Results of data collected
Content Specificity
Syntax Specificity
Math
2.05
1.60
Mean
Mean
2.10
2.03
1.66
1.65
2.16
2.15
2.05
Math
1.70
2.20
2.00
1.55
1.55
1.50
1.50
1.45
1.95
1.40
1
2
3
1
2
Goal number
Goal number
Complexity Means
Measurability -Math
1.60
1.59
Mean
Mean
Math
1.65
1.60
1.55
1.50
1.45
1.40
1.35
1.30
1.42
1
2
Goal number
3
3
0.90
0.80
0.70
0.60
0.50
0.40
0.30
0.20
0.10
0.00
0.77
0.36
0.25
1
2
Goal number
3
Flaws/Challenges in Methodology
 Applying hypothesis to problematic data
 Retroactive analysis


Abundant data
Edu purposes
 Lack of standardization


Goals
Raters
 Measurement
 Definitions of scale items
 Consensus reached through extensive discussion
What was learned
 Standardization is difficult to achieve in
teaching practice
 Should it be standardized in the first place?
 Limits in student knowledge and skills
 Actual work vs. projected work
 Teaching styles
Follow up: Next Steps
 Discussions with specialists




improve understanding (using
the rubric)
uniformity
Revising goal writing form
Building in measurability
 Generating templates of goals
Follow up: Next Steps
 Collaboration of LEC and faculty
 Participant feedback on rubric & study
 Connection between research and
teaching