Transcript Value Added Feedback

Introduction to
CEM Secondary Pre-16
Information Systems
Nicola Forster & Neil Defty
Secondary Systems Programme Managers
London, June 2011
Ensuring Fairness
Principles of Fair Analysis :
1. Use an Appropriate Baseline
2. Compare ‘Like’ with ‘Like’
3. Reflect Statistical Uncertainty
CEM Systems
Principle of Fair Analysis No1 : Use an Appropriate Baseline
The Projects
A / AS / Btec / IB etc
Year 13
Year 12
Alis
GCSE
Computer Adaptive Baseline Test
(paper test also available)
GCSE
KS4
Year 11
Yellis
Year 10
Computer Adaptive Baseline Test
or paper-based test
INSIGHT
Combines curriculum tests with
developed ability
Year 9
KS3
Year 8
MidYIS
Year 7
Computer Adaptive Baseline Test
or paper-based test
(+ additional)
The Assessments
Year
Groups
MidYIS
7
8
9
When
Term 1 + catch ups
Delivery
Paper or Computer
Adaptive
1 session
Yellis
10
11
Term 1 + catch ups
Paper or Computer
Adaptive
Includes
Developed Ability
Vocabulary
Maths
Non Verbal
Skills
Developed Ability
Vocabulary
Maths
Non Verbal
1 session
INSIGHT
9
4 week testing
window
mid April – mid May
+ catch ups
Computer Adaptive
3 or 4 sessions
Curriculum-based
Reading
Maths
Science
+ Attitudes
& Developed Ability
The Assessments
INSIGHT – curriculum-based Assessment
Maths - Number & Algebra
- Handling Data & Space
- Shapes & Measures
Science - Biology
- Chemistry
- Physics
- + Attitudes to Science
Reading - Speed Reading
- Text Comprehension
- Passage Comprehension
Additionally for INSIGHT:
Developed
Ability
- Vocabulary,
Non verbal, Skills
- Attitudinal measures
The Assessments
What is Computer Adaptive Assessment?
• Questions adapt to pupil
• Efficient
• No time Wasting
• Wider Ability Range
• More Enjoyable
• Green
Computer-adaptive vs Paper-based testing
Computer Adaptive
Number of
Students per
Test Session
Limited by number of
computers available – multiple
testing sessions
Cost
Roughly 30% cheaper than
paper-based test
Processing of Baseline feedback available
within a couple of hours of
Baseline
testing
Feedback
Preparation
Student
Experience
Must be able to install the
software or access the internet
version of the test
“Tailored” assessment
Paper-based
Can test all students in a
single session (in hall or in
form groups) or in more than
the one session
Standard cost
Takes around 2-4 weeks for
papers to be marked
No pre-test set up
All students see all questions,
irrespective of suitability
The Analysis
.
Linear Least Squares Regression
Outcome
Subject
SubjectXX
8
7
6
5
4
3
2
1
0
-ve VA
+ve VA
50
70
80
90
100
100
110
120
150
130
Baseline
Residuals
Regression Line (…Trend Line, Line of Best Fit)
Outcome = gradient x baseline + intercept
Correlation Coefficient (~ 0.7)
.
Making Predictions
Subject
SubjectXX
Outcome
e.g.
GCSE
A*
8
A7
A*
B6
C5
D
4
E3
F2
G1
U0
50
70
80
90
100
100
110
Baseline
e.g. MidYIS, INSIGHT, Yellis
standardised scores
120
150
130
Some Subjects are More Equal than Others….
Trend segments for 2009 GCSEs: drawn over 68% of Yellis intake
(mean +/-1 standard deviation)
GCSE point score (A*=8, A=7, B=6, …)
8
7
1.5 grades’ difference!
6
English
5
Maths
Art & Design
Geography
4
Biology
French
3
Business Studies
Core Science
2
Additional Science
Add Appl Science
1
20
30
40
50
60
70
80
Yellis Test Score (%)
Principle of Fair Analysis No2 : Compare ‘Like’ with ‘Like’
The Assessments
Developed Ability - Maths
Developed Ability - Vocabulary
Developed Ability - Non-verbal
Developed Ability - Skills
INSIGHT - Maths
INSIGHT - Science
INSIGHT - Reading
Baseline Assessment and
Predictive Feedback
Baseline Feedback
Nationally-Standardised Feedback
• How did your pupils perform on the assessment?
• What strengths and weaknesses do they have?
• As a group how able are they?
Predictions
• Given their performances on the test, how well might
they do at KS3 or GCSE?
Baseline Feedback
Feedback can be used at the pupil, class and cohort level.
- to guide individuals
- to monitor pupil progress
- to monitor subject-wide and department level progress
For classroom teachers, Head teachers or SMT as a quality
assurance tool.
Data can be aggregated at other levels. We support &
provide software tools to help schools to do this e.g. Paris
software.
Baseline Feedback – Test Scores
Baseline Feedback – Test Scores
·
National Mean =100, Standard Deviation =15
·
4 Performance Bands A, B, C, D
·
Averages & Band Profiles for the cohort
·
95% of scores lie between 70 & 130
·
No ceiling at 130+ or floor at 70
Baseline Feedback – Band Profile Graphs
Baseline Feedback-Gifted Pupils
Standardised Test Score
Baseline Feedback
Individual Pupil Recordsheets (IPRs)
Predictive Feedback
Predictions…...
Average performance by similar
pupils in past examinations
Predictive Feedback
Predictive Feedback
Predictive Feedback- Chances Graphs
English Language - band D
English Language - band C
50
40
31
30
24
23
20
10
Percent
Percent
50
10
9
2
1
20
20
0
0
29
30
10
0
35
40
1
2
U
G
7
6
U
G
F
E
D
C
B
A
A*
F
E
Grade
0
50
Percent
21
20
7
10
0
0
5
2
0
0
U
G
F
E
D
C
Grade
B
A
A*
50
40
24
C
English Language - band A
40
30
D
Grade
English Language - band B
Percent
1
0
B
A
A*
35
40
26
30
23
20
10
8
0
0
0
1
U
G
F
E
7
0
D
C
Grade
B
A
A*
Predictive FeedbackIndividual Chances Graphs
Point Prediction = 3.8
Chances Graphs based on Pupil’s actual Test Score NOT Band
30% chance of
a grade D – the
most likely
single grade.
70% chance of
a different grade
Chances Graphs
• The Chances graphs show that, from almost any
baseline score, students come up with almost any grade
- - - there are just different probabilities for each grade
depending on the baseline score.
• In working with students these graphs are more useful
than a single predicted or target grade
• Chances graphs serve as
– a warning for top scoring students and
– a motivator for low scoring students
Value Added Feedback
Value Added Feedback
For each subject, answer questions:
• Given their abilities, have pupils done
better or worse than expected?
• Can we draw any conclusions at the
department level?
Value Added Feedback
For each pupil in each subject:
• Raw residual = Achieved – predicted
• Standardised residual – allows fair comparison between
different subjects and years
At subject level:
• Confidence bounds are narrower with more pupils
• If average standardised residual lies within bounds you cannot
draw any conclusions
• If average standardised residual lies outside bounds you can be
confident that something significant is happening in that
subject.
Value Added Feedback
Burning Question :
What is my Value-Added Score ?
Better Question :
Is it Important ?
Principle of Fair Analysis No3 : Reflect Statistical Uncertainty
Value Added Feedback –
Scatter Plot
GCSE Points
Equivalent
GCSE English
Baseline Score
Look for Patterns…
General under- or over-achievement ?
Do any groups of students stand out ?
– high ability vs low ability ?
– male vs female ?
Value Added Feedback
Year 7 Pupil Level Residuals to GCSE
Value Added Feedback –
Standardised Residuals Graph
Standardised Residuals shown with confidence limits at 2 (95%) and 3 (99.7%) standard deviations
Standardised Residuals can be compared fairly between subjects and over years
Value Added Feedback Statistical Process Control (SPC) Chart
Subject: X
Attitudinal Surveys
Attitudinal Feedback
Your data is above the average
Your data is below the average
Your data is about the same as the average
Your data is about the same as the average
Attitudinal Feedback
Secondary Pre-16 Contact Details
Tel:
0191 334 4255
Email:
[email protected]
Web:
www.cemcentre.org