Transcript ALIS - Centre for Evaluation and Monitoring

NEW USERS OF ALIS WORKSHOP
JUNE 2011
London Conference
Geoff Davies
FIVE EXERCISES
COVERAGE
Baseline data
Initial feedback
Use of Chances graphs
‘Predictions’
Use of Scatter Graphs
Setting ‘target’ grades
Interpreting Statistical process charts
Basic ALIS is concerned mainly with value added analysis using
baseline data collected early in the sixth form
Full ALIS covers this as well as an in depth analysis of attitudes,
teaching and learning processes, resources, extra curricular
activities, and more
TWO BASELINES
• Average GCSE score --- repeatedly been found to be the
best single indicator of post 16 performance
• Computer adaptive test---CEM centre’s jewel in the crown
across all spheres of education (TDA paper test still available
for A level students)
INITIAL
FEEDBACK
Intake Profiles
GCSE
(Alis)
TDA / CABT
(Alis)
Predictions
Average performance by similar students in past exams
Predictions are available in the following forms:
• Formal Reports
• Spreadsheets
• Paris Software
Student Prediction Reports
Student ID
Name
Gender
Date of Birth
Year Group
GCSE Score
Subject
Level
GCSE Based
Predictions
Points
Grade
The ‘prediction’ for each subject is based on the previous year’s ALIS ‘trendline’ for that subject
Example:
Lisa Fry has GCSE grades 2A*, 3A, 4B and 1C
Total GCSE Points = 66
AVGCSE = 66 / 10
(i.e. 2x8 + 3x7 + 4x6 + 1x5)
= 6.6
Use ALIS subject formula (21.71 x 6.6) – 44.69 =99
(B at A level)
Chances Graphs
(A2) Geography 2008
Based on 14069 students :
Correlation coefficient = 0.72, Standard Deviation = 17.47
Regression Graph
The thick region indicates the interquartile range (i.e. the middle 50% of students).
PRACTICAL EXERCISES FOR NEW ALIS USERS
Exercise 1
Situation
You are the subject teacher and are discussing AS
target grades with individual students in the Autumn
term of Year 12. You are about to talk to Belinda who
achieved an average GCSE score of 5.9.
This gives a statistical prediction=14.75x5.9-59.57 =
27.5 UCAS points using the regression formula at AS
for this subject (Grade D/E at AS).
The computer adaptive test seems to confirm this
prediction.
Chances graph showing % of students achieving each AS grade in previous year based on 24175 students
Individual
Chances graph
example
AS GRADE
Belinda (5.9)
Other examples Chemistry different cohorts
a)
‘Most candidates with Belinda’s GCSE background
score achieved a D in my subject last year so
Belinda’s target grade should be a D’. What are the
weaknesses of this statement?
b)
What other factors should be taken into
consideration apart from chances graph data,
when determining a target grade?
Exercise 1
The strength of the chances graph is that it shows more than a bald prediction.
True, most students starting from an average GCSE score like Belinda did
achieve a D grade at AS in examinations for this subject. However the
probability of a C grade is also high since her score was not at the bottom
of this range. Students are not robots who will always fit with statistics so it
is dangerous to make sweeping statements based on one set of results.
As well as looking at the prediction you should use the chances graph as a
starting point with your professional judgement taking into account factors
such as previous performance in the subject, her attitude to work and what
she is likely to achieve based on your own experience. You might want to
start with the most popular outcome grade D and use your judgement to
decide how far up (or down!) to go. She may be a very committed student,
so a B grade may be more appropriate though A looks unlikely. If you are
using aspirational targets for psychological reasons with students then B
may be appropriate
Exercise 2
Situation
Jane has completed her 6th form studies and a review has been received
for her by the college after A level results. Choose one subject at a time
and look carefully at what happened in that subject both from a baseline of
average GCSE grades and from a baseline of the computer adaptive test.
Profile Sheet: Jane (from Average GCSE)
Year: 2007
DOB: 01/06/89
(Average GCSE = 6.00 (Band B))
FINAL RESULTS
PREDICTIONS
Review: Final_Result
Review Review
Subject
Points Grade
(A1) Health &Social Care 30.00
D
(A2) Religious Studies
80.00
C
(A2) English Literature
80.00
C
(A2) Drama &Theatre St.100.00 B
Average Average
Points Grade Residual Std. Residual
39.64
C
-9.64
-0.66
89.84
B/C
-9.84
-0.52
87.29
B/C
-7.29
-0.40
91.84
B/C
8.16
0.46
STANDARDISED
RAW
Chances Graphs - Band B from average GCSE
Individual Chances Graphs from average GCSE score
Profile Sheet: Jane from Computer adaptive test
Year: 2007
DOB: 01/06/89
(Online adaptive test = 0.11 (Band C))
Review: Final_Result
PREDICTIONS
Review
Subject
Points Grade
(A1)Health &Social Care 30.00
D
(A2) Religious Studies
80.00
C
(A2) English Literature
80.00
C
(A2) Drama &Theatre St. 100.00 B
Average Average
Points
Grade Residual St
25.97
D/E
4.03
0.24
79.01
C
0.99
0.04
74.3
C/D
5.7
0.26
85.16
B/C
14.84
0.70
RAW
STANDARDISED
a) Can you suggest possible reasons for the differences between the predictions (average
grade) made by the two baselines?
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------b) Did Jane reach her potential in all subjects?
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------c) Jane had been set aspirational targets prior to AS and A level by her teachers as below
Health and Social Care C
Drama
B
Religious Studies
B
English
B
Where these reasonable target grades?
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------Why should these grades not be used for accountability of her teachers?
-----------------------------------------------------------------------------------------------------------------------
Exercise 2
a) This student has been placed in different bands, band B from
average GCSE score and band C from the computer adaptive test.
This sometimes happens. It may be that the student had an off day
when she did the computer adaptive test or it may be that there could
have been a lot of ‘spoon feeding’ at GCSE. Jane may do better at
coursework! Even though we may not know the cause it can act as a
warning when analysing results though the predictions are not wildly
out.
b) Jane certainly reached her potential in Drama and Theatre studies
with positive standardised residuals by both methods. On the basis of
the computer adaptive test she broadly reached potential in all
subjects. On the basis of average GCSE two A level subjects were
broadly down about half a grade and she dropped a grade in the AS
c) I hope you agree these were reasonable target grades. Remember
we don’t know the student, but use the chances graphs and if these
were aspirational grades for the student then accountability of
departments staff on that basis is not appropriate, but it certainly is on
the basis of a whole classes average standardised residuals
particularly over a number of years.
Exercise 3
Scatter Graphs
(A2) Subject Report
A2 Subject Report
Year: 2008
Review: Final_Result
Baseline: Average GCSE
Predicted Grade = 23.16 * Average GCSE - 66.31
120
UCAS Points
100
120
BC
100
80
80
60
60
40
40
20
20
0
0
0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0 4.5 5.0 5.5 6.0 6.5 7.0 7.5 8.0
Average GCSE Score
Exercise 3
Hopefully this did not create too much difficulty
The arrows are vertical and join the point to the trend line.
Not easy to judge from this. It might be worth plotting
standardised residual against average GCSE score and look
for deviations from a horizontal line (more advanced)
Standardised residuals allow one subject to be compared with
another because different subjects have varying distributions of
grades. These are not dependent on the year either.
Exercise 4
Change in teaching methods used had major effect
Sept 2007 probably a factor!
Exercise 5
Some schools and colleges use average GCSE points together
with chances graphs to tweak appropriate course choices for their
students on the basis of probability of success. Perhaps more
useful is alerting year 12 tutors to the strengths and weaknesses
of students as shown by the computer adaptive test.
Using the chances
graphs
Of the students with
a point-score below
5.0 a total of 11
achieved an ‘A’
grade and a total of
562 achieved a ‘U’
grade!
Example: A/S Accounting (2005)
Using the chances
graphs
Of the students with
a point-score above
6.4 a total of 113
achieved an ‘A’
grade and a total of
27 achieved a ‘U’
grade!
Example: A/S Accounting (2005)
Comparison between
subjects
AS Biology (2005)
AS Film Studies (2005)
NEW USERS OF ALIS WORKSHOP
JUNE 2011
London Conference
Geoff Davies