Transcript 2014-11-10 Bridge ANA discussion

Analysing ANA
results
Nicky Roberts
Wits School of Education
Kelello Consulting
[email protected]
Context
Focus on Primary Maths project
 Errors in marking, adding and recording
results (2012)
 Using ANAs as a diagnostic tool for teaching
and development (2013)

Research in two schools in Cape Town
• Focus on Primary Maths 2012-2014
– School based maths teams
– Professional development for FP teachers in
maths
• Mini seminars
• Co teaching interventions
• Mathematical thinking course
– Responding to ANA attainment
– Sharing lessons
Responding to ANA attainment
Tracking ANA results year on year
 Question-by-question capture and analysis
 Planning for fourth term based on ANA
analysis
 Rasch analysis comparing the two schools

Background to the two schools
• 2 schools are equal partners in development project, chose
to be named at school level
• Capricorn Primary School (CPS)
– 700 learners, 2011 below provincial mean (ANA and systemics)
– Township, backyard dwellers, poverty, gangsterism and taxi
violence, drugs (Tic), alcohol (FAS). Mostly households in
poverty.
– Minority not English main lang, LOLT is English
• The Grove Primary school
– 700 learners, history of excellent attainment (ANA and
systemics)
– Suburban, model C, mostly affluent households
– Majority English main lang, LOLT is English




No anchor questions
Predictable test
Not year-on-year
comparison
Assume system is
stable, then
performance relative
to national mean is
indicator of overall
performance
ANA annual comparison
100
90
80
79
70
60
60
50
40
83
85
65
53
56
2013
2014
37
41
30
20
10
0
2012
Township school
Suburban school
National average
ANA annual comparison (pass rate)
100
90
80
70
60
50
40
30
20
10
0
99
100
84
75
59
65
38
36
2012
Township school

100
2013
Suburban school
2014
National percentage passing
Indicator of proportion of learners meeting grade level
requirements (relative to national pass rate, on that ANA)
WCED Systemic results (Grade 3)
100%
80%
61%
60%
40%
20%
0%
40%
25%
43%
67%
52%
58%
2013
2014
21%
2011
2012
Township school pass rate
Township school mean

WCED systemic (standardized questions and marking, so yearon-year comparison)
Question by question data capture

Real power for instructional design
Q1 Q2 Q3 Q4...
Learner 1
Learner 2
Learner 3
Learner 4
Learner 5
….
Correct
Incorrect
Facility
ANA analysis template
Class capture sheet
 Summary sheet
 Question summary sheet
 Spreadsheet template suburban school
 Spreadsheet template township school

What we did in 2013

Before ANA marking
 Import the learner names from EMIS into classes. One
class per sheet.
 Check formulas and that the template works as you want it
to

On the marking day
 Laptops/ computers with template in marking venue
 Capture marks for each script
 Queries – go back to teacher who was marking
What we did in 2013

After marking days
 Create grade summaries
 Compare to previous years
 Look for questions
of common difficulty
Very poor: colour code in green
 Poor: colour code code in pink
 Revise maths teaching and development plan for
fourth term

Functions used in Excel template
SUM
 AVERAGE
 MIN
 MAX
 STDEV


Some notes on using Excel

IF-THEN

COUNT IF
What we did in 2014

Repeated the descriptive statistics
 Used this to create grade level ‘mini lesson
studies’ cycles, eg:
Teaching bonds in Grade 1
 Teaching views and positional language in Grade 2
 Teaching analogue time in Grade 3

 Plan – teach- reflect cycle
 Teacher presentations
at mini conference, AMESA
regional and AMESA national
Mini conference (Day 1 internal)
Mini conference (Day 2 district)
Mini conference
AMESA regional
What we did in 2015
Repeated the descriptive statistics
 Used the data to conduct Rasch analysis:

 Misfitting items
 Biases between schools (based on DIF values)

Used this to target problematic questions and
review these qualitatively
Qualitative analysis (from initial qunat
data)
Total marks
Mean
Standard deviation
Mean percentage
Number writing
Number getting 0
Percentage getting 0
1
0.02
0.14
2%
80
79
99%
Identified as a major
area of weakness
Errors on Q 25.1

Some completely off

Most common error was to break up number
rather than converting
Conclusion





ANA is a major investment in SA education
It gives annual standardised data across schools
If anchor questions were included, it would also give
year-on year comparisons
It gives schools rich data for error analysis (if item
level analysis is conducted) which can lead to lesson
study cycles/ interventions
Teachers need support in marking ANAs (again
directs attention to the mathematics) and reporting
common errors to district/ province
[email protected]