Transcript No Slide Title

Andreas Schleicher
National Conference on
Next Generation Assessments
Washington, 8-9 March 2010
Future of Assessments
11
The future of Assessments
Lessons learned internationally
Washington, 9 March 2010
Andreas Schleicher
Head, Indicators and Analysis Division
OECD Directorate for Education
The future of assessments
Andreas Schleicher
r
National Conference on
Next Generation Assessments
Washington, 8-9 March 2010
Future of Assessments
22
Or the Alchemists’ Stone?
The Holy Grail?
Know why you are looking


r
Know what you are looking for




r
A new assessment culture
Responsive to changing skill requirements
Capitalising on methodological advances
Not sacrificing validity gains for efficiency gains
Know how you will recognise it when you find it


r
You cannot improve what you cannot measure
The yardstick for success is no longer
just improvement by national standards but the
best performing education systems globally
Gauging predictive validity
Impact on improving learning and teaching
Implications and lessons learned .
Andreas Schleicher
National Conference on
Next Generation Assessments
Washington, 8-9 March 2010
Future of Assessments
33
The old bureaucratic system
The modern enabling system
Hit and miss

Universal high standards
Uniformity

Embracing diversity
Provision

Outcomes
Bureaucratic look-up

Devolved – look outwards
Talk equity

Deliver equity
Prescription

Informed profession
Conformity

Ingenious
Australia
Austria
Czech Republic
Denmark
Finland
Germany
Greece
30000
Hungary
Iceland
25000
Ireland
Italy
20000
Japan
Netherlands
New Zealand 15000
Norway
Poland
10000
Portugal
Slovak Republic
5000
Spain
Sweden
United Kingdom 0
United States
1995
Cost per student
Expenditure per student at tertiary level (USD)
A world of change – higher education
0
Graduate supply
10
20
30
40
50
60
70
Tertiary-type A graduation rate
Australia
Austria
Czech Republic
Denmark
Finland
Germany
Greece
30000
Hungary
Iceland
25000
Ireland
Italy
20000
Japan
Netherlands
New Zealand 15000
Norway
Poland
10000
Portugal
Slovak Republic
5000
Spain
Sweden
United Kingdom 0
United States
1995
Cost per student
Expenditure per student at tertiary level (USD)
A world of change – higher education
0
United States
Finland
Graduate supply
10
20
30
40
50
60
70
Tertiary-type A graduation rate
Australia
Austria
Czech Republic
Denmark
Finland
Germany
Greece
30000
Hungary
Iceland
25000
Ireland
Italy
20000
Japan
Netherlands
New Zealand 15000
Norway
Poland
10000
Portugal
Slovak Republic
5000
Spain
Sweden
United Kingdom 0
United States
A world of change – higher education
Expenditure per student at tertiary level (USD)
2000
Australia
Finland
United Kingdom
0
10
20
30
40
50
60
70
Tertiary-type A graduation rate
Australia
Austria
Czech Republic
Denmark
Finland
Germany
Greece
30000
Hungary
Iceland
25000
Ireland
Italy
20000
Japan
Netherlands
New Zealand 15000
Norway
Poland
10000
Portugal
Slovak Republic
5000
Spain
Sweden
United Kingdom 0
United States
A world of change – higher education
Expenditure per student at tertiary level (USD)
2001
0
10
20
30
40
50
60
70
Tertiary-type A graduation rate
Australia
Austria
Czech Republic
Denmark
Finland
Germany
Greece
30000
Hungary
Iceland
25000
Ireland
Italy
20000
Japan
Netherlands
New Zealand 15000
Norway
Poland
10000
Portugal
Slovak Republic
5000
Spain
Sweden
United Kingdom 0
United States
A world of change – higher education
Expenditure per student at tertiary level (USD)
2002
0
10
20
30
40
50
60
70
Tertiary-type A graduation rate
Australia
Austria
Czech Republic
Denmark
Finland
Germany
Greece
30000
Hungary
Iceland
25000
Ireland
Italy
20000
Japan
Netherlands
New Zealand 15000
Norway
Poland
10000
Portugal
Slovak Republic
5000
Spain
Sweden
United Kingdom 0
United States
A world of change – higher education
Expenditure per student at tertiary level (USD)
2003
0
10
20
30
40
50
60
70
Tertiary-type A graduation rate
Australia
Austria
Czech Republic
Denmark
Finland
Germany
Greece
30000
Hungary
Iceland
25000
Ireland
Italy
20000
Japan
Netherlands
New Zealand 15000
Norway
Poland
10000
Portugal
Slovak Republic
5000
Spain
Sweden
United Kingdom 0
United States
A world of change – higher education
Expenditure per student at tertiary level (USD)
2004
0
10
20
30
40
50
60
70
Tertiary-type A graduation rate
Australia
Austria
Czech Republic
Denmark
Finland
Germany
Greece
30000
Hungary
Iceland
25000
Ireland
Italy
20000
Japan
Netherlands
New Zealand 15000
Norway
Poland
10000
Portugal
Slovak Republic
5000
Spain
Sweden
United Kingdom 0
United States
A world of change – higher education
Expenditure per student at tertiary level (USD)
2005
0
10
20
30
40
50
60
70
Tertiary-type A graduation rate
Australia
Austria
Czech Republic
Denmark
Finland
Germany
Greece
30000
Hungary
Iceland
25000
Ireland
Italy
20000
Japan
Netherlands
New Zealand 15000
Norway
Poland
10000
Portugal
Slovak Republic
5000
Spain
Sweden
United Kingdom 0
United States
A world of change – higher education
Expenditure per student at tertiary level (USD)
2006
United States
Australia
Finland
0
10
20
30
40
50
60
70
Tertiary-type A graduation rate
Education Indicators
Programme
2009 edition of
Education at a Glance
13
13
Latin America then…
GDP/pop
1960
Years
schooling
Asia
1891
4
Sub-Saharan Africa
2304
3.3
MENA
2599
2.7
Latin America
4152
4.7
Europe
7469
7.4
11252
9.5
Orig. OECD
Hanushek 2009
Education Indicators
Programme
2009 edition of
Education at a Glance
14
14
Latin America then and now…
GDP/pop
1960
Years Growth 1960schooling
2000
GDP/pop
2000
Asia
1891
4
4.5
13571
Sub-Saharan Africa
2304
3.3
1.4
3792
MENA
2599
2.7
2.7
8415
Latin America
4152
4.7
1.8
8063
Europe
7469
7.4
2.9
21752
11252
9.5
2.1
26147
Orig. OECD
Hanushek 2009
15
15
Latin America then and now…
Education Indicators
Programme
2009 edition of
Education at a Glance
Why quality is the key
GDP/pop
1960
Years Growth 1960schooling
2000
GDP/pop
2000
PISA
score
Asia
1891
4
4.5
13571
480
Sub-Saharan Africa
2304
3.3
1.4
3792
360
MENA
2599
2.7
2.7
8415
412
Latin America
4152
4.7
1.8
8063
388
Europe
7469
7.4
2.9
21752
492
11252
9.5
2.1
26147
500
Orig. OECD
Hanushek 2009
National Conference on
Next Generation Assessments
Washington, 8-9 March 2010
Andreas Schleicher
Future of Assessments
16
16
Know what you are looking for
The Holy Grail was a well-described object,
and there was only one true grail…
National Conference on
Next Generation Assessments
Washington, 8-9 March 2010
Andreas Schleicher
Future of Assessments
17
17
Schooling in the
medieval age:
The school of
the church
National Conference on
Next Generation Assessments
Washington, 8-9 March 2010
Andreas Schleicher
Future of Assessments
18
18
Schooling in the
industrial age:
Uniform learning
Andreas Schleicher
National Conference on
Next Generation Assessments
Washington, 8-9 March 2010
Future of Assessments
19
19
Schooling in the
industrial age:
Uniform learning
The challenges today:
Universal quality
Motivated and self-reliant citizens
Risk-taking entrepreneurs, converging and
continuously emerging professions tied to
globalising contexts and technological
advance
20
20
How the demand for skills has changed
Mean task input as percentiles of the 1960 task distribution
Andreas Schleicher
National Conference on
Next Generation Assessments
Washington, 8-9 March 2010
Future of Assessments
Economy-wide measures of routine and non-routine task input (US)
Routine manual
65
60
Nonroutine manual
55
Routine cognitive
50
Nonroutine analytic
45
40
1960
1970
(Levy and Murnane)
1980
1990
2002 Nonroutine interactive
The dilemma of assessments:
The skills that are easiest to teach and
test are also the ones that are easiest to
digitise, automate and outsource
Andreas Schleicher
National Conference on
Next Generation Assessments
Washington, 8-9 March 2010
Future of Assessments
23
23
Education today needs to prepare students…
… to deal with more rapid change than ever before…
… for jobs that have not yet been created…
… using technologies that have not yet been invented…
… to solve problems that we don’t yet know will arise
It’s about new…

Ways of thinking
– involving creativity, critical thinking, problem-solving and
decision-making

Ways of working
– including communication and collaboration

Tools for working
– including the capacity to recognise and exploit the
potential of new technologies

The capacity to live in a multi-faceted world as
active and responsible citizens.
Mathematics in PISA
Andreas Schleicher
The real world
National Conference on
Next Generation Assessments
Washington, 8-9 March 2010
Future of Assessments
24
24
The mathematical World
Making the problem amenable
to mathematical treatment
A model of reality
Understanding,
structuring and
simplifying the
situation
A mathematical
model
Using relevant
mathematical
tools to solve
the problem
A real situation
Validating
the results
Mathematical
results
Real results
Interpreting
the mathematical results
Andreas Schleicher
National Conference on
Next Generation Assessments
Washington, 8-9 March 2010
Future of Assessments
25
25
National assessment systems differ…
By whom?
•Evaluators
•Users of feedback
•Evaluation
agencies
Student
assessment
How? Methods
and procedures,
mix of criteria
and instruments
•Mapping of
feedback to
different units
Classroom
Teacher appraisal
Who isSchool
assessed
School evaluation
System
System assessment
What?
•Inputs
•Processes
•Outcomes
With whom?
•Agents involved
For what? E.g.
•Accountability
•Improvement
Assessment cultures
Putting the pressure on top of the education system is
the easy part, building capacity is harder
Participative/internal
Andreas Schleicher
National Conference on
Next Generation Assessments
Washington, 8-9 March 2010
Future of Assessments
26
26
Interactive, reflective,
critical friend
Survey
Summative
Formative
Standardised
assessment
Inspectorate
Administrative external
Andreas Schleicher
National Conference on
Next Generation Assessments
Washington, 8-9 March 2010
Future of Assessments
28
28
…but there are global trends
• Multi-layered,
Understanding what the
assessment assessment
reveals
coherent
systems
about students’ thinking to shape better
from
classrooms
to
schools
to
• Integrate,
synthesizeto
andregional
creatively apply
opportunities
for student
learning
content can
knowledge
in novelthat…
situations
• national
Responding to
enhance
toassessments
international
levels
Activate
as owners of their own
student learning if• tasks
arestudents
well crafted
learning
activate
students
as levels
learning of the
 Support improvement
of
learning
at all
to
incorporate principles
of and
learning
resources for one another
education
system
• Capitalise on improved data handling tools
and
technology connectivity to combine
 Are largely performance-based
formative and summative assessment
 Make students’
thinking
visible
interpretations
for a more
complete
pictureand allow for
of student
learning
divergent
thinking



Are adaptable and responsive to new developments
Add value for teaching and learning by providing
information that can be acted on by students,
teachers, and administrators
Are part of a comprehensive and well-aligned
continuum, communicate what is expected and hold
relevant stakeholders accountable .
Andreas Schleicher
National Conference on
Next Generation Assessments
Washington, 8-9 March 2010
Future of Assessments
29
29
Know how you will recognise it
when you find it
The Alchemists’ stone was to be recognised
by transforming ordinary metal into gold…
Andreas Schleicher
National Conference on
Next Generation Assessments
Washington, 8-9 March 2010
Future of Assessments
30
30
Increased likelihood of postsec. particip. at age 19/21
associated with PISA reading proficiency at age 15 (Canada)
after accounting for school engagement, gender, mother tongue,
place of residence, parental, education and family income
(reference group PISA Level 1)
Odds ratio
College entry
20
18
16
14
12
10
8
6
4
2
0
Age 19
Age 21
Age 21
Level 2
Level 3
Level 4
Level 5
Relationship between test performance
31
31
Annual improved GDP from raising performance by 25 PISA points
35%
Percent addition to GDP
Andreas Schleicher
40%
National Conference on
Next Generation Assessments
Washington, 8-9 March 2010
Future of Assessments
and economic outcomes
30%
25%
20%
15%
10%
5%
0%
2010 2020 2030 2040 2050 2060 2070 2080 2090 2100 2110
Andreas Schleicher
National Conference on
Next Generation Assessments
Washington, 8-9 March 2010
Future of Assessments
33
33
Implications and lessons learned
The medieval Alchemists’ followed the dictates of a wellestablished science but that was built on wrong foundations
The search for the Holy Grail was overburdened by false
clues and cryptic symbols
Andreas Schleicher
towards outcome driven reform
National Conference on
Next Generation Assessments
Washington, 8-9 March 2010
Future of Assessments
34
34
From assessment-inhibited practice
Strong focus on processes
Good will
and trust
Weak outcomebased management
Integrated quality
management
Strong outcome-based
management
Deprivation
External control,
uninformed prescription
Weak focus on processes
Some criteria used in the world
Coherence
Andreas Schleicher

National Conference on
Next Generation Assessments
Washington, 8-9 March 2010
Future of Assessments
35
35

Built on a well-structured conceptual base—an expected
learning progression—as the foundation both for large
scale and classroom assessments
Consistency and complementarity across administrative
levels of the system and across grades
Comprehensiveness


Using a range of assessment methods to ensure
adequate measurement of intended constructs and
measures of different grain size to serve different
decision-making needs
Provide productive feedback, at appropriate levels of
detail, to fuel accountability and improvement decisions
at multiple levels
Continuity

A continuous stream of evidence that tracks the
progress of both individual students .
Learning targets
Andreas Schleicher

National Conference on
Next Generation Assessments
Washington, 8-9 March 2010
Future of Assessments
37
37
Understanding learning progressions
Defining what mastery means for a given skill level
Progress variables


Delineate a pathway that characterise the steps that
learners typically follow as they become more proficient
Evaluation of students reasoning in terms of the
correctness of their solutions as well as in terms of
their complexity, validity and precision
Levels of achievement

Describing the breadth and depth of the learner’s
understanding of the domain at a particular level of
advancement
Learning performances

The operational definitions of what student’s
understanding would look like at each of the stages of
progress .
Wilson, ATC21S
OECD Level 2
39
39
Andreas Schleicher
Future of Assessments
Students can determine if
scientific
measurement can be
Context
applied
to a given variable in an
- Personal
investigation. Students can
- Social/public
appreciate the relationship
- Global
between
a simple model and
the phenomenon it is modelling.
OECD Level 6
Students can demonstrate
ability to understand and
articulate the complex
modelling inherent in the
design of an investigation.
Competencies
National Conference on
Next Generation Assessments
Washington, 8-9 March 2010
scientific
Students can recall -Identify
an
Students issues
can draw on
appropriate, tangible,
a range ofscientifically
abstract scientific
-Explain phenomena
scientific fact applicable in a
knowledge and concepts and
-Use
scientific
evidence
simple and straightforward
the relationships between
context and can use it to
these in developing
explain or predict an outcome. explanations of
processes
Identifying
Recognising issues that can
be investigated
scientifically
Identifying keywords in a
scientific investigation
Recognising the key
features of a scientific
investigation
Explaining
Applying knowledge of
science in a situation
Describing or interpreting
phenomena scientifically or
predicting change
Students demonstrate
Students
can point to an
Knowledge
Attitudes
Using evidence
ability to compare and
obvious feature in a simple
-Knowledge of science differentiate among-Interest in
science scientific
Interpreting
table in support of a given
evidence
and drawing
competing explanations
by
-Knowledge
about
-Support
for
scientific
enquiry
statement.
They are
able science
to
conclusions
examining supporting
recognise if a set of given
-Responsibility
Identifying the
evidence.
They
can
formulate
characteristics apply to the
assumptions, evidence and
arguments by synthesising
function of everyday
reasoning behind conclusions
evidence from multiple
artifacts.
sources.
Andreas Schleicher
National Conference on
Next Generation Assessments
Washington, 8-9 March 2010
Future of Assessments
40
40
Some methodological challenges
Can we sufficiently distinguish the role of context
from that of the underlying cognitive construct ?
Do new types of items that are enabled by
computers and networks change the constructs
that are being measured ?
Can we drink from the firehose of increasing data
streams that arise from new assessment modes ?
Can we utilise new technologies and new ways of
thinking of assessments to gain more information
from the classroom without overwhelming the
classroom with more assessments ?
What is the right mix of crowd wisdom and
traditional validity information ?
How can we create assessments that are
activators of students’ own learning ?
Wilson, ATC21S
41
41 A real-time assessment
High policy value
Quick wins
Andreas Schleicher
National Conference on
Next Generation Assessments
Washington, 8-9 March 2010
Future of Assessments
haves
environment thatMust
bridges
the
gap between formative and
Examine individual, institutional
summative assessment .
and systemic factors associated
Extending the range of
with performance
competencies through which
quality is assessed
Monitor educational progress
Measuring growth in learning
Low feasibility
High feasibility
Assuming that every new
skill domain is orthogonal
to all others
Establish the relative standing
of students and schools
Money pits
Low-hanging fruits
Low policy value
Seeing Japanese schools
through the prism of PISA
42
42
Getting the sequencing right
Phases of development
Poor  Adequate
Main focus of
assessment
PISA
OECD Programme for
International Student Assessment
Role of government
Role of professions
Nature of relationship
between government and
professions
Main outcomes
Adequate  Good
Good  Great
•World class
performance.
•Continuous learning
and innovation
.
•Transparency
•Spreading best
practice
.
•Regulating
•Capacity-building
. •Enabling
•Incentivising
•Implementing
•Accepting evidence
•Adopting minimum
standards
•Accommodating
•Evidence-based
•Adopting best
practice
•Leading
•Evidence-driven
•Achieving high
.
reliability and
innovation
•Top-down
•Antagonistic
•Negotiated
•Pragmatic
•Principled
. •Strategic partnership
•Tackling
underperformance
•Prescribing
•Justifying
.
.
•Improvement in
outcomes
•Reduction of public
anxiety.
•Steady improvement •Consistent quality
•Public engagement
•Growing public
satisfaction
. and co-production
.
.
.
Andreas Schleicher

National Conference on
Next Generation Assessments
Washington, 8-9 March 2010
Future of Assessments
43
43
www.oecd.org; www.pisa.oecd.org
– All national and international publications
– The complete micro-level database


email: [email protected]
Twitter: @SchleicherEDU
Thank you !
… and remember:
Without data, you are just another person with an
opinion