Transcript Evaluation - some ideas and suggestions

Evaluation – some ideas and suggestions
Elke Wild



University of Bielefeld

Educational psychologist,
focus on motivation, selfregulated learning and designing
powerful problem-oriented
learning environments
member of the national
counseling group in PISA 2004> assessment of students‘
attitudes and perceptions of
family and school environment
I‘m in charge of the university,
responsible for the organizational
development
Evaluation - Definition


Evaluation is often seen as unnecessary or threatening
because it‘s used to justify allocations of ressouces,
responsibilities etc.
The actual purpose of evaluation, however, is to



assess the effects of your interventions (observation) and
to value them in terms of standards (by referring to objective
criterias, temporal and/or social comparisons; most common
are social comparisons by referring to matched control groups)
Thus, evaluation is a means to


obtain or preserve (attributed or self-defined) standards of
quality („Qualitätssicherung“)
to get feedback how to improve quality / desired goals or
standards.
„traditional“ vs. „innovative“ kinds of evaluation


researcher-designed field studies aiming at the
documentation of success or failure (empirical
support of theoretical assumptions)
„collaborative innovation“ as an alternative (and
probably superior) form of evaluation; it means that
practicians contribute to




the definition of goals (theoretical constructs) and criterias
(empirical indicators)
the development of conceptual features (including scripts,
materials, assessment instruments)
the implementation of measures (i.e., exchanges in the
course of teacher education)
the kind of evaluation (decisions concerning the relative
importance of goals, the dedication of time and money)
proposal held out the promise of two kinds of
evaluation


formative evaluation means an ongoing assessment;
emphasis is on the continous and qualitative feedback
concerning the process
-> evaluation as a kind of coaching, counseling
designed to optimize interventions/projects
final evaluation („summative Evaluation“) means to
assess goals attaiment at the end of an intervention in
order to come to definite conclusions concerning the
benefit of the project („product oriented evaluation“)
issues to clarify




frequency / times of measurements: before projects starts
(expectations)? after the first run (problems, challenges)? after
each run (progress, new goals?) and/ or at the end (lasting effects
in terms of implemantation, transfer, dissemination)?
relevance of content areas/dimensions: knowledge (repertory of
instructional strategies in different domains, languages etc.),
attitudes (awareness of the „european dimension“; sensitivity for
social and political issues), changes in (professional) self-concepts
and motivation to continue exchange, intra- and interdisciplinary
cooperation, use information technologies (internet, email)
frequency of feedback: third and sixth meeting (see proposal)?
interviews in two selected sites / countries: most „typical“
institutions/sites? most divergent institutional / organizational
conditions (restrictions, resources, experiences, ....) to consider a
variety of potential problems?
issues to clarify: target (groups of) persons







members of the organizing committee /coordinators,
mentors,
fellow worker,
guest families,
students (Refrendare),
students (Schüler),
teachers / teacher training institutions who need informations
concerning the establishment of intercultural exchange-projects;
strategies to distribute CD-ROMs (Nachhaltigkeit und
Verbreitung – see expert comments!)
materials / strategies:




a) to facilitate participants‘ problem solving
b) to improve the communication within INCLUDEMEmembers: utility of includeme-domain, homepages,
INTERWISE conference system;
c) to ensure dissemination: utility of handouts / reports /
„survival kids“; instruments for ongoing self-evaluation;
marketing
d) to influence probability of acceptance and transfer:
informations concerning „critical issues“ (requirements that are
necessary or desirable; factors that may moderate success /
effects),
formative evaluation – methodological suggestions




questionnaires (questionnaires, online scales, diaries/portfolio as a means to
foster self-monitoring) in order
a) to assess informations concerning similarities and differences between
locations/countries (issue of generalizability)
b) to examine effects on participants (vs. comparative students, teachers etc.)
c) to get some transfer information (changes in attitudes of staff members not
participating directly)
interviews (semi-structured, with selected individuals that represent
participation students, mentors and coordinators) in order to identifiy
challenging issues and moderating factors
expert ratings (professionals for multimedia learning environments, teachers
interested in european exchanges) to assess the utility of handouts / reports /
procedures / instruments used for evaluation
additional methods: tests (competencies)? observations (variance in
videotyped instructional strategies)? records / files (representativenes of
participants)
some final remarks...


result of the first evaluation (on base of the
application for grant) of two experts
critical feedback: only few ideas to improve longlasting effects and dissemination within
institutions and beyond


ideas how to inform and motivate colleagues,
students, classmates...
idea how to distribute CD-ROM