Transcript Chapter14_Outline

Chapter 14
Collecting Evaluative Information:
Design, Sampling and Cost
Choices
OST Certificate Program
Program Evaluation Course
Fitzpatrick, Sanders, and Worthen. (2004). Program Evaluation. Boston, MA: Pearson
Education Inc.
Planning Data Collection






What information is necessary to answer the
questions?
What design or designs are most appropriate for
answering each question?
What sampling strategy is most appropriate for the
purposes?
Who has this information?
Who will collect it? When? How? With what
training and instructions?
What statistical tests will be used?
Designs for Collecting Causal and
Descriptive Information

Case Study Design
Characteristics:



In-depth descriptive data collection and analysis
A focus on a selected case(s)
Useful when it is important to study the program in its “natural setting,” when context is
important, or multiple sources of evidence are sought
Benefits:





Thick description (a term used in qualitative research to characterize the richness of
program and outcome descriptions that arise from qualitative methods like case studies)
Does not require manipulation or control
Triangulation of data
Portrays the multiplicity of causes associated with various outcomes
Allows for knowledge of unintended outcomes
Disadvantages:




Cannot generalize
Evaluator bias
Time- consuming and costly
Amount of data to be analyzed may be prohibitive (because of skill or time)
Designs for Collecting Causal and
Descriptive Information (cont’d)

Experimental Designs
 Post-Test Only Control Group Design
Characteristics:


Two groups may be randomly assigned, with one group experiencing the program
Both groups given the post-test at the same time, following the intervention
Benefits:





Controls “other factors” (ex: motivation to learn…both groups are equally motivated because they
were randomly assigned)
Can be accomplished with one data collection time
Can gather information as part of change event
Rules out testing and instrumentation effects
Allows actual comparison of program vs. “not program” (also referred to as a “control group,” the
“not program” scenario is the alternative activities in which children would participate if they
were not in your program)
Disadvantages:



Assumes/requires random assignment to groups
Raises ethical problems of withholding program from control group
Does not rule out possibility that everyone knew the information before the intervention
Designs for Collecting Causal and
Descriptive Information (cont’d)

Experimental Designs (cont’d)

One Group Pre-test/Post-test Design
Characteristics:

Involves data collection before and after the program

Typically involves collecting post program data on the last day of the
program or immediately following
Benefits:

Can be simple and cost effective

Can reduce time and costs for data collection

Can gather as part of change event

Compares actual pre- and post- program attitudes, skills
Disadvantages:

Cannot rule out testing effects (content of the test may change participants)

Vulnerable to attrition

Added cost of two testing periods
Designs for Collecting Causal and
Descriptive Information (cont’d)

Experimental Designs (cont’d)

Pre-test/Post-test Control Group Design
Characteristics:
 Two groups randomly assigned to program or “not program” condition
 Pretest allows the evaluator to determine empirically whether the two
groups are similar before the program
Benefits:
 Controls for other factors
 Pre-program measure
 Compares actual pre- and post-program skills, etc.
Disadvantages:
 Resources required for two testing sessions
 Testing and instrumentation effects
 Attrition
 Difficulty in obtaining a control group
Designs for Collecting Causal and
Descriptive Information (cont’d)

Descriptive Designs

Time Series Design
Characteristics:



Involves repeated data collection to establish a stable baseline for comparison
Graph the results obtained each time
Look for dramatic changes in the trend, particularly before and after the program
was introduced
Benefits:



Controls for the effects of history
Provides evidence over time
Establishes a baseline of performance with which to compare post-program
performance
Disadvantages:

Requires resources for repeated data collection


Testing and instrumentation effects
May experience attrition
Designs for Collecting Causal and
Descriptive Information (cont’d)

One Shot Designs
Characteristics:

Commonly used in evaluating learning, performance, and change
interventions (ex: measuring performance after training)

Typically involves a post-event survey
Benefits:

Simple

Reduces time and costs for data collection

Can gather data as part of the change event
Disadvantages:

Assumes that the measurement is related to the intervention rather than
some other factor

Assumes that positive reactions and knowledge tests lead to behavior
changes

Findings cannot be generalized
Designs for Collecting Causal and
Descriptive Information (cont’d)

Retrospective Designs
Characteristics:
Similar to One Shot Design

Participants report retrospectively on their attitudes, knowledge, or skills (ex: asking
respondents how they would have handled a situation before training, and how they would
handle it now)
Benefits:

Simple and cost effective

Reduces time and costs for data collection

Can gather information as part of change event

Compares pre- and post-tests

Avoids attrition of pre- group

Decreases likelihood of testing effects
Disadvantages:

Assumes that measurement is related to the intervention rather than some other factor

Possible distortions in retrospective reports because of memory or current attitudes/beliefs

Sampling Terms

Sampling - the method the evaluator uses to select the units
(people, classrooms, schools, etc.) of study

Population - group about which you wish to make statements

Sampling unit - element or collection of elements in the target
population

Sampling frame - the source from which sampling units are
defined or listed and from which a set of units is selected
 the degree to which the sampling frame includes the entire target
population, and no others, greatly influences the accuracy of the
sampling process
Why Take a Sample?

Time constraints


Cost constraints


Using the same example, it would also be much more costly to pay phone interviewers
to call and interview 10,000 participants versus just 200.
Limited accessibility


For example, collecting data for each of 10,000 program participants via phone
interviews, could take many, many months. Taking a sample of 200 participants, for
example, will save time.
If your evaluation plan involves conducting site visits, but the sites are spread across
the entire United States, you may take a sample in the state or region closest to the
evaluators, or use some other factor to select a region.
Accuracy may be compromised

A sample is a better option if your evaluation plan calls for random assignment of
groups. In youth development settings, it is very challenging to assign a group to a
program and another group to the “not program” condition and maintain the
assignments (some students will drop out of the program, and program staff may
unknowingly allow some of the “not program” children to participate). Given the
challenges, it is sometimes better to take a small sample that will allow the evaluator
and the program staff to control the conditions and ensure accuracy of the study.
Drivers of Sampling Methods






Budget
Size of population
Geographic location of population
Availability of sampling frame
Data collection methods
Variance in the population
Common Sampling Methods
Probability:

Simple random sampling


Each unit has an equal and independent chance of being selected
Usually uses a random number table or software.


Stratified random sampling - used when evaluator wants to examine
differences among subgroups in population
Cluster sampling – used in studies covering large geographic regions
Non-probability:

Convenience sampling



Purposive sampling


Individuals are chosen for sampling because of accessibility
Little concern for the make-up of the sample
Used to thoroughly understand and explore issues within a small
group or case
Snowball sampling

Selected individuals are asked to refer additional individuals, who in
turn are asked to refer additional individuals, and so on
Cost Analysis

Cost-Benefit Analysis





analysis of well-defined alternatives by comparing their costs and
benefits when both are expressed in monetary terms.
each alternative is examined to see whether benefits exceed costs
difficult to translate into dollar terms all benefits
educational benefits are often translated into:
 projected gains in earnings or
 amount of money one would have to pay for educational
services if they were not provided
Cost-Effectiveness Analysis



used when client needs to choose from among several different
ways to achieve the same goal
expressed as a ratio of costs to outcome
only programs with common goals and common measures of
effectiveness can be compared using this method
Major Concepts and Theories in Chapter 14






Evaluators often use multiple designs and methods in evaluation
studies
Different designs include: experimental, quasi-experimental, and
descriptive
Descriptive designs are most common
Most evaluation questions can be answered by gathering data
from a subset of the population of interest
Random sampling can be used with large populations
Cost studies determine feasibility of program outcomes
 Most common are cost-benefit and cost-effectiveness