Transcript Quasi-experimental Design

Quasi-Experimental Design
Jung Eun (Jessie) Hong
Feb. 23, 2009
Outlines

Experimental Design





Definition
Process
A Key Point
Types of Experimental Designs
Quasi-Experimental Design





Designs
Strengths
Weaknesses
Examples in Geography
Ongoing Debate
Definition of Experimental Design

A structured, organized method

To determine whether some program or treatment
causes some outcome or outcomes to occur.


If X, then Y
Because there may be lots of reasons, other than
the program, for why you observed the outcome,

If not X, then not Y needs to be addressed, too
Process of Experimental Design

To show that there is a casual relationship,

Two “equivalent” groups




The program or treatment group gets the program
The comparison or control group does not
The groups are treated the same in all other respects
Differences in outcomes between two groups
must be due to “the program”
A Key Point of Experimental Design

How do we create two groups that are
“equivalent”?

Assign people randomly from a common pool of
people into the two groups

The experiment relies on the idea of “random
assignment” to obtain two similar groups.


A key to the success of the experiment
Assume that two groups are “probabilistically
equivalent”
Types of Designs
Is random assignment used?
Yes
Randomized or
True experiment
No
Is there a control group or
multiple measures?
Yes
Quasi-experiment
No
Non-experiment
Quasi-Experimental Design



Similar to the experimental design, but lacks
the key ingredient, “random assignment”
Easily and more frequently implemented
Extensively used in the social sciences


A useful method for measuring social variables
Two classic quasi-experimental designs


The Nonequivalent Groups Design
The Regression-Discontinuity Design
The Nonequivalent Groups Design


The most frequently used in social research
Try to select groups that are as similar as
possible to compare the treated one with the
comparison one


e.g. two comparable classrooms or schools
Cannot be sure whether the groups are
comparable

The groups may be different prior to the study


Any prior differences between the groups may affect the
outcome of the study
Require a pretest and posttest
The Regression-Discontinuity Design


A useful method for determining whether a program
of treatment is effective
Participants are assigned to program or comparison
groups based on a cutoff score on a pretest

e.g. Evaluating new learning method to children who
obtained low scores at the previous test.




Cutoff score = 50
The treatment group: children who obtained 0 to 50
The comparison group: children who obtained 51 to 100
The program (treatment) can be given to those most
in need
The Regression-Discontinuity Design
With no treatment effect
With Ten point treatment effect
The Regression-Discontinuity Design
Discontinuity
Strengths of Quasi-Experimental Design

Useful in generating results for general trends
in social sciences


Easily integrated with individual case studies



Difficult pre-selection and randomization of groups
Generated results can reinforce the findings in a
case study
Allow statistical analysis to take place
Enable to reduce the time and resources
required for experimentation

Not required extensive pre-screening and
randomization
Weaknesses of Quasi-Experimental
Design

Without proper randomization, statistical tests
can be meaningless

Do not explain any pre-existing factors and
influences outside of the experiment


The researcher needs to control additional factors that
may have affected the results
Some form of pre-testing or random selection may be
necessary to explain statistical results thoroughly
Quasi-experiments vs. Non-experiments
to address similar questions

Both designs are applicable when the subjects are
not able to be randomized

Some variables cannot ethically be randomized

e.g. Studying the effect of maternal alcohol use when the
mother is pregnant
Quasi-experiments Non-experiments
Strengths
Enable to compare with
other groups
Enable to focus on one
variable
Weaknesses
Unexpected factors might
affect the results
Interpretations might be
improper
Example of Quasi-Experimental Design
in Geography

Baker and White (2003)


The Effects of GIS on Students’ Attitudes, Self-efficacy,
and Achievement in Middle School Science Classrooms
Conducted the Nonequivalent quasi-experimental design




Two eighth grade teachers, across ten classrooms
Total 192 eighth grade students participated
Treatment group: used a Web-based GIS application
Control group: used paper maps
Treatment
Group
Instructor 1 51
Instructor 2 42
Control
Group
36
63
Example of Quasi-Experimental Design
in Geography



Impossible to randomly assign each student
to a GIS or paper mapping conditions
Randomly assigned whole classes to two
conditions
Different instructors affected the results
differently

Instructor effect played a substantial role in
student attitudes and self-efficacy
Ongoing Debate


Whether true experiments or quasi-experiments
represents the superior design
Supporters of true experiments




Difficult to isolate the program effects using quasiexperiments
Quasi-experimental results are biased and sensitive to
minor changes
Not sure about whether quasi-experimental designs can
adequately control selection bias
Hard to determine better design

True experiments are impossible and impractical in some
cases
Any questions???