Transcript The research designs:

The research designs:
• 1. pre-experimental designs
• 2. experimental designs
• 3. quasi-experimental designs
PRE-EXPERIMENTAL
RESEARCH DESIGNS
• #1. One-shot Case Study
X O
• #2. One-Group PretestPosttest Design
OXO
• #3. Static-Group Comparison
X O
O
#1 One-shot case study
• an incident is carefully studied
and compared with other events
casually observed or remembered
treatment
X
observation
O
–Advantage: real-world
exploration possible
–Disadvantage: very little ability
to eliminate confounding
#2 One-group pretest posttest design
• tests a single group before and after a
treatment O X O
–a. Advantage -- comparison between
performances by the same subjects
–b. Disadvantage -- inability to assess
whether or not differences due to
treatment or to confounding extraneous
variables (other things besides the
treatment might have occurred between
pretest and posttest
#3 Static-group comparison
• apply treatment to one group then
posttest two groups, the one that
received the treatment and
another that did not X O
Equivalent??
O
–Advantage -- can provide group
comparisons to evaluate a
program after it is completed.
–Disadvantage -- equivalence of
the groups is unknown.
True-experimental designsrandomization is critical
(to establish equivalence of
the groups)
TRUE-EXPERIMENTAL DESIGNS
• #4. Pretest-Posttest Control Group Design
RO X O
RO
O
• #5. Posttest-Only Control Group Design
R X O
R
O
• #6. Solomon Four-Group Design
RO X O
RO
O
R
X O
R
O
ALL WITH RANDOM ASSIGNMENT TO
GROUPS TO ENSURE EQUIVALENCE
#4 Pretest-posttest control group design
• two randomized groups tested
before treatment to determine prior
state
• one given treatment
• posttest measures taken on both
groups to compare change
RO
O
RO X O
COMPARE
CHANGE SCORES
#4 Pretest-posttest control group design
• two randomized groups tested
before treatment to determine prior
state
• one given treatment
• posttest measures taken on both
groups to compare change
»R 5 X 10
»R 4
7
increased 5 points
increased 3 points
Various comparisons possible
compare
change
»R O X O
»R O
O
»R O X O
»R O
O
compare pretests
to check equivalence
»R O X O
»R O
O
compare
posttests
to see
where they
end up
Advantages
• can measure effects of pretest itself (O---O)
• can expand to include more than one treatment
– (e.g., testing two, three, or more kinds of reading instruction
programs against only one control group in a single study).
• Allows many comparisons: (1) between groups;
(2) pre to post on one group: or (3) group and a
predetermined criterion (a norm)
• Statistical power can be increased by using the
pretest measure as a covariate to statistically
equate the groups (STATISTICAL CONTROL).
Disadvantages
• treatment can become
confounded with the pretest
(“internal validity”)
ROX O
RO
O
#5 Posttest-only control
group design
• one group receives treatment and both
groups are posttested
• design assumes groups are equivalent
due to random assignment.
R X O
R
O
#5 Posttest-only control
group design
• one group receives treatment and both
groups are posttested
• design assumes groups are equivalent
due to random assignment.
»R
»R
X 6
4
this is
higher
than
this
Advantages
• easy design to implement
• used in instances when a pretest is too
costly or inappropriate
– (e.g., evaluating a foreign language instructional program
where students arrive with no prior knowledge of the
language)
• or impossible
– (e.g. we forgot to pretest)
• eliminates interaction effect of
»OtheX
combined pretest -treatment from
group comparison
– and all other effects of pre (O---O)
• optimal design for applying the t-test
or analysis of variance.
Disadvantages
• requires care in selection of a
posttest instrument which is truly
sensitive to changes caused by
the treatment.
ROX O
• can’t measure change
RO
O
• will not allow as powerful
statistical tests as is permitted by
the standard pretest-posttest
control group design (NO
STATISTICAL CONTROL HERE).
compare
change
#6 Solomon four-group design
Standard pre-post
with control group
ROX O
RO
O
#6 Solomon four-group design
ROX
RO
X
Solomon adds these R
(posttest only control group)R
Standard pre-post
with control group
O
O
O
O
pre post
R 5 X 10
increased 5 points
pre post
R 5 X 10
R4
7
measure change but
leaves possibility
of pre + treatment
interaction
R 5 X 10
R4
7
R X 6
measure size of
pre + treatment
interaction
R 5 X 10
R4
7
R X 6
R
5
provides a baseline
against which to
compare everything
example of a pre + treatment
interaction
videotqpe
promoting
marriage
attitude
toward
marriage
attitude
toward
marriage
»R 5 X 10
»R 4
7
»R X 6
increase 5 points
increase 3 points
example of a pre + treatment
interaction
so maybe this
is a package
videotqpe
promoting
marriage
attitude
toward
marriage
attitude
toward
marriage
»R 5 X 10
»R 4
7
»R X 6
increase 5 points
increase 3 points
THIS POST IS THE SAME
SIZE AS THE PRE-POST
CONTROL ABOVE
»R 5 X 10
»R 4
7
»R X 6
»R
4
no pre + treatment
interaction but no
measure of change
pre post
»R 5 X 10
»R 4
7
»R X 6
»R
4
measure change but
leaves possibility
of pre + treatment
interaction
measure size of
pre + treatment
interaction
no pre + treatment
interaction but no
measure of change
Advantages of Four Group Design
–control threats to external validity inherent
in the standard design, e.g., possibility of
pretest sensitizing groups to the treatment.
–assess more accurately effects of pretest
alone, the treatment alone, and the
interaction of pretesting and treatment.
–If we disregard pretests, analysis of
variance procedures are possible.
–Alternatively, by using pretest as a
covariate, analysis of covariance
procedures are possible.
Disadvantage
• difficulty in arranging the
logistics of the design.
QUASI-EXPERIMENTAL DESIGNS
• #7. Time Series
OOOOXOOOO
• #8. Equivalent Time Samples
Design
X1 O X0 O X1 O X0 O, etc.
• #9. Nonequivalent Control
Group Design
OX O
O
O
#7 Time-series design
• characterized by periodic
measurements on the same group
and with a one-time intervening
treatment.
• popularly used in physical
sciences and biological research
OOOOXOOOO
what do you look for?
treatment
behavioral
measure
time
what do you look for?
treatment
behavioral
measure
FIT LINES TO THE DATA
time
what do you look for?
treatment
behavioral
measure
SIGNIFICANTLY DIFFERENT?????
time
what do you look for?
treatment
behavioral
measure
time
REMEMBER THESE-- from “behavior
coding”
• comparison design = compare baseline and
treatment
• two experimental designs =
–reversal design = sequence of baseline,
treatment, and reversal to baseline.
–multiple-baseline design = same treatment on
two classes of behavior at different times.
like these, the time series designs rely
upon extensive baseline data
Advantages
• can be used to study variables
that can’t be studied any other
way
• since there’s no control group, the
experiment might be repeated in
many different places by different
researchers to gain external
validity
Disadvantages
• inability to control for effects of
history (e.g., events occurring
between two measurements) and
this design typically extends over
long periods of time.
#8 Equivalent-time samples design
• an extension of the time series
design by alternating treatment
and no treatment prior to each
measurement.
X1 O X0 O X1 O X0 O, etc.
what do you look for?
treatment
behavioral
measure
time
what do you look for?
treatment
behavioral
measure
time
what do you look for?
treatment
behavioral
measure
time
what do you look for?
treatment
behavioral
measure
time
Advantage
• Useful design when effect of
treatment is shortlived
• Controls for history, overcoming
a weakness of other times series
designs.
Disadvantage
• inability to generalize the findings
to other subjects in other settings
• but can be replicated often
#9 Nonequivalent-control group
design
• pretest-posttest control group design
but no random assignment of subjects
to groups
• often groups are prearranged natural
units (e.g., classrooms, schools, etc.)
• however does assume random
assignment of treatments to groups
OX O
O
O
• Regardless of design limitations, it is still
superior to the one group pretest-posttest
design.
• "Matching" groups on some relevant
variables can greatly add precision to the
design.
• Common problem is differing rates of
maturation between the experimental and
control groups.
Summary:
Pre-Experimental Designs
• #1. One-shot Case Study
»X O
• #2. One-Group PretestPosttest Design
»O X O
• #3. Static-Group Comparison
» XO
»
O
True Experimental Designs
• #4. Pretest-Posttest Control Group Design
»R O X O
»RO
O
• #5. Solomon Four-Group Design
»R O X O
»R O O
»R X O
»R
O
• #6. Posttest-Only Control Group Design
»R X O
»R
O
Quasi-Experimental Designs
• #7. Time Series
»O O O OXO O O O
• #8. Equivalent Time Samples Design
»X1 O X0 O X1 O X0 O, etc.
• #9. Nonequivalent Control Group
Design
» OX O
» O O
SOURCES OF ERROR
• sources of confounding
History
• effects of environmental
events between pretest and
posttest may be confused with
the effects of the treatment
Example of history as source of confounding:
Elaborate presidential campaign focused
on “family values”
videotqpe
promoting
marriage
attitude
toward
marriage
attitude
toward
marriage
»R 5 X 10
»R 4
7
»R X 6
increase 5 points
increase 3 points
Maturation
• effects of changes within the
person related to passage of
time may be confused with the
effects of the treatment
Example of maturation as source of
confounding:
one year later
videotqpe
promoting
marriage
attitude
toward
marriage
attitude
toward
marriage
»R 5 X 10
»R 4
7
»R X 6
increase 5 points
increase 3 points
Selection bias
• some ulterior motive
associated with participating
in the experiment
Example of selection bias as source of
confounding:
students working in
the marital attitudes
project volunteer
videotqpe
promoting
marriage
attitude
toward
marriage
attitude
toward
marriage
»R 5 X 10
»R 4
7
»R X 6
increase 5 points
increase 3 points
Interaction between
selection bias and the
treatment
• those who volunteer for a
threatening treatment (remember
informed consent) have a
“special” motivation and that
treatment might work only with
them
Example of interaction between selection bias
and treatment as source of confounding:
individuals in
marital therapy
end their therapy
this way
videotqpe
promoting
marriage
attitude
toward
marriage
attitude
toward
marriage
»R 5 X 10
»R 4
7
»R X 6
increase 5 points
increase 3 points
Interaction between selection
bias and maturation
• the ulterior motive that got the
person to volunteer also makes
them “mature” faster
Example of Interaction between selection bias
and maturation as source of confounding:
one year later
16 yr old
volunteers
videotqpe
promoting
marriage
attitude
toward
marriage
attitude
toward
marriage
»R 5 X 10
»R 4
7
»R X 6
increase 5 points
increase 3 points
Testing
• effects of testing may be
confused with the effects of
the treatment
Example of testing as source of confounding:
videotqpe
promoting
marriage
attitude
toward
marriage
attitude
toward
marriage
»R 5 X 10
»R 4
7
»R X 6
increase 5 points
increase 3 points
Interaction between
testing and treatment
• pretest may sensitize a
subject's responsiveness to
the treatment
Interaction between testing
and treatment
remember this?????
so maybe this
is a package
videotqpe
promoting
marriage
attitude
toward
marriage
attitude
toward
marriage
»R 5 X 10
»R 4
7
»R X 6
increase 5 points
increase 3 points
THIS POST IS THE SAME
SIZE AS THE PRE-POST
CONTROL ABOVE
Instrumentation
• something inherent in the
measurement process may result
in inaccurate assessment
Example of instrumentation as source of
confounding:
people don’t understand
what you’re getting at videotqpe
promoting
marriage
attitude
toward
marriage
attitude
toward
marriage
»R 5 X 10
»R 4
7
»R X 6
people now understand what you’re
getting at
increase 5 points
increase 3 points
Response sets
• response tendencies not related
to the trait you are measuring
Statistical regression
• subjects automatically score closer to the
mean on retesting because of random
fluctuation
• e.g. subjects selected for a program on the
basis of extremely low or high scores on a
test
• program erroneously judged a success for
low achieving students
– because they tended to score “more
average” on the posttest
• and judged a failure for high achievers
–because they tended to score “more
average” on the posttest
Example of statistical regression as source of
confounding:
people give extreme
answers
videotqpe
promoting
marriage
attitude
toward
marriage
answers are now less
extreme
attitude
toward
marriage
»R 5 X 10
»R 4
7
»R X 6
increase 5 points
increase 3 points
Experimental mortality
• loss of subjects at different
rates from different groups
Example of experimental mortality as
source of confounding:
gung ho subjects
videotqpe
promoting
hang in there
uninterested subjects marriage
attitude
attitude
drop out
see next slide
toward
marriage
toward
marriage
»R 5 X 10
»R 4
7
»R X 6
increase 5 points
increase 3 points
example people who want to
change stick with the treatment
everybody
takes the
pretest
people who would
hate the video have
dropped out
»R 5 X 10
people who would
hate the video
people who would
like the video
pretest
S1 5
S2 5
S3 5
S4 5
S5 5
posttest
4
5
1
2
1
S6 5
S7 5
S8 5
S9 5
S10 5
10
9
11
10
10
Effects of unique
experimental arrangements
• experimental situation may be
part of the treatment-- precludes
generalization to nonexperimental
settings
Example of unique experimental
arrangement as source of confounding:
videotqpe
promoting
marriage----shown by a couple that
attitude
attitude obviously happily
toward
toward
married
marriage
marriage
»R 5 X 10
»R 4
7
»R X 6
increase 5 points
increase 3 points
is
Multiple-treatment
interference
• same subjects (especially in onegroup designs) are used in
experiments of multiple
treatments
Example of multiple treatment interference as
source of confounding:
individuals in
marital therapy
end their therapy
this way
videotqpe
promoting
marriage
attitude
toward
marriage
attitude
toward
marriage
»R 5 X 10
»R 4
7
»R X 6
increase 5 points
increase 3 points
Hawthorne effect
• behavior of subjects changes
because they feel special (LIKE
SOMEBODY CARES)
Example of Hawthorne effect as source of
confounding:
somebody’s trying to help
videotqpe
promoting
marriage
attitudes
toward
marriage
attitudes
toward
marriage
»R 5 X 10
»R 4
7
»R X 6
increase 5 points
increase 3 points
Expectancy (or
Pygmalion) effect
• researcher sees those attributes
he or she "expects to see"
Example of expectancy as source of
confounding:
videotqpe
promoting
marriage
experimenter’s
rating of attitudes
toward
marriage
experimenter expects
the videotape to
work
experimenter’s
rating of attitudes
toward
marriage
»R 5 X 10
»R 4
7
»R X 6
increase 5 points
increase 3 points
Halo effect
• specific ratings biased by a more
global impression
–like selecting a certain brand of soap
because one likes the movie star
who advertises it
Example of expectancy as source of
confounding:
videotqpe
promoting
marriage
experimenter’s
rating of attitudes
toward
marriage
and these people seem
like people who really care-who really want to change
experimenter’s
rating of attitudes
toward
marriage
»R 5 X 10
»R 4
7
»R X 6
increase 5 points
increase 3 points
Control effect
• control condition is so stupidlooking that it lowers scores and
makes the experimental group
look good by comparison
example of control effect as
source of confounding
videotqpe
promoting
marriage
»R 5 X 10
MICKEY
MOUSE
CARTOON
»R 4 XC 1
The "we'll show them"
attitude
• performance of a control group,
placed in competition with an
experimental group, outpaces
what they ordinarily would do
EXAMPLE OF THE WE’LL
SHOW THEM ATTITUDE AS A
SOURCE OF CONFOUNDING
videotqpe
promoting
marriage
»R 5 X 10
HOW COME THEY
GET ALL THE SPECIAL
TREATMENT
»R 4
20
The following errors are likely
to appear in the Discussion
section
"Figures prove. . . "
• "80% of the doctors interviewed"
can mean four of only five
questioned
Law of the instrument
• the human inclination to become
attached to a particular instrument
and apply it to every problem
–give a surgeon a scalpel and
everything looks like it needs to be
cut out
Elementalism
• notion that one thing causes
everything (not considering
complexity and alternative
explanations)
Post hoc, ergo propter hoc:
"After this, therefore because of
this."
–e.g., superstitious beliefs
Ad verecundiam: "According
to the truth . . . "
• a “truth halo”
–buying a poor product simply
because it is endorsed by a
known authority
–G. B. Shaw said: "How can I tell
if I like a play if I don't know its
author?"
Secundum quid: "According to
which . . ."
–The fallacy of inadequate
sampling.
–bad luck follows walking under
a ladder --- forgetting the
tremendous number of times
one walked under a ladder with
no bad consequences
Ad hominem: "To the man
rather than the issue."
–disliking a speech because the
orator has not gone past the
seventh grade
–evaluation report should cover
all findings, not just selected
findings which support desired
outcomes.
Tu quogue: "You also . . .
(you're another! So's your old
man.)."
• A retort accusing an accuser of a
similar offense or similar behavior
» stopped for speeding and complaining
that others are driving faster than you
were