Transcript Research I Basics_Creating a - 47-269-203-spr2010

Who are the participants?
Creating a Quality Sample
47:269: Research Methods I
Dr. Leonard
March 22, 2010
The importance of a research sample
It is rarely possible to study an entire
population, or all people the research is
focused on, so to be more efficient we draw
a sample
We use findings from the sample to infer
conclusions about the population
Therefore, the quality of our conclusions
about the population depends on how
good, or representative our sample is
Population vs. Sample
Selecting an unbiased sample
Ideally, all samples would be without bias,
meaning any individual in the population
has an equal chance of being in the study
To do this a researcher needs to have
access to every single member of the
population, which is unlikely
Therefore, most of our samples have built
in biases
Picking a sample
Bias in sampling
 When a sample does not reflect some of the similarities or
differences present in the population
 Chance (by accident) OR Selection Bias (due to your
sampling methods)
 Example: I want to test whether or not a new
pedestrian assist device will aid blind individuals’ in
navigating busy intersections
• What is my target population?
• What is my accessible population?
• How might I select participants to be in the study?
Gold standard: Random Sampling

Increases representativeness of population using PROBABILITY
 Characteristics of population are known so likelihood of getting
each type of individual can be estimated
 Each member of the population has an equal chance of being
selected into the sample
 Each possible sample of a given size (e.g., n=10) has an equal
chance of being selected from the population (N=100)

We want our samples to reflect all of the similarities and differences
found in our target populations
 Age
 Gender
 Ethnicity
 Political or Religious Affiliation
 Others specific variables of interest
 The range of characteristics or variety in the sample is almost
never as big as in the population but it should be close
Types of Random/Probability sampling)
 Simple Random sampling:
 Each individual has an equal & independent chance of being
selected (like drawing names out of a hat)
 Stratified Random Sampling:
 Divide population into strata, or sub-groups, before randomly
selecting participants and then draw representative percentage
from each strata
 Systematic Sampling:
 Line up the population, randomly select a starting point, and take
every nth (lets say 10th) person
 Cluster Sampling:
 Imagine being interested if learning about high-school students’
attitudes towards military service. You are interested in collecting
a sample of 500 students.
 Instead of randomly sampling to get 500 students, list the 30
schools and randomly select 5 schools.
 Then test 100 students from each of those schools.
Stratified Random Sampling
If we wanted sample to represent the SES
breakdown of a given population…
Consider need for different recruiting methods
Systematic Sampling
Cluster Sampling
100
100
100
100
100
n = 500
Nonprobability Sampling Methods
 Characteristics of entire population not known
 Therefore, probability of selecting certain types of individuals
can not be estimated
 Sampling is nonrandom, but with an effort to maintain
representativeness and avoid bias.
 Types of nonprobability sampling
Convenience (e.g., Use the next ten people through the
door)
Quota (e.g., need a certain number of individuals with a
specific trait in each group)
Volunteer (e.g., undergraduate psychology majors)
Purposive (e.g., looking for expert informants)
Snowball (e.g., hard time finding participants so ask them
to recruit others they know)
 Which kind of research is more likely to use random
(probability) sampling?
Experimental research (Any scientific study in
which the researcher systematically varies one or
more variables, holding all others constant, to see
if another variable is affected)
 Why?
Exerts more control than non-experimental
Seeks to rule out extraneous variables &
confounds
More manipulation of independent variable
More likely to use groups of participants
Reminders about experimental
research
 Any scientific study in which the researcher
systematically varies one or more variables, holding all
others constant, to see if another variable is affected
 Intervention made or treatment given to observe effects
(causal -- does X cause changes in Y?)
 Independent variable must have two or more levels for
comparison
 Most often accomplished by having experimental group(s)
and control group
 True experiment if assignment of participants to groups
is random
Random Sampling vs. Random Assignment
Random sampling is concerned with every
individual in the population having an
equal chance of being in the study
Random assignment is concerned with
every participant in the sample having an
equal chance of being in an experimental
group (rules out bias)
True experimental designs (1)
 Pretest-posttest randomized control group design
Participants randomly assigned to groups
Results in two “equal” or equivalent groups
Experimental and control group
Both groups tested, or observed, before treatment (pretest)
Treatment given (independent variable being manipulated)
Both groups tested, or observed, again (posttest)
Any pre-post gains or differences between groups are likely
attributable to a) the treatment or b) random error
R
R
O
O
X
O
O
True experimental designs (2)
 Beware pretest sensitization Sometimes the pretest
can cause problems if it also affects the groups in
addition to the treatment
 Posttest only randomized control group design
Participants randomly assigned to groups
Results in two “equal” or equivalent groups
Treatment given (independent variable being manipulated)
Both groups tested, or observed, again (posttest)
Any posttest differences between groups are more likely
attributable to a) the treatment or b) random error
R
R
X
O
O
True experimental designs (3)
 Best of both designs…
 Solomon randomized four-group design
Still have participants randomly assigned to groups (may
need more participants to get equivalent groups)
Like running two experiments at the same time
Can test for pre-posttest gains and control for pretest
sensitization
R
R
R
R
O
O
X
X
O
O
O
O
Problems in True Experimental Designs
 What if the observed change is NOT due to the treatment
but due to random error? We may find a design lacks
validity (just like measures can lack validity)
 Experimental validity can be…
 Internal - the degree to which an experiment’s methods
are controlled and free from confounds
 External - the degree to which findings from research can
be generalized to the real world context (other
populations, other settings, other times???)
Threats to internal validity of experiment
 Any factors that lessens the degree of control
 Pre- and post-test problems
History - events that occur outside of the study that may affect
the outcome
Maturation - changes within individual during the study that may
affect the outcome
Instrumentation - changes in the measures between pre- and
post-tests; could be capturing different construct
Testing - any feature of the test or task that could change the
responses the second time; practice effect
Statistical regression to the mean - over time, scores tend to
move toward the average
Threats to internal validity of experiment
 Participant problems - any problems related to how the
individuals participating in the study may challenge the
validity of the findings; most common:
 Selection effects - participants in samples should be
equivalent to each other except for the independent variable
but human beings are not clones
Also, selection into the study should be totally random AND
each participant should have an equal chance of being
assigned to groups
 Morality/Attrition - every single participant may not complete
the study, which can be especially problematic if a certain
type of participant is more likely to drop out
Threats to external validity of experiment
Was our sample good enough that we can
we generalize our findings???
Population A
Population B
Population C
Sample
Setting A,
Time A
Setting B,
Time B
Setting C,
Time C