Transcript Research planning - University of Warwick

Sampling, sample size
estimation, and randomisation
PS302
Overview
Sampling
representative sampling (e.g. for surveys)
homogenous sampling (e.g. for experiments)
Sample size estimation
Based on power
Gathering the information you need
Power calculations (G*Power software)
- ANOVA
- regression
Rules of thumb for multivariate tests
Presentation of power analysis in your report
Practical randomising
Random selection (e.g. for surveys)
Random allocation (e.g. for experiments)
Getting a representative sample
Survey of UK Households
want a sample from each SES group
each age group
each sex
Proportions should match the population
Matching the population
Percent of population  percent of sample
Assume, sample size = 1200
Population = Women 60%, Men 40%

Sample: Women 720, Men 480
Problem for you to try
Population figures:
Men 65 years+ = 1 million
Women 65 years+ = 1.5 million
Men 25-65 years = 8 million
Women 25-65 years = 8.5 million
Men < 25 years = 5 million
Women < 25 years = 5.2 million
Total population size = 29.2 million
Percent W25-65 = (5.2 / 29.2) * 100
= 17.8%
Given a sample size of 200, how many women
<25 years should be included?
quota sampling
Recruiters are given a quota of each stratum
Problem – biased selection by
recruiter/interviewer
Advantage – random selection very difficult
to achieve, quota sampling a good
compromise
Homogenous sampling
Restrict sampling to a narrow group
Sample only Warwick students
Sample only one Sex
Sample only one Age group
Advantages
reduces error variance by reducing individual
differences
Homogenous sampling ctd
Disadvantage – may reduce generalisability
generalisability will need to be considered
and assessed separately
Suitability
– experimental work
– studies where individual differences are not
directly relevant and power is more important
concern
power
Probability that any particular (random) sample will
produce a statistically significant effect
Eg. power = 0.9
 90% chance of detecting an effect if there really
is an effect
Researchers usually aim to have power at 80-90%
Power and sample size
All else being equal, to get more power you
need more participants
Where “all else” means:
reliability of measures
other sources of error variance
p-value
the true size of the effect
These concepts are inter-related
Desired power ↑
N↑
Acceptable p-value ↓
N↑
Effect size to detect ↓
N↑
Reliability of measures ↓
Other error variance ↑
N↑
N↑
if you know these…
effect size
variance of measures
you can often work out what the
sample size should be
So where can you find them?
Previous research studies
Calculating using G-power
•
First step, assemble the figures needed
For between subjects ANOVA:
1.
2.
3.
4.
5.
Effect size (Cohen’s f, or partial eta squared)
Significance level [.05, usually]
Power [.8, usually]
Numerator degrees of freedom (df)
Number of cells in design (groups)
1. Effect size
… from previous studies
Easy – they reported effect sizes
“There was not a significant main effect of Sex on
response time, F(1, 42) = 2.03, p = .16, η2 = 0.046”
Harder – they reported only the F and df, so
you have to make a calculation
partial η2
= (dfeffect * F) / [(dfeffect * F) + dferror]
= (1 * 2.03) / [(1 * 2.03) + 42]
= 0.046
measures of effect size for ANOVA
Roughly, the correlation between an effect
and the outcome (DV)
eta squared
The proportion of variance in the outcome variable (DV)
that is explained by the IV
SSeffect / SS[corrected] total
partial eta squared (SPSS prints this out)
The proportion of the effect + error variance explained by
the effect
SSeffect / (SSeffect + SSerror)
4. Numerator df
“There was a non-significant main effect of Gender on
response time, F(1, 42) = 2.03, p < .05, η2 = 0.09”
5. Number of cells (groups)
Two way ANOVA
2 x 3 ANOVA  6 cells
4 x 2 ANOVA  8 cells
Etc.
Calculating using G-power
•
First step, assemble the figures needed
For this 2 X 3 between subjects ANOVA:
1.
2.
3.
4.
Effect size (η2 = 0.046)
Significance level [.05, as usual]
Power [.8, normal]
Numerator degrees of freedom (df = 1, 2 for
the respective main effects, or 2 for the
interaction)
5. Number of cells in design (groups = 6)
tip: power & ANOVA
Each effect in the ANOVA has its own power
Eg. 2 x 3 ANOVA
Main effect A
Main effect B
Interaction effect A * B
Tip: power is lower for interactions
than for main effects
Sample size – ethical issues
Too small a sample
-- can’t detect significant effects
 waste all participants’ time
Too large a sample
-- waste resources
-- waste the extra participants’ time
Sample size – practical issues
Resources
Time
Cost of running each participant
Availability
Clinical populations are often small
Access can take time & require permission
Choosing an appropriate sample size for
established laboratory paradigms
Shortcut
Base sample size on sample size used in previous
research
This is often perfectly appropriate
(but make sure the previous research is of high
quality!)
Rules of thumb for multivariate tests
multiple regression
cases (N) / predictors (p)
N at least 50 + 8p for R2
N at least 104 + p for testing a predictor
Need more cases if outcome is skewed,
anticipated effect size is small, measures
less reliable…
Rules of thumb for multivariate tests
PCA (exploratory FA)
50 no good
100 poor
300 good, but ideally need more
Random allocation
For example
3 between subjects conditions (e.g. control, happy,
sad)
Who does which condition?
first come? Interviewer choice?
Must avoid confounds. But can’t check all possible.
Solution is random allocation.
Random allocation needs truly
random numbers
Different ways to do that
SPSS
random.org
Research randomiser
scripting language like python
Python
to randomly assign 9 participants to 3 conditions:
from random import shuffle
numbers = [1,1,1,2,2,2,3,3,3]
shuffle(numbers)
numbers
[3, 2, 1, 2, 3, 1, 3, 1, 2]
Research randomiser
http://www.randomizer.org/form.htm
3 conditions, 48 planned participants
randomly: allocate each participant (identified by order of
recruitment) to one of 3 conditions
How many sets of numbers to generate? [1]
numbers per set? [48]
Number range? [From 1 To 3]
Do you wish each number in a set to remain unique? [No]
[Don’t “sort”!]
Result
Set #1:3, 3, 1, 3, 3, 1, 2, 2, 2, 1, 2, 2, 1, 1, 2,
1, 3, 3, 1, 3, 2, 1, 3, 3, 1, 2, 1, 3, 1, 2, 2, 2,
3, 3, 1, 3, 3, 1, 1, 3, 1, 3, 3, 2, 3, 3, 1, 2
Research randomiser
http://www.randomizer.org/form.htm
3 sentence types, 48 sentences
16 in each group, create a random sequence, but
limit runs of the same type
How many sets of numbers to generate? [16]
numbers per set? [3]
Number range? [From 1 To 3]
Do you wish each number in a set to remain
unique? [Yes]
Research randomiser
http://www.randomizer.org/form.htm
3 types, 48 sentences, 16 of each type
limit run of a given type, while still
randomising order of presentation
16 Sets of 3 Unique Numbers Per Set
Range: From 1 to 3 -- Unsorted
Job Status:
Set #1:2, 3, 1
Set #2:3, 1, 2
Set #3: ….
Web links
http://www.randomizer.org/
http://www.random.org/
measures of effect size for ANOVA
Roughly, the correlation between an effect
and the outcome (DV)
eta squared
The proportion of variance in the outcome variable (DV)
that is explained by the IV
SSeffect / SS[corrected] total
partial eta squared (SPSS prints this out)
The proportion of the effect + error variance explained by
the effect
SSeffect / (SSeffect + SSerror)