Experimental Design, Statistical Analysis
Download
Report
Transcript Experimental Design, Statistical Analysis
Experimental Design,
Statistical Analysis
CSCI 4800/6800
University of Georgia
March 7, 2002
Eileen Kraemer
Research Design
Elements:
Observations/Measures
Treatments/Programs
Groups
Assignment to Group
Time
Observations/Measure
Notation: ‘O’
Examples:
Body weight
Time to complete
Number of correct response
Multiple measures: O1, O2, …
Treatments or Programs
Notation: ‘X’
Use of medication
Use of visualization
Use of audio feedback
Etc.
Sometimes see X+, X-
Groups
Each group is assigned a line in the
design notation
Assignment to Group
R = random
N = non-equivalent groups
C = assignment by cutoffs
Time
Moves from left to right in diagram
Types of experiments
True experiment – random assignment
to groups
Quasi experiment – no random
assignment, but has a control group or
multiple measures
Non-experiment – no random
assignment, no control, no multiple
measures
Design Notation Example
R
O1
R
O1
X
Pretest-posttest treatment
comparison group
randomized experiment
O1,2
O1,2
Design Notation Example
N
O
N
O
X
Pretest-posttest
Non-Equivalent Groups
Quasi-experiment
O
O
Design Notation Example
X
Posttest Only
Non-experiment
O
Goals of design ..
Goal:to be able to show causality
First step: internal validity:
If x, then y
AND
If not X, then not Y
Two-group Designs
Two-group, posttest only, randomized
experiment
R
R
X
O
O
Compare by testing for differences between means
of groups, using t-test or one-way Analysis of
Variance(ANOVA)
Note: 2 groups, post-only measure, two distributions each
with mean and variance, statistical (non-chance) difference
between groups
To analyze …
What do we mean
by a difference?
Possible Outcomes:
Measuring Differences …
Three ways to estimate effect
Independent t-test
One-way Analysis of Variance (ANOVA)
Regression Analysis (most general)
equivalent
Computing the t-value
Computing the variance
Regression Analysis
Solve overdetermined system of equations for β0 and
β1, while minimizing sum of e-terms
Regression Analysis
ANOVA
Compares differences within group to
differences between groups
For 2 populations, 1 treatment, same as
t-test
Statistic used is F value, same as square
of t-value from t-test
Other Experimental Designs
Signal enhancers
Factorial designs
Noise reducers
Covariance designs
Blocking designs
Factorial Designs
Factorial Design
Factor – major independent variable
Setting, time_on_task
Level – subdivision of a factor
Setting= in_class, pull-out
Time_on_task = 1 hour, 4 hours
Factorial Design
Design notation as
shown
2x2 factorial design
(2 levels of one
factor X 2 levels of
second factor)
Outcomes of Factorial Design
Experiments
Null case
Main effect
Interaction Effect
The Null Case
The Null Case
Main Effect - Time
Main Effect - Setting
Main Effect - Both
Interaction effects
Interaction Effects
Statistical Methods for
Factorial Design
Regression Analysis
ANOVA
ANOVA
Analysis of variance – tests hypotheses
about differences between two or more
means
Could do pairwise comparison using ttests, but can lead to true hypothesis
being rejected (Type I error) (higher
probability than with ANOVA)
Between-subjects design
Example:
Effect of intensity of background noise on
reading comprehension
Group 1: 30 minutes reading, no
background noise
Group 2: 30 minutes reading, moderate
level of noise
Group 3: 30 minutes reading, loud
background noise
Experimental Design
One factor (noise), three levels(a=3)
Null hypothesis: 1 = 2 = 3
Noise
None
Moderate
High
R
O
O
O
Notation
If all sample sizes same, use n, and
total N = a * n
Else N = n1 + n2 + n3
Assumptions
Normal distributions
Homogeneity of variance
Variance is equal in each of the populations
Random, independent sampling
Still works well when assumptions not
quite true(“robust” to violations)
ANOVA
Compares two estimates of variance
MSE – Mean Square Error, variances within
samples
MSB – Mean Square Between, variance of
the sample means
If null hypothesis
is true, then MSE approx = MSB, since
both are estimates of same quantity
Is false, the MSB sufficiently > MSE
MSE
MSB
Use sample means to calculate sampling
distribution of the mean,
=1
MSB
Sampling distribution of the mean * n
In example, MSB = (n)(sampling dist) =
(4) (1) = 4
Is it significant?
Depends on ratio of MSB to MSE
F = MSB/MSE
Probability value computed based on F value,
F value has sampling distribution based on
degrees of freedom numerator (a-1) and
degrees of freedom denominator (N-a)
Lookup up F-value in table, find p value
For one degree of freedom, F == t^2
Factorial Between-Subjects
ANOVA, Two factors
Three significance tests
Main factor 1
Main factor 2
interaction
Example Experiment
Two factors (dosage, task)
3 levels of dosage (0, 100, 200 mg)
2 levels of task (simple, complex)
2x3 factorial design, 8 subjects/group
Summary table
SOURCE
Task
Dosage
TD
ERROR
TOTAL
df Sum of Squares
1
47125.3333
2
42.6667
2
1418.6667
42
5152.0000
47
53738.6667
Sources of variation:
Task
Dosage
Interaction
Error
Mean Square
F
p
47125.3333 384.174 0.000
21.3333
0.174 0.841
709.3333
5.783 0.006
122.6667
Results
Sum of squares (as before)
Mean Squares = (sum of squares) /
degrees of freedom
F ratios = mean square effect / mean
square error
P value : Given F value and degrees of
freedom, look up p value
Results - example
Mean time to complete task was higher
for complex task than for simple
Effect of dosage not significant
Interaction exists between dosage and
task: increase in dosage decreases
performance on complex while
increasing performance on simple
Results