Transcript Experiment Design 4 - Learning Research and Development Center

Experiment Design 5:
Variables & Levels
Martin, Ch 8, 9,10
Recap

Different kinds of variables
– Independent, dependent, confounding, control,
and random

Different kinds of validity
– Internal, construct, statistical, external
– Each associated with a question

Randomization
– Random sampling: generalization
– Random assignment: causation
Picking a design

Choosing how to assign participants to
levels of an independent variable
– Between vs. within
Choosing how many levels of an
independent variable
 Choosing how many independent variables

Between vs. Within designs

Condition 1:
– Fred
– Ginger
– Mary


5
8
6
Condition 2:
– Ed
– Mabel
– George
– Fred
– Ginger
– Mary

6
9
7
Condition 1:
5
8
6
Condition 2:
– Fred
– Ginger
– Mary
6
9
7
Within vs. Between Subjects

Cost
– Between: More participants
– Within: More time per participant

Confounding variables
– Between: Group differences possible
• Use randomization, many subjects, matching
– Within: Order effects possible
• Use counterbalancing
Transfer effects (order effects)

Definition:
– When taking part in earlier trials changes
performance in the later trials

Types
– Learning
– Fatigue
– Range or context effect

Problem:
– Makes within-subjects designs difficult to
interpret
Counterbalancing

Adjust condition order to unconfound
transfer effects with condition effects
–
–
–
–
–
–
A,B,C
A,C,B
B,A,C
B,C,A
C,A,B
C,B,A
Counter-balancing either withinor between- subjects

Between:
– Joe: A,B
– Mary: B,A

Within:
– Joe: ABBA
– Mary: ABBA
Things to worry about in
counter-balancing

If within-subjects counter-balancing:
– Linear transfer effects?
• Is the transfer from the 1st position to the 2nd
position the same as the transfer from 2nd to 3rd
position?
– E.g., sometimes most learning happens in 1st trials

Always worry about asymmetrical transfer
– Does A influence B more than B influences A?
Asymmetrical transfer
Quiet
Quiet
% trigrams
remembered
Noisy
Time 1
Noisy
Time 2
Effect of noise depends on order
 People stick with the strategy they pick first

– Or mix strategies
Partial counterbalancing:
Latin Square

Every condition appears in every position
equally:
– Joe: A
– Mary: C
– John: B
B
A
C
C
B
A
Matching
Try to reduce between-group differences
 E.g., rank hearing as Good, Fair, Poor
 Unmatched, could get

– Noisy: Poor1, Poor2, Fair1
– Quiet: Good1, Good2, Fair2

Matched, get:
– Noisy: Poor1, Fair2, Good1
– Quiet: Poor2, Fair1, Good2
Matching
Match variable(s) and DV’s should be
strongly correlated
 Caveat: Match test should not affect DV

– e.g., use existing match variable (SAT-M)

Note: Within-subjects designs “match”
automatically
Number of levels

How many different groups or conditions
that change just one independent variable
– Two:
• Experimental vs. control
• Massed vs. Distributed practice
– More:
• Drug vs. Placebo vs. No pill
• # of times an item is studied: 1,2,4,8, or 16 times
inside
outside
Inter- and extra-polating
60
40
20
0
0
1
2
3
# study repetitions
?
4
?
100
60
40
20
0
50
0
0
1
2
3
# study repetitions
4
0
1
2
3
# study repetitions
4
Floor & Ceiling Effects
Single Variable vs.
Multiple Variables

Single Variable:
– Only one independent variable
– Cannot look at interactions

Multiple Variables:
– Two or more independent variables
– If use factorial design, can look at interactions
– Can require a lot of participants (between) or
time (within)
Interactions
100
PrepLevel
Manuscript
Draft
% errors
detected
0
Author
Editor
Proofreader
Who finds more errors, author or editor?
 How to spot the interaction graphically?

Interactions
Two independent variables interact when
the effect of one depends on the level of the
other
 Independent vs. Control vs. Random

– What if PrepLevel had been a control variable?
– What if PrepLevel had been a random variable?
– Make it an independent variable if there is
reason to believe its effect might depend
Factorial design

Do all combinations of factors (cells)
– E.g., Language learning
German
Old
Young
Male
Female
French
Old
Young
Male
Female

A factor can be within or between
Converging Operations
(≠converging series)

Using more than one method to test the
same hypothesis
– E.g., using experimental and observational
methods
– E.g., using cross-sectional and longitudinal
designs
Baseline procedure

Example 1: Clinical
– No drug, drug, no drug, drug,...
 Example
2: Education
– Regular class, new format, regular
class, new format,..