Transcript Testing for Validity - Hanover College Psychology Department

Web vs. Lab
John H. Krantz
Hanover College
Approach



Not focus on technology
Focus on the research question
What is best way to answer question
Approach

Web is a tool


to use in determining best way to ask question
Much as field experiments have become a
tool:


e.g. Milgram, S. , Bickman, L., & Berkowitz, L.
(1969). Note on the drawing power of crowds of
different sizes. Journal of Personality and Social
Psychology, 13(2), 79-82.
While this example is trivia it shows an example of
a study not possible in the laboratory
Approach

What this implies:


For some questions web is a superior method to
lab
May depend on your situation as we all use what
we have available
Characteristics of Web vs. Lab

Web





Distributed
User or remotely
controlled environment
and location and some
factors are unknowable
Accidental sampling of
large population
Potentially anonymous
Requires computer

Lab





Confined
Experimenter controlled
environment or at least
determinable
Accidental sampling
generally but random
sampling of restricted
population is possible
Experimenter is known to
be present or near
May not have computer
Characteristics of Web vs. Lab

Similarities


A certain act of belief is needed to accept selfreport responses
Cannot control state and previous state or events
of participant


This is an issue in control
Random assignment is possible
Reasons for Use – Musch & Reips (2000)






In descending order (7 very important – 1
not important)
Large N
5.5
High statistical power
4.5
Speed data collection/participants
from other countries
3.6
External validity
3.4
Low cost
3.2
Reasons for Use

Possible responses not possible on paper






Dynamic and interactive (but any computer would
do)
Access to special populations
More precise replication
Reduction of experimenter effects
Comparability to local sample
New set of variables to study
Reasons for Concern

Ethical issues



Data validity


Informed consent/debriefing
Data security
Subject falsification/collaboration
Loss of experimental control

Equipment/environment Variation
Reasons for Concern



Computers are necessary
Lack of ability to clarify instructions and
informed consent
New class of variables that may impact
responses

Does this last one sound familiar?
Importance of Problems Faced


Musch & Reips (2000)
In descending order






Lack of control of behavior
Lack of control of motivation
Inability to ask questions
No control over hardware
Nonrepresentative sample
Ethical problems
3.6
3.4
3.3
2.9
2.8
1.5
Issues regarding the sample


Web is stated to be not representative
Questions



How many labs have representative sample?
Do we know what person variables are significant
or do we control out of the assumption of a
possible effect?
The greater diversity of samples (Krantz & Dalal,
2000 and “College Sophomores”) may allow this
to become a research question instead of
assumption
The Web Study In Context

Relevant Dimensions




Methodological Issues
Subject-Related Issues
Ethical Issues
A Decision Guide
Relevant Dimensions – Method

Statistical Power


Refers to noisy data
Not in all cases will noise be better on Web

Ruppertsberg et al (2000).



Web, n = 151, s = 12.5, SE = 1.02
Control, n = 14, s = 7.4, SE = 1.9
Effect Size or Robustness of the Effect

Basically is it easy to obtain the result
Relevant Dimensions – Method

Sample Bias


Is the effect sensitive to the characteristics of the
sample?
External Validity


Does this effect or relationship happen outside of
laboratory
Does this effect or relationship happen away from
a computer
Relevant Dimensions – Method

Need for Control

Stimulus

Problematic Dimensions
Color
 Luminance
 Temporal
 If can keep on one page


Relative spatial dimensions are fine
Example Study: Column Taper

Description:



Method



Doric columns appear to lean when vertical
Apparently due to taper of column width
Three columns. Adjust so out two same position
as inner (vertical or horizontal)
Validation
Results
Manipulation Validation
6
y = 0.99x
Measured Angle
5
4
3
2
1
0
0
1
2
3
Target Angle
4
5
Vertical Results
250
Tilt at Perceived Vertical (arcmin)
200
150
y = 386.88x - 3.36
r2 = 0.87
100
50
0
-0.4
-0.2
0
0.2
-50
-100
y = 325.75x - 11.72
r2 = 0.99
-150
-200
Log Column Proportion
0.4
Web Vert
Lab Vert
Linear (Web Vert)
Linear (Lab Vert)
Relevant Dimensions – Method

Need for Control

Measurement





Environment
Effect of Web Known


Need will be sensitive to type of measurement
Think of all question order effects
Now type of response objects
Has the method been examined on Web
Time for data collection

Response time for results
Relevant Dimensions – Subject

Special Populations




Ethnic groups
Age
Education
Other targeted populations (Buchanan, 2000)

Self-Monitoring: found groups predicted high and low on
scale to validate response
Relevant Dimensions – Subject

Interaction with Subjects



Need for Clarification
As part of research design
Demand Characteristics

Web removes us, but our design might still
provide some hints
Relevant Dimensions – Ethics

Sensitivity of Ethical Issues



Deception
Potential for harm
Effects of Fraud by Participants on Data


Faking responses
Participant Collaboration
A Decision Guide - Method
Sensitivity of Study to/Need:
Low
Statistical Power Either
High
Web
Effect Size
Depends1
Either
Sample bias
Either
Usually Lab2
Ecological
Validity
Either
Depends3
Decision Guide – Method
Low
High
Control
Either
Lab
Known Valid on
Web
Time for Data
Lab at least as
check
Web
Web
Either
Decision Guide – Subject
Low
High
Special
Populations
Either
Web, in most
cases
Subjects from
other countries
Either
Web
Interaction with
subjects
Either
Lab
Demand
Possibility
Either
Web
Decision Guide – Ethics
Sensitivity to
Ethics
Effects of Fraud
Low
High
Either
Lab
Either
Lab
Factors not Included in Decision Guide

Did not include ease or cost


Also not important factor in early researchers,
Musch & Reips, 2000
These factors should not drive design criteria.
Conclusions




Web and lab distinct environments
Each method has strengths
Match method to research question
Still useful to examine methods


Study locally as well as globally
Differences do not necessarily imply web is less
valid than lab


Differences could be meaningful
Differences could imply problem in lab envirnment