Transcript Research Methods – Part 2

Research Methods Review
Advanced Cognitive Psychology
PSY 421 - Fall, 2004
Overview


The Basics
Experimental Methods







Designs
Within-subjects
Between-subjects
Factorial
Statistical Review
Example of Experiment
Don’t let me forget a simple assignment to
give you before you leave
What are Research Methods?






Application of the scientific method to
studying behavior
Scientific Method = observe,
hypothesize/predict, test, conclude
Work with structured observations
Deal with solvable problems
Produce publicly verifiable information
Hypotheses, theories, and explanations
MUST BE FALSIFIABLE
General Methods – The Basics




Watch people in public settings and drawing
conclusions = Observational/Descriptive
Conducting a survey, questionnaire, interview,
poll, etc. and comparing responses on one
question to other questions =
Relational/Correlational
Randomly forming 2 or more groups, treat them
all differently, see how the outcomes differ =
Experimental
Compare two or more groups (that are not
randomly formed and are different to begin with)
in a variety of ways = Quasi-Experimental
Experimental Method


This type of method attempts to answer
questions about CAUSE
Variables




Independent = manipulated or changed for
various subjects; IV
Dependent = the way to measure a change in
behavior (or not change); DV
Control = a potential IV that is held constant at
one level – all subjects are exposed to that one
level
Manipulation = change
Manipulating Variables - Designs

One variable – multiple levels




Univariate or one-way design
Level = one aspect of the variable; condition
Example
Manipulation Types

Between Subjects = ONE


To expose some subjects to one level of the IV and
other subjects to another level of the IV – subjects
are not exposed to all aspects of the variable
Within Subjects = ALL

To expose all subjects to all the levels of the IV
The BOX

Boxes with rows and columns – essential
for understanding experiment design and
statistics
Variable A
Level 1
Level 2
Level 1
Row 1
Level 2
Row 2
Variable B
Column 1
Column 2
Factorial Designs






When more than one variable is
manipulated in an experiment
2 Variables = Two way
3 or more variables = multivariate
Between subjects design = all variables are
manipulated between/across the levels
Within subjects design = all subjects
receive all the levels of all the variables
Mixed design = at least one variable is
manipulated between subjects and at least
one variable is manipulated within subjects
Statistical Review



Populations and Samples
Hypothesis Testing
Using methods and statistical tests
together
Population vs. Sample

Population = everyone that you are interested in studying
or at the very least, generalizing your results to



Sample = a subset of the population that contains the
important characteristics of the population; a sample is
representative of its population



Ex: Women; children; Men over 40
You can’t possibly hope to study the entire population
Various techniques for sampling from a population
Ex: PSU Women; children at BFC; Male faculty in psychology over
40
Why does this matter?


Allows research to occur without the impossible task of studying
everyone
Important assumptions for statistics
Hypothesis Testing



Comparing the Null and Experimental
hypotheses to predict the likelihood of one
being show to be true
Null Hypothesis = There is no difference;
nothing will change; zero
Experimental Hypothesis = There will be a
difference; something will change; nonzero
Hypothesis Testing
Null
Hypothesis
There is no
difference in
GRE scores
between males
and females
 Experimental
Hypothesis
There is a
difference in
GRE scores
between males
and females

In the real world, the Null is
True
Your
decision
Reject Type I
H0
error
False
Correct
Decision
Retain Correct Type II
H0
Decision error
Hypothesis Testing Example
Barbie and Kendall – Chocolate Eaters
(to be read in class)

From this, what could we conclude about this contradiction?
1. According to our assumption, in the real world, H0 is true. Therefore,
if Barbie rejected H0 because she thought it was wrong (based on
her study's results), what has happened? Did she commit an error or
make a correct decision?
2. If Barbie would have retained H0 (and to do that, her study would
have resulted in no differences between the mean exam scores from
the two groups), and we assume that in the real world, H0 is true,
did she commit an error or make a correct decision?
3. If we now assume that Kendall is wrong and in the real world, H0 is
false, and Barbie rejected H0because she thought it was wrong
(based on her study's results), what has happened? Did she commit
an error or make a correct decision?
4. Again, assume that Kendall is wrong, and H0 is false in the real world.
If Barbie would have retained H0 (and to do that, her study would
have resulted in no differences between the mean exam scores from
the two groups), and we assume that in the real world, H0 is true,
did she commit an error or make a correct decision?
Hypothesis Testing
Alpha
Beta
Effect Size
Statistical Power
Definition
The probability of
committing a Type I
error
The probability of
committing a Type
II error
The size of the
effect (difference/
relationship)
The probability of rejecting a false
null hypothesis (or the probability
of finding an effect if one exists)
When to use
Set prior to collecting
data
Based on how
much power you
want (determined
before collecting
data) or how much
power you actually
have (determined
after collecting
data)
To be determined
every time you run
an inferential
statistic
(correlation, t-test,
ANOVA, chisquare)
Prior to collecting data: estimate
power and effect size to determine
how many subjects you need to
achieve certain level of power.
After analyzing data: determine
effect size and combine that with
sample size to determine power
Interpretation
Percent chance of
actually committing a
Type I error (if the
null is true)
Percent chance of
actually
committing a Type
II error (if the null
is false)
Small, medium, or
large (quantitative
values depend on
type of effect size
test you used)
Percent chance that you will find a
statistically significant result given
your sample size (N) and effect
size (and assuming the null is false)
Example
α = .05 means a 5%
chance of committing
a Type I error
β = .20 means a
20% chance of
committing a Type
II error
d = .50 means a
medium effect size
for the effect size
corresponding to a
t-test
Power = .80 means an 80% chance
of finding a statistically significant
result given your N, ES, and
assumption that the null is false
Scales of Measurement

Nominal = used to identify a particular characteristics of
the scale; also called categorical (categories are mutually
exclusive)


Ordinal = numbers indicate whether there is more or
less of the measured variable; order is important


EX: Levels of education (Freshman, Sophomore, Junior, Senior);
Olympic medals
Interval = numbers correspond exactly to changes in the
measured variable and there are equal distances
between numbers that correspond to equal changes in
the measured variable


EX: Sex (M/F); ZIP Codes
EX: IQ; Temperature (Fahrenheit, Celcius)
Ratio = like an interval scale (equal intervals) but also
includes a true zero point (the absence of the measured
variable). This allows for multiplication and division of
scale values.

EX: Weight; Height; Temperature (degrees Kelvin)
Descriptive Statistics
Decide how to summarize and represent data based on the
TYPE of data that you have (its scale of measurement)



Measures of Central Tendency
1. Nominal Scale – Mode
2. Ordinal Scale – Median
3. Interval/Ratio Scales - Mean
Measures of Dispersion
1. Nominal Scale – Range
2. Ordinal Scale – Absolute Deviation from the Median
3. Interval/Ratio Scales – Variance, Standard Error, Standard
Deviation
Graphical Representations of Data
1. Nominal/Ordinal Scales (Qualitative Data) – Bar Graph, Pie Chart
2. Interval/Ratio Scales (Quantitative Data) – Line Graph,
Frequency Polygon, Histogram
Inferential Statistics
Decide which test to use based on the TYPE of data you
have and the KIND of outcome you are looking for

Relationships (Correlations)
1. Nominal Scale – Chi-Square Test of Independence,
or Phi
2. Ordinal Scale – Kendall’s Tau or Spearman’s r
3. Interval/Ratio Scales – Pearson’s r

Differences
1. One Independent Variable
 Between-Subjects manipulation
 2 levels
 Nonparametric – Chi-Square Goodness of Fit (nominal)
and Mann Whitney U (ordinal)
 Parametric – Independent means t-test or one-way
ANOVA
 3+ levels
 Nonparametric – Chi-Square Goodness of Fit (nominal)
and Kruskal Wallace (ordinal)
 Parametric – One-way ANOVA
 Within-Subjects manipulation
 2 levels
 Nonparametric – Chi-Square Goodness of Fit (nominal)
and Mann Whitney U (ordinal)
 Parametric – Independent means t-test or one-way
ANOVA
 3+ levels
 Nonparametric – no good tests
 Parametric – One way Repeated-Measures ANOVA

Differences, continued
2.
Two Independent Variables

Between-Subjects manipulation



Within-Subjects manipulation



Nonparametric – Wilcoxon-Wilcox
and Friedman tests
Parametric – Factorial ANOVA
Nonparametric – Friedman test
Parametric – Factorial RepeatedMeasures ANOVA
Mixed manipulation – Mixed Factorial ANOVA
Putting this all together…




To study behavior, we have to create conditions
that are controlled enough to be able to predict an
outcome in the controlled conditions
We have to think about how to study the behavior
of interest and how to make it change in a
predictable fashion
Experimental methodology allows researchers to
control the sample and expose the participants to
changes that are predicted to influence the
outcome
When you construct a particular experiment or use
a particular research method, a certain logic applies
when choosing the “proper” statistical method to
analyze the results/outcome
Experiment Example

This will be shown in class

Signal Detection Experiment




IV – Presence of Target
Levels: Present or Absent
Manipulation: Within-Subjects
Measure/DV: Hits, False Alarms, Correct
Rejections, Misses