Transcript Exam 3. - Psychology 242

Foundations of
Research
Final Exam Review
1
© Dr. David J. McKirnan, 2014
The University of Illinois Chicago
[email protected]
Do not use or reproduce without
permission
Psychology 242, Dr. McKirnan
Cranach, Tree of Knowledge [of Good and Evil] (1472)
Foundations of
Research
What is science?
 Understand that science is not just
2
about the numbers.
 Rather, it embodies some key values:
 Critical thought
 Empirical approach to
What is science?
Values




understanding the world.
Critical thought
Theory: Why? or How?
Evidence: How do you know?
Discover the natural world
Content
 Empirical findings: Facts
 Ways of classifying nature
 Well supported theories
Methods




Core empirical approach
Basic experimental design
Specific research procedures
Statistical reasoning
Foundations of
Research
Four basic sources of knowledge or information:
How do we know things?

Authority:
Credible / powerful people
Social institutions
Tradition

Intuition:
Emotionality or a “hunch”
“Emotional IQ”

Empiricism:

Rationalism:
Simple sensation / perception
Direct observation; data
Logical coherence
Articulation with other ideas
Most central to
Science
3
Foundations of
Research

What does science do?
What does science do?
Describe the world
 Leads to hypotheses

4
Predict events
 Have a clear sense that science has
multiple aspects
 Simple description
 Systematic, hypothesis testing.
 Core feature of a hypothesis: if “X” then “Y”.

Test theories
 Cause and effect questions involving hypothetical constructs.
 Often controlled experiments or complex correlation designs.

Test applications of theories
 Testing interventions or policy change
Foundations ofBasic
Research
Elements of a Research Project
Phenomenon
Big picture / question
Begin with the “big question”
Core elements of a research study
You should understand
Theorythese steps by now
… articulate a clear theory
Hypothetical Constructs
Causal explanation
Hypothesis
Operational definition
Specific prediction
Methods
Measurement v.
experimental
Data / Results
• Descriptive data
…and derive concrete
hypotheses.
Then specific methods, the
core of a scientific study.
Then actual data & results…
• Test hypothesis
Discussion
… implications for the theory
Implications for theory
Conclusions
Future research?
5
…and larger issues.
Foundations of
Research
Basics of Design: Internal Validity
6
Internal Validity:
Can we validly determine what is causing the results of the
experiment?
General Hypothesis: the outcome (DV) is caused only
by the experiment itself (Independent Variable).
Confound: a “3rd variable” (unmeasured variable other than the
Independent Variable) actually led to the results.
Core Design Issues:
 The experimental & control groups are exactly the
same at baseline
 A Confound is when the groups differ for some other
reason, e.g., self-selection into the study.
Foundations of
Research
True v. quasi-experimental
designs
7
True experiments:
Quasi-experiments:
Emphasize internal validity
 Assess cause & effect (in
relatively artificial environment)
 Test clear, a priori hypotheses
Emphasize external validity
 Describe “real” / naturally
occurring events
 Clear or exploratory hypotheses
Has a control group
Participants randomly assigned
to exp. or control groups
 Participants & experimenter
Blind to assignment
Non-equivalent groups
 Existing groups
 Non-random assignment
 Participants not blind
 Self-selection
Full control may not be possible
Control study procedures
 Manipulate independent variable  May not be able to manipulate the
independent variable
 Control procedures & measures
 Partial control of procedures &
measures
Foundations of
Research
8
External validity: summary
External
Is the
sample Validity:
typical of the
Can we validly generalize
frompopulation?
this experiment to the larger world?
larger
Is the
outcome
measure
representative,
valid &
reliable?
The
research
Sample:
The
Dependent
Variable
The study
structure & The research
Setting:
context
The
Independent
Variable
Is this
typical of
“real world”
settings
where the
phenomenon
occurs?
Does the experimental manipulation (or measured
predictor) actually create (validly assess…) the phenomenon
you are interested in?
9
Validity & Research approaches
Foundations of
Research
Observation or Measurement
Simple Description
Qualitative
Quantitative
Explore the actual
process of a
behavior.
External
Describe a
behavioral or
social trend.
Experiments
Correlational
Studies
Quasiexperiments
“True”
experiments
Relate measured
variables to each
other to test
hypotheses.
Test hypotheses
in naturally
occurring events
or field studies.
Test specific
hypotheses via
controlled “lab”
conditions.
validity
Internal validity
Less control:
More control:

Observe / test phenomenon under
natural conditions.

Create the phenomenon in a
controlled environment

More accurate portrayal of how it
works in nature

Address specific questions or
hypotheses

Less able to interpret cause & effect

Better interpret cause & effect
 Know what these research strategies represent & how they differ.
 Understand the trade-off of internal & external validity across them.
Foundations of
Research
Group
10
Quasi-experiments without a control group:
Observe1
Intervention or event
Observe2
Observe1
Confound
Observe2
Threats to internal validity (confounds):
 History
Historical / cultural events occur between baseline &
follow-up.
 Maturation
Individual maturation or growth occurs between
baseline & follow-up.
 Reactive measures
People respond to being measured or being a
measured a second time.
 Statistical regression
Extreme scores at baseline “regress” to a more
moderate level over time.
 Mortality / drop-out
People leave the experiment non-randomly (i.e., for
reasons that may affect the results…).
 You do not need to memorize these, just get the logic.
 What is a confound? Why is that important?
Foundations of
Research
Sampling
Sampling overview
Who do you want to generalize to?



Who is the target population?

broad  external validity

narrow  internal validity
How do you decide who is a member?

demographic ?

behavioral?

attitudes or beliefs?
What do you know about the population already
– what is the “sampling frame”?
11
Foundations of
Research
12
Sampling overview
Who do you want to generalize to?

Who is the target population?

How do you decide who is a member?

What do you know about the population already
– what is the “sampling frame”?
Will you use a:
Probability or random sample?
Non-probability or convenience
sample?
targeted / multi-frame
 snowball…

 Most externally valid & representative
 Assumes: • Clear sampling frame
• Population is available
 Less valid for hidden groups.
 Less externally valid
 Best when:
 No clear sampling frame
 Hidden / avoidant population.
Foundations of
Research
The “Common Rule” criteria for Human Subjects Protection
The Common Rule
 Minimize risks
These comprise a
Cost – Benefit
analysis.
 Risks must be reasonable
 Recruit participants equitably
 Informed consent
 Understand what each of
these mean.
 Document consent
 Monitor for safety
 Protect vulnerable participants &
maintain confidentiality
13
Foundations of
Research
Belmont Report
14
(CITI training)
1. Respect For Persons

Exercise autonomy & make informed choices.
2. Beneficence
 Minimize risk & maximize of social/individual benefit.
3. Justice
 Do not unduly involve groups who are unlikely to benefit.
 Include participants of all races & both genders
 Communicate results & develop programs/ interventions
 You know these from your
CITI training.
 Generally understand them;
be able to recognize these
key values.
Foundations of
Research
15
Descriptive research
Quantitative
Qualitative or
Observational
Existing data
Describe an issue via
valid & reliable
numerical measures
Study behavior “in
nature” (high
ecological validity).
Use existing data for
new quantitative (or
qualitative) analyses
Simple: frequency
Qualitative
Accretion
 Interviews
 Study “remnants” of
behavior
counts of key
behavior
“Blocking” by other
variables
Correlational
research: “what
relates to what”
 Focus groups
 Textual analysis
Observational
 Direct
 Unobtrusive
 Wholly non-reactive
Archival
 Use existing data to
test new hypothesis
 Typically nonreactive
 What does it mean for research to be ‘reactive’?
Psychology 242, Dr. McKirnan
Descriptive Research.
Foundations of
Research
16
Correlation designs: Drawbacks & fixes
Causality; a simple correlation may confuse cause & effect.
?
Depression
Alcohol
consumption
Confounds!; unmeasured 3rd variable problem
General optimism
Hemlines
?
Stock market
Dealing with confounds: Use complex measurements or samples to
eliminate alternate hypotheses.
 Understand both these interpretation difficulties.
Foundations of
Research
17
Types of numerical scales
Ratio
zero point grounded in physical property; values
are “absolute”
physical description: elapsed time, height
Continuous
scales (scores
on a continuum)
Interval
no zero point; scale values relative
behavioral research, e.g., attitude or rating scales.
 Be able to provide or
Ordinal
rank order: Simple finish place, rank in
organization...
Categorical
‘values’ = categories only
inherent categories: ethnic group, gender, zip
recognize examples
of these scale types
Foundations of
Research
18
Scales and Central Tendency
Measure of Central tendency


Mode (most common score)

Median (middle of distribution) 

Mean (average score)

Typically used for:
categorical variables
often: bimodal distributions
categorical or continuous variables
highly skewed data
continuous variables only
“normal” distributions
 use different
measures of central
tendency.
Foundations of
Research
Two
measures of variance
Measures of Dispersion or Variance
1. Range of the highest to the lowest score.
2. Standard deviation of scores around the Mean


“Average” amount each score deviates from the M.
“Standardizes” scores to a normal curve, allowing for basic
statistics.
 You should know these by now
19
Foundations of
Research
20
z
Z
You must know the Z score
 It is the core form of the critical ratio.
 It represents the:
 Strength of the experimental effect
 Adjusted by the amount of error variance
Z=
How far is your score (X) from the mean (M)
How much variance is there among all the
scores in the sample [standard deviation (S)]
=
X–M
S
Foundations of
Research
21
Z and the normal distribution

The normal distribution is a hypothetical distribution of cases in a sample

It is segmented into standard deviation units.

Each standard deviation unit (Z) represents a fixed % of cases

We use Z scores & associated % of the normal distribution to make statistical
decisions about whether a score might occur by chance.
 Remember
approximations of these
numbers
 If you do not fully understand
this slide go back to the
Statistics 1 focus module and
figure it out!!
Foundations of
Research
Normal distribution; Z scores
Use Z to evaluate a score
Distance from M / “error” variance
1.
Calculate how far the score (X) is from the mean (M); X–M.
2.
“Adjust” X–M by how much variance there is in the sample via
standard deviation (S).
3. Z = X–M / S
How “good” is a score of ‘6' in two groups?
Table 1, high variance
Table 2, low(er) variance
Mean (M) = 4, Score (X) = 6
Mean (M) = 4, Score (X) = 6
Standard Deviation (S) = 1.15.
(X-M = 6 - 4 = 2)
Z (X-M/S) = 2/1.15 = 1.74
Standard Deviation (S) = 2.4.
(X-M = 6 - 4 = 2)
Z (X-M/S) = 2/2.4 = 0.88
22
Foundations of
Research
23
Evaluating scores using Z
C. Criterion for a “significantly good” score
X = 6, M = 4, S = 2.4, Z = .88
If your criterion for a “good”
score is that it surpass 90%
of all scores…
X = 6, M = 4, S = 1.15, Z = 1.74
 With high variance a ‘6’ is
not “good”.
 With lower variance ‘6’ is
good.
70% of cases
 I need you to understand the
90% of cases
-3
-2
-1
0
+1
Z Scores
logic of this approach.
+2
(standard deviation units)
Exam #3 study guide
+3
Foundations of
Research

24
Summary
Statistical decisions
follow the critical ratio:
 Z is the prototype critical ratio:
X–M
S
How far is your score (X) from the mean (M)
Z=
How much variance is there among all the scores in the
sample [standard deviation (S)]
=
 t is also a basic critical ratio used for comparing groups:
How different are the two group Means
t=
How much variance is there within each the two groups;
(“standard error of the mean”)
=
M1 – M2
Variance
n grp1
grp1

Variance
n grp2
grp2
 You must understand what a
critical ratio is.
 This slide needs to make
perfect sense to you!!
Foundations of
Research
Plato’s Cave, 6
What does Plato’s Allegory of the Cave tell us about
scientific reasoning?
We cannot observe “nature” directly, we only see its
manifestations or images:
 We are trapped in a world of
immediate sensation;
 Our senses (or
measurements…) routinely
deceive us (they have error).
25
Foundations of
Research
Plato’s Cave, 2
We study hypothetical constructs; basic “operating
principles” of nature
 e.g., evolution, gravity, learning, motivation…



Processes that we cannot
“see” directly…
…that underlie events that
we can observe.
We test hypotheses
about what we can see and
use rational analysis –
theory – to deduce what
the “form” of these
processes must be, and
how they work.
26
27
Foundations of
Research
Why can’t we just observe “nature” directly?
1. We can only observe the effects of hypothetical constructs,
not the processes themselves.
2. We examine only a sample of the world; no sample is 100%
representative of the entire population
3. Our theory helps us develop hypotheses about what we
should observe if our theory is “correct”.
4. We test our hypotheses to infer how nature works.
5. Our inferences contain error: we must estimate the probability
that our results are due to “real” effects versus chance.
 You must understand
these basic concepts
and terms!
Foundations of
Research
28
“Statistical significance”
Testing statistical significance
 We assume that a score with less than 5%
probability of occurring (i.e., higher or lower than 95% of the
other scores) is not by chance alone … p < .05)
 Z > +1.98 occurs < 95% of the time (p <.05).
 If Z > +1.98 we consider the score to be
“significantly” different from the mean
 To test if an effect is “statistically significant”…
 Compute a Z score for the effect
 Compare it to the critical value for p<.05; + 1.98
 Really important
Foundations of
Research
29
Statistical significance & Areas under the normal curve
95% of scores are between Z = -1.98 and Z = +1.98.
Z = -1.98
Z = +1.98
2.4% of
cases
2.4% of
cases
About 95%
of cases
-3
-2
-1
0
+1
+2
Z Scores
(standard deviation units)
+3
Foundations of
Research
With Z > +1.98 or < -1.98 we
reject the null hypothesis &
assume the results are not
by chance alone.
In a hypothetical
distribution:

2.4% of cases are higher
than Z = +1.98

2.4% of cases are lower
than Z = -1.98

30
Statistical significance & Areas under the normal curve
Thus, Z > +1.98 or < -1.98
will occur < 5% of the time
by chance alone.
34.13% 34.13%
of
of
cases
cases
Z = -1.98
of
cases
2.25%
of
cases
-3
-2
2.4% of
cases
95% of cases 13.59%
13.59%
of
cases
2.4% of
cases
Z = +1.98
2.25%
of
cases
-1
0
+1
+2
Z Scores
+3
(standard deviation units)
Foundations of
Research
31
Critical ratio
The strength of the results (our direct
observation of nature)
Critical ratio =
Amount of error variance (the odds that our
observation is due to chance)
Difference between Ms for the two groups
t=
Variability within groups (error)
Between group variance
Mgroup2
Mgroup1
Within-group
variance, group1
control group
Within-group
variance, group2
experimental group
Within group variance
Foundations of
Research
The Critical Ratio in action
 All three graphs have = difference
between groups.
 They differ in variance within
groups.
 The critical ratio helps us determine
which one(s) represent a statistically
significant difference.
32
Be able to answer these:
 How do the between group
variance & within group variance
constitute the critical ratio.
 t represents the critical ratio for
group comparisons: how does t
vary for these three examples?
 Which might reflect a statistically
significant difference?
Low variance
Medium variance
High variance
Foundations of
Research
Central limit theorem
True
scores
(rs, fs, etc.)
Population
M
 We test t
against a hypothetical
“True” normal
sampling distribution of possible ts.
distribution
 We assume that a t derived from a smaller sample will
have more error variance (within-group variance).
 When df > 120 we assume a perfectly normal
distribution of ts.
 When df < 120 we compensate by becoming more
conservative in our judgments.
 This means we set our critical value for for testing t
to a higher value.
<-- smaller
M
larger --->
33
34
Central limit theorem
Foundations of
Research
True
Population M
“True” normal
distribution
Small samples:
 We assume each sample to
have more error
 So a distribution of
samples will be “flatter” &
more errorfull.
Score
Score
Score Score
Score Score
<-- smaller
Score
Score Score Score
M
larger --->
Score
35
Foundations of
Research
True
Population M
“True” normal
distribution
Medium samples:
 We assume less error in
each sample;
 So a distribution of
samples will be more
“normal”.
Score
Score Score Score
Score
Score Score
Score Score
Score Score
Score Score Score Score Score Score
Score Score Score Score Score Score
Score Score Score Score Score Score Score
<-- smaller
M
larger --->
Foundations of
Research
36
The Central Limit Theorem; large samples
True
Population M
Score Score
Score
Large samples:
 We assume each sample
have less error
 The distribution of large
samples will close to
normal.
“True” normal
distribution
Score
Score
Score
Score Score
Score
to
Score Score
Score
Score
Score Score
Score
Score
Score Score
Score
Score Score Score
Score
Score Score Score
Score
Score Score
Score Score
Score Score
Score Score Score Score Score Score
Score Score Score Score Score Score
Score
Score Score
<-- smaller
Score
M
Score Score
Score
larger --->
 Be able to apply the central limit theorem logic to evaluating t.
 Translate that to using the t table.
Foundations of
Research
Central limit theorem & evaluating t scores
1. Smaller samples (lower df) have more variance.
2. So, t must be larger for us to consider it statistically
significant (< 5% likely to have occurred by chance alone).
3. Compare t to a sampling distribution based on df.
4. Critical value for t with p <.05 goes up or down
depending upon sample size (df)
37
Foundations of
Research
A t-table specifies Critical Values:
Alpha Levels
df
8
9
10
11
12
13
14
15
18
20
25
30
40
60
120

0.10
0.05
0.02
0.01
0.001
1.860
1.833
1.812
1.796
1.782
1.771
1.761
1.753
1.734
1.725
1.708
1.697
1.684
1.671
1.658
1.645
2.306
2.262
2.228
2.201
2.179
2.160
2.145
2.131
2.101
2.086
2.060
2.042
2.021
2.000
1.980
1.960
2.896
2.821
2.764
2.718
2.681
2.650
2.624
2.602
2.552
2.528
2.485
2.457
2.423
2.390
2.358
2.326
3.355
3.250
3.169
3.106
3.055
3.012
2.977
2.947
2.878
2.845
2.787
2.750
2.704
2.660
2.617
2.576
5.041
4.781
4.587
4.437
4.318
4.221
4.140
4.073
3.922
3.850
3.725
3.646
3.551
3.460
3.373
3.291
Critical values for testing
whether an effect is
Statistically Significant
Alpha = .05, df = 8
Alpha = .05, df = 18
Alpha = .05, df = 120
Alpha = .01, df = 40
Know how to use a t table.
 What is ‘Alpha’?
 What are Degrees of Freedom
(df)?
 What is a ‘Critical Value’?
38
Central Limit Theorem; variations in
sampling distributions
Foundations of
Research
df = 120, t > ±1.98, p<.05
As samples sizes ( df ) go
down…
df = 18, t > ± 2.10, p<.05
the estimated sampling
distributions of t scores
based on them have more
variance,
df = 8,
 This increases
the critical value
for p<.05.
giving a more “flat”
distribution.
-2
t > ± 2.31, p<.05
-1
0
Z Score
+1
(standard deviation units)
 Get this! -- Be able to go to a t table and apply this logic.
 Give yourself the Statistics focus modules for details.
+2
39
Foundations of
Research
Taking a correlation approach
Correlations
t-test
 We create group differences
on the Independent Variable.
 …and assess how the groups
differ on the Dependent Var.
Difference between groups
standard error of M
Correlation;
 We measure individual differences
on the predictor variable…
 and see if they are associated with
differences on the outcome.
Σ (Z var1* Z var2)
Df (n-1)
40
Foundations of
Research
41
Statistics summary: correlation
Pearson Correlation (r): measures how similar the
variance is between two variables (“shared variance”)
within a group of participants.
 For each participant multiply the
Z scores for the two variables
 Sum across all participants
 Divide by df:
r=
Σ (Z var1* Z var2)
Df (n-1)
Foundations of
Research
Multiple independent variables
Testing hypotheses about > 1
independent variable
Factorial Designs:
 Main effects,
 Additive Effects,
 Interactions
42
Foundations of > 1 independent variable
Research
43
Include a ‘control’ variable as a second I.V.
1. Block the data by gender, age, race, attitudes, etc.
2. Test if the main Independent Variable has the same
effect within both groups
What is the effect of self-reflection on stress reduction?
EXAMPLE
 Hypothesis: training in self-reflection helps buffer the stress of
exams.
 2nd Question: is that effect the same in women and men? [old v.
young, etc…]
 Main effect: Self-reflection training  less stress
 Interaction: training  less stress worked for women, not men.
 Conclusion: Including a ‘control’ variable helped clarify the results.
44
Foundations of > 1 independent variable
Research
Testing more than one Independent Variable
① Separate, ‘main effects’ of
each I.V.
Does each variable by itself
significantly affect the outcome?
② Additive’ effects of 2+ I.V.s.
What is the combined effect of
these variables?
③ Test interaction of 2 or
Does the effect of each I.V.
depend upon the other IV?
more I.V.s
 Know the difference between a main effect, an
additive effect, and an interaction.
Foundations of
Research
Interaction example: Genetics, stress and depression
Participants’ genotype and level of childhood trauma
interact in depression.
There is a general
(main) effect
whereby more
trauma leads to
greater likelihood
of adult
depression
45
Foundations of
Research
Interaction example: Genetics, stress and depression, 2
However … the effect of trauma interacts with genetics
 So, the effect of maltreatment on adult
depression depends on the level of a second
variable, genetic disposition.
 Unless there is vulnerability there is no
effect of trauma on later depression.
Childhood trauma
has no effect in
people who have no
genetic vulnerability.
 Understand clearly why/how this is an
interaction, not a main effect or additive
effect.
 Also understand how the interaction tells us
much more than the simple main effect.
The effect of
different levels of
trauma on
depression depends
on the person’s level
of genetic
vulnerability
46
Foundations of
Research
Example of a 3-way interaction
47
Figure
Mean
ratings of subjective
stimulation
Here3 is
another
variation
on and sedation on the BAES
under 0.65 g/kg alcohol and placebo in women and men.
an interaction effect.
Alcohol (v. placebo) made
men much more stimulated.
Alcohol made women much
more sedated
Multiple independent variables
Foundations of
Alternate
portrayal of 3-way mood interaction
Research
Placebo conditions do not show
much effect
The alcohol conditions show a
classic “cross-over” effect for
gender & mood;
50
M BAES subscale scores
45
Men get aroused
40
35
Men, Alcohol
Men, Placebo
Women, Alcohol
Women, Placebo
30
25
20
15
10
Women get sedated
5
0
Stimulation
Sedation
48
Foundations of
Alternate
portrayal of 3-way mood interaction
Research
The alcohol conditions show a
Placebo conditions do not show
 In this
interaction
the effect of alcohol classic “cross-over” effect for
much
effect
gender & mood;
on emotions had an opposite effect for
men v. women.
50
 Men
45 get aroused, women get sedated.
M BAES subscale scores
40
 In both
examples the effect of one IV on the
35 depends upon a second (or third)
outcome
IV. 30
25
20
Men get aroused
Men, Alcohol
Men, Placebo
Women, Alcohol
Women, Placebo
15
10
Women get sedated
5
0
Stimulation
Sedation
49
Foundations of
Research
Multiple IVs; summary 2
50
Multiple Independent Variables / Predictors:

Are critical to theory development and testing:
Stress or other environmental events can “switch on” genes that create
psychological or other problems; genetic dispositions and environment
are not separate processes.

Establish key “boundary conditions” to theory: when
and among whom does a basic psychological process
operate?
Alcohol makes it more difficult to inhibit behavior, but primarily among
men.

51
Summary
Foundations of
Research
Basics:

Science = values, not just content and methods

Ways of knowing:

Authority

Intuition

Empiricism

Rationalism
Most central to
Science

Internal Validity

External Validity

Threats to Internal Validity
Internal  external validity
tradeoff
(from lack of control group)
Foundations of
Research

Summary
Sampling:

Who is the target population?

How do you decide who is a member?

Probability sampling


Random element to sampling

Most representative
Non-Probability sampling

Less representative

Best for highly targeted sub-populations
52
Foundations of
Research

Summary
The numbers:

Problems determining causality with correlations

Number scales


Ratio

Interval

Ordinal

Categorical
Distributions

Bi-modal

Skewed

Normal
53
Foundations of
Research

54
Summary
The numbers:

Measures of Central Tendency

Mode
Best for bi-modal data.

Median
Best for highly skewed data.

Mean
Best for normally distributed data.

Standard Deviation (S)

Know Z

Statistical Significance

Z = X–M / S
Critical Ratio
Summary
Foundations of
Research

The numbers:

5% of scores greater
than Z = +1.98
 1.98 is the Critical Value
 … at alpha = p < .05
 t + 1.98 occurs less than 5% of the time by chance
alone
 It is “statisticallly significant”.
 Central Limit Theorem
 Adjust critical value for greater variance stemming from
smaller samples (fewer df)
 t – table shows critial values for different sample dfs.
55
Foundations of
Research

Summary
The numbers:

Statistical effects

Main effect

Additive effect

Interaction
56