Transcript Experimental Design

Experimental Design
Anecdotal Evidence (and
worse!)…
Anecdotal Evidence: based on haphazardly
selected individual cases. No attention given to
outliers, lurking variables etc. Works from a fallacy of
making an un-warranted generalization from a small
group of individuals to a larger population. One of the
most common “misconceptions” about statistical
evidence.
What’s wrong with this?
Observation vs Experimentation


An observational study looks for patterns,
correlations etc in a population without
interfering with the population.
An experimental study deliberately introduces
a treatment to elicit a response and to
observe this in a population.
Group Discussion…




Let’s look at 3 questions from your text and
discuss them.
Spend 3 minutes discussing each of these in
turn (we will discuss as a whole between
each one)
Look for problems with the statistical claims
or analyze what kind of statistical argument is
being made in each case.
The questions: 3.1, 3.5, 3.8
Placebo Effect…
Doctor: I’ve got to give you a shot.
Do you have private health
insurance?
Patient: No.
Doctor: Nurse, fetch the placebo.
Doc Martin’s Amazing Brain
Pills…
Suppose you were
foolish enough to take
on of doc Martins
amazing brain pills
before a stats exam –
and you ACED it!
You said it was the pills
but shortly after doc M
was arrested it was
discovered that the pills
were just Smarties!
How do you explain
your performance?
The Placebo Effect (and
variants)


Often the very act of carrying out statistical
trial (interview, treatment etc) affects the
results
A placebo is a “fake” or facsimile treatment
given in place of the real treatment
Good experimental design…

Some terms to learn…






Subjects/units are the individuals being studied
Treatment is a specific experimental condition being
administered
Factors are the explanatory variables in the study
Comparative experiment follows a simple
Treatment  Observe response structure
Bias is a systematic trend to favour a specific outcome
Control group is a group that receive a placebo treatment
Randomized Experiment
Control
Principles of Good
Experimental Design…



Control lurking variables – compare two or
more treatments
Randomize choice between subject/unit and
treatment received
Replicate on as many subjects/units as
possible
Trying to eliminate bias –
Double Blind Experiments…




Researcher bias can creep into an experiment in subtle
ways
Suppose you wanted to test a potentially life saving drug
on a group of very sick children. You were so convinced
about the drug’s effectiveness that you really didn’t want
to give a placebo to some so…
When you gave the real drug you were up-beat and
encouraging but when giving the placebo you were much
more reserved…
You get the idea! To avoid this a double-blind
experiment would randomize the patients and the drugs
so that neither you or the patient would not know ahead
of time which drug you were giving.
Comparison and looking for
Statistical Significance


The purpose of setting up controlled
experiments with controls is to test for
effected change
A result is considered statistically significant if
the observed effect is so large that it is very
unlikely to have occurred by chance.
To be continued…
Block Design Experiments…


A block design experiment divides the
subjects/units into identifiable groups prior to
random sorting.
For example, you may wish to divide the
group by gender, or religion or political
affiliation
Sample Block Design: 3 treatments/2 groups
Group Work…



Discuss the following questions from your
text: 3.10, 3.12, 3.21
If helpful, draw a block diagram
Table B refers to the table of random digits at
the back. To see how to use this consult
example 3.7 on page 235
Sampling and Experiment
Design


A call-in late night radio show runs a caller
poll concerning funding for private, religious
colleges and universities. Callers were asked
to vote Yes – they should be funded or No –
they should not be funded by the
government. The results were 23% Yes and
77% No.
Should policy be based on this survey?
Simple Random Sample


If any one subject or unit in a population is as
likely as another to be sampled then random
selection from this set produces a simple
random sample.
Question: does the random sample represent
the entire population?
Go to excel example…
Stratified Random Samples…



Sometimes (often) a population consists of
numerous identifiable sub-groups or strata.
A random sample should include selections
from each of the strata.
This is called a stratified random sample.
In conclusion…




Make sure you familiarize yourself with the
many terms and definitions introduced in 3.13.3
Understand what anecdotal evidence is and
why it cannot be used as a strong statistical
argument
Understand what a double blind experiment
is
Sample questions: 3.33, 3.34,3.39, 3.47