Transcript Document

Randomization workshop
MAY 22, 2014
ECOTS WORKSHOP
PRESENTERS: NATHAN TINTLE AND BETH CHANCE
HOUR #2
Overview of next hour
 Hour #2 (after short 5 minute break)
 First 10-15 minutes: The ISI curriculum: What, how and
why*
 Next 15-20 minutes: Activity: Is yawning contagious?*
 Final 10-15 minutes: Cautions, implementation,
assessment*
 Final 10-15 minutes: Next steps, class testing, ongoing
discussion*
 *Ask questions both during and immediately
following each presentation
The ISI curriculum
INTRODUCTION TO STATISTICAL
INVESTIGATIONS
AUTHORS: TINTLE, CHANCE, COBB, ROSSMAN,
ROY, SWANSON, AND VANDERSTOEP
PRELIMINARY EDITION AVAILABLE VIA WILEY
FALL 2014
Goals
 Introduce the particular curriculum that we have
developed over the last 4+ years




Goals
Distinctive features
Technology
Key things to keep in mind
Vision
 An alternative Stat 101 (Algebra-based intro stats)
course which uses randomization and simulation to
motivate inference
 GAISE from ground up
 Not alienating client departments
Six distinctive features to achieve the goals
 1. Spiral approach to the 6-steps of statistical
investigation
Six-Step Process
Six Step Process
Students have been able
to consider the entire
statistical process:
 Can dolphins
communicate?
 Performance of
Buzz/Doris
 Assessing statistical
significance
 Draw appropriate
conclusions
6 distinctive features to achieve the goals
 1. Spiral approach to the 6-steps of statistical
investigation

Start in Preliminaries

Revisit repeatedly throughout the book, starting with simpler data
(e.g., single binary variable), and moving through a variety of more
complex data situations

Deeper and deeper look at the 6-steps as the course moves on

Emphasizes a big picture, research-oriented view of statistical
reasoning
6 distinctive features to achieve the goals
 2. Randomization-based introduction to statistical
inference

Use simulation and randomization to first introduce statistical
inference

Transition to traditional (asymptotic methods) as a prediction
to the simulation/randomization results

Simple and direct connections between method of data
production, method used to analyze the data, and the
appropriate scope of conclusions
6 distinctive features to achieve the goals
 3. Focus on the logic and scope of inference (the “pillars” of
statistical inference
 Logic:
 Significance (How strong is the evidence?)
 Confidence (How large is the tendency for difference and how confident
can we be in our inferences?)
 Scope:
 Generalizability (To which population can the conclusion be reasonably
generalized?)
 Causation (Is a cause-effect conclusion possible?)
 Once we have the tools, we ask these four questions of nearly
every data set we look at; as part of the entire 6-step statistical
investigation process we walk through nearly every time
6 distinctive features to achieve the goals
 4. Integration of exposition, explorations, and
examples.
Overview of each section in the book
Common introduction
Exploration
Example
Common conclusion
6 distinctive features to achieve the goals
 Lots of flexibility with how to walk through material
within each section

Key idea: Examples and explorations do not depend on each
other; for example, definition boxes are in both
6 distinctive features to achieve the goals
 5. Easy to use technology throughout

Freely available suite of web-applets
Visualizing simulation and randomization
 Integration of simulation and theory-based approaches
 Pasting datasets


Allows for supplementing with a traditional software package
6 distinctive features to achieve the goals
 6. Real data from genuine studies

Taken from a variety of fields of interest; popular appeal

Real, published research in many cases; some student gathered
datasets as well

Exercises, in-depth investigations, research articles
Content sequencing
 Traditional Stat 101
 1. Descriptive statistics and study design
 2. Probability and sampling distributions
 3. Inference
 Our Stat 101
 Unit 1. Introduction to the four pillars of statistical inference
 Unit 2. Comparing two groups
 Unit 3. Analyzing more general situations
Content sequencing
 Unit 1. Four pillars

Preliminaries. Statistical thinking (6-steps), Variability, and
Probability (Long-run frequency)

Chapter 1. Significance (3-S process, chance model); one proportion

Chapter 2. Generalization (To whom can we generalize?); one
proportion, one mean; types of errors

Chapter 3. Confidence (Range of plausible values; 2SD); one
proportion, one mean

Chapter 4. Causation (Is cause-effect possible?)
Content sequencing
 The following chapters all have a similar flow
 Descriptive statistics
 Simulation/Randomization approach
 Theory-based approach
 Unit 2. Comparing two groups
 Chapter 5. Comparing two proportions
 Chapter 6. Comparing two group on quantitative response (means or
medians)
 Chapter 7. Comparing two paired groups (on quantitative response; and,
one sample t-test)
 Unit 3. More general situations
 Chapter 8. Comparing more than two groups using proportions
 Chapter 9. Comparing more than two groups using means
 Chapter 10. Analyzing two different quantitative variables
Content sequencing
 Comments
 Descriptive statistics are “just in time”; chapters are focused
more on type of data (allows for the application of all 6-steps)

Probability and sampling distributions come up throughout
the book using tactile and computer simulations to estimate
sampling distributions; no formal rules of probability needed

Theory-based approaches are merely convenient alternatives
to simulation which predict what would happen if you
simulated, assuming certain conditions are met
Pedagogy
 Built a course from the ground up that was based on
GAISE principles






Statistical literacy and thinking
Conceptual
Active
Real data
Technology to drive understanding
Assessments for continuous improvement
Is it working?
 Preliminary evidence is positive

Students, instructors enjoy it and appear to be learning more

We’ve documented learning gains in a number of key areas with
preliminary versions of the curriculum (Tintle et al. 2011), with little
to no evidence of declines vs. the standard curriculum in other areas

These gains are retained longer by students in this curriculum than
with the traditional curriculum (Tintle et al. 2012)

Still are actively gathering assessment data across multiple
institutions every semester with a long-term vision of continual
improvement to maximize student learning
MORE LATER THIS HOUR
Comparisons with other curricula
 CATALST—focus on modelling
 Lock5 – different content ordering; bootstrapping
**Note: Others are under-development
Final remarks
 Why so much time on proportions and not
quantitative?


Easiest place to start
Cover 4-pillars, 6-steps and 3-S then apply them all,
everywhere
Comparing Two Proportions
Chapter 5
5.1: Descriptive statistics for 2 proportions
5.2: Inference with Simulation-Based Methods
5.3: Inference with Theory-Based Methods
Exploration 5.2: Is yawning
contagious?
 http://www.discovery.com/
tv-shows/mythbusters/
videos/is-yawning-contagious-minimyth.htm
Is yawning contagious?
Are people who see someone yawn more likely
to yawn themselves?
 Mythbusters recruited 50 people
 Randomly assigned to 3 rooms; 2 with yawn seed
planted, one without
Example questions from guided discovery
‘exploration’

Think about why the researchers made the decisions they did.

Why did the researchers include a group that didn’t see the yawn seed in this
study? In other words, why didn’t they just see how many yawned when
presented with a yawn seed?

Why did the researchers use random assignment to determine which subjects
went to the “yawn seed” group and which to the control group?

Is this an observational study or a randomized experiment? Explain how you
are deciding.

The researchers clearly used random assignment to put subjects into groups.
Do you suspect that they also use random sampling to select subjects in the
first place? What would random sampling entail if the population was all flea
market patrons?
Example questions
The researchers found that 11 of 34
subjects who had been given a yawn seed
actually yawned themselves, compared with
3 of 16 subjects who had not been given a
yawn seed.
Organize this information into the
seed planted
Yawn seed not
following 2×2Yawn
table:
planted
Subject yawned
Subject did not
yawn
Total
Total
Is yawning contagious?
100%
90%
Did subject yawn?
Results
80%
Didn't yawn
70%
Yawned
60%
50%
40%
30%
20%
10%
0%
Yawn seed
No yawn seed
Did subject get randomly assigned to yawn seed
group?
Yawned
Didn’t yawn
Total
No yawn
Yawn seed
seed
Total
11 (32.4%) 3 (18.8%)
14
23
34
13
16
36
50
Is yawning contagious?
 The difference in proportions of yawners is 0.324 –
0.188 = 0.136.
 There are two possible explanations for an observed
difference of 0.136.


A genuine tendency to be more likely to yawn with seed
The 14 subjects who yawned were going to yawn regardless of
the seed and random chance assigned more of these yawners
to the “yawn seed” group
Is yawning contagious?
Null hypothesis: Yawn seed doesn’t make a
difference


Yawn seed or not has no association with whether someone
yawns
𝜋seed = 𝜋no seed or 𝜋seed  𝜋no seed = 0
Alternative hypothesis: Yawn seed increases
chances of yawning
Yawn seed increases the probability of someone yawning;
association
 𝜋seed > 𝜋no seed or 𝜋seed  𝜋no seed > 0

Is yawning contagious?
 The parameter is the (long-run) difference in the
probability of yawning between yawn seed and no
seed groups
 Our statistic is the observed difference in
proportions 0.324 – 0.188 = 0.136
Is yawning contagious?
If the null hypothesis is true (yawn seed makes no
difference) we would have 14 yawners and 36 nonyawners regardless of the group they were in.
 Any differences we see between groups arise solely
from the randomness in the assignment to the
groups.

Is yawning contagious?
 We can perform this simulation with index cards.


14 blue cards represent the yawners
36 green cards represent the non-yawners
 We assume these outcomes would happen no
matter which treatment group subjects were in.
 Shuffle the cards and put 34 in one pile (yawn
seed) and 16 in another (no seed)
 An yawner is equally likely to be assigned to each
group
 In class we do this!
Is yawning contagious?
 First simulation
 9 blue (yawners) and 25 green (non-yawners) in yawn seed
group
 5 blue (yawners) and 11 green (non-yawners) in no yawn seed
group

Difference in proportions?

9/34 – 5/16 = -0.048

“Chance” value of the statistic

Repeat many times
Is yawning contagious?
 Confession
 We tweaked the data!

Actually 10/34 yawners in seed group
4/16 in control group

Difference is only 4.4%!

By this time our students realize that’s not enough to be
statistically significant (even though Adam and Jamie didn’t)

Cautions, implementation and
assessment
Cautions
 The good; Question on small p-values
 National sample (not randomization)


Fall 2013, dozen institutions using ISI text


Pre-test: 50%, Post-test: 69%
Pre-test: 44%, Post-test 84% (some nearly 100%)
SERJ article (2012)

Retention of this concept is good 4 months later
 Not going to solve all of your problems!
 Assessment data is positive, but doesn’t mean everything is
better (concepts some better, much the same; attitudes
similar)
Cautions
 Biggest misconceptions we create with this approach:
1. Need multiple samples in real life to analyze data
Solution: Emphasize reality vs. pretend world where null is true
2. Thinking you have proven /gotten evidence for the null hypothesis
Solution: focus on the idea of the assumptions behind the simulation and the idea of
modelling in general
3. Still get a little dependent on mean/proportion
Solution: We are hoping to show more transfer questions so they can use any statistic
they come up with
4. Assuming too much student background?
Solution: Have included the preliminary chapter for those who want a real quick
introduction to “background” ideas
How to convince others
 Focus on what they get
 Better understanding of logic and scope inference
 Focus on 6-steps (scientific reasoning)
 Real studies/research
 Still coverage of the theory based test or tests you know
 Still coverage of descriptive statistics topics
 Conceptual understanding of probability and sampling
distributions
 Good transition to applied second course (stat/math dept or
client dept.
How to convince others
 Content re-ordering and re-focus more than content
change
 What are your needs? It will still meet them, and
likely do even better at meeting them then the
current course.
How to convince others
 Embracing active, guided discovery pedagogy which engages
students and improves student learning (Guidelines of Assessment
and Instruction in Statistics Education; GAISE)
 “How do you do this all in one semester?”
 Efficiency of approach/similarities between framework of inference
 More accessible
 Focusing more on the important stuff (inference)

Topics we don’t emphasize:

Reading probabilities from a table
 Notation
(doing more and more “in words”)
 Using a random number table
 More getting less on data cleaning, other sampling methods
and other subtleties
How to convince others
 But what about probability? What about the central
limit theorem?



Not core to our approach
Some are supplementing for AP
More questions from math/stat colleagues than clients
 Does it really work?
 Published assessment data
How to convince others
 But what about the second course?
 Developing second course materials that flows out of this
 Segway to research methods or other ‘traditional’ second courses fine
 But what about a large class/online?
 Nathan and Beth both have done/are doing online/hybrid
 Applets are good outside of class; demo in class, use outside of class
 But what about other text/software
 Integration has been done many ways so far (R, SPSS, Minitab, etc.); just
explorations as ‘labs’ with traditional text
 But what about more ‘mathy’ students
 Faster
 More formulas
Q+A
Next steps, class testing,
ongoing discussion
Next steps
 What first?



Examine different curricula
Think about an action plan. Try a day? Try a course? Talk with
colleagues? Talk with an author? What are your learning goals? How
are you doing on them?
Participate in a longer workshop

Chicago, Sioux Center (IA), San Luis Obispo, Flagstaff and Boston (all
this summer). See http://math.hope.edu/isi for details.
 Lots of differences? Lots of options—what’s correct?



No consensus
Would like to make decisions based on assessment
YOU have something to offer here!
Next steps
 Sign up for copy of the book (on the post-workshop
evaluations)
 You will be contacted soon (mid-late summer) re:

Participation in blog
Questions/discussion on randomization
 Monthly themes
 Stipend
 We will be up and running soon



Assessment project—could use your/colleagues students; we’ll
provide reports. Especially non-randomization users!!
We are happy to help you work with your
institution/colleagues to assist in implementation/discussions