Transcript Methods and Tools for Supporting Generalization

Methods and Tools for Supporting
Generalizations
Generalizations from Empirical Studies Session
ISERN 2005
Jeffrey Carver
Introduction

Generalization is more than Meta Analysis

Reactive vs. Proactive



Reactive – Combine existing data to draw conclusions
Proactive – Purposefully plan studies to allow more
general conclusions to be drawn
Existing Data vs. New Data


Existing Data – Work with what is available
New Data – Generate data that allows for the desired
analysis
Generalizations from Empirical Studies
ISERN 2005
2
Mining Existing Data

Data from previous studies

Data sets can be:



Analyzed together with other data sets
Reanalyzed with a different underlying goal
Can be reactive or proactive


Reactive – Meta Analysis
Proactive – Hypothesis Generation
Generalizations from Empirical Studies
ISERN 2005
3
Mining Existing Data:
Meta Analysis

Goal: Draw more general conclusions than are possible
from individual studies

It is difficult to do!

Partially our own fault




No standard template for describing our studies
We do not always completely describe our experimental setting
We do not do a good job of examining and reporting our
assumptions
Partially the nature of our field


Deal with small data samples
People by nature are different, making it difficult to combine
data sets from multiple studies
Generalizations from Empirical Studies
ISERN 2005
4
Mining Existing Data:
Hypothesis Generation

Goals:




Steps:





Develop hypotheses to be tested in future studies
Understand the limits of previous studies
Not to statistically prove anything
Examine datasets for common features (variables)
Analyze each data set separately
Look for patterns to form across data sets and form “Grounded
Hypotheses”
Look for holes in the data
Output:


“Grounded Hypotheses” are good candidates for future
confirmatory studies
Holes in the data provide good opportunities for new exploration
Generalizations from Empirical Studies
ISERN 2005
5
Generating New Data




A more proactive approach
Need to develop a more organized way of
planning new studies
Focus on how to choose the specific
hypotheses or research questions
Two approaches


Lab Packages
Grounded Hypotheses
Generalizations from Empirical Studies
ISERN 2005
6
Generating New Data:
Lab Packages

Facility for the replication and generalization process

Can specific tailoring points



Trade-off: Level of specificity



Helps to maintain some level of consistency
Provides a framework to aggregate different studies
Fixing too many details limits the scope of generalization
Allowing too much tailoring may reduce the generalizability
Goal: Set up a series of studies that complement each
other without fixing too many details
Generalizations from Empirical Studies
ISERN 2005
7



Generating New Data:
Use of Grounded Hypotheses
Hypotheses that are generated based on real
data from previous studies
Provide a good starting point for conducting
confirmatory studies
Grounded Hypotheses:



Help show the relationships that are likely to be true
Reduce the chances of an “unsuccessful” study
The lack of a Grounded Hypothesis could
indicate a good starting point for new research
Generalizations from Empirical Studies
ISERN 2005
8
Implications for Experimenters


To facilitate any of these methods, we need to
do a better job of reporting our studies
In choosing the details and parameters for a
new study, we need to know as much as
possible about the existing studies



Threats to Validity
Assumptions
Threats to Validity

Provide good variants for replications to increase the
generalizability of the whole set of studies
Generalizations from Empirical Studies
ISERN 2005
9
Implications for Experimenters

Researchers also make assumptions:




For each of our studies, we need to carefully
examine any assumptions we may be making



People
Processes
Products
Eliminate them if possible
Otherwise, report them
Assumptions also provide ideal points for
variation in a replication
Generalizations from Empirical Studies
ISERN 2005
10