Transcript Slides - Gove Allen

A Little Help can be a Bad Thing:
Anchoring and Adjustment
in SQL Query Reuse
Gove Allen
Tulane University
Jeffrey Parsons
Memorial University of
Newfoundland
Gove Allen
November 10, 2006
University of Houston
Code Reuse in
Information Systems
• Code reuse (Cox 1990)
• Study of IS reuse (Frakes & Terry 1996)
• Query Reuse
• SQL is: A standard, portable across
development environments, portable
across DBMS, easily learned
Anchoring and Adjustment
• Tversky and Kahanman 1973
• Heuristic used to reduce cognitive burden
• Bias arises due to insufficient adjustment
• Example from Judgment
• Example from Problem Solving
Anchoring and Adjustment
in Query Reuse
• A query is a candidate for reuse if it
satisfies a similar information request
• Anchoring is promoted by the fact query
executes and produces results
(plausibility)
• Adjustment bias:
Surface level vs. Deep structure
• Also consider role of domain familiarity
Hypotheses
When users reuse queries:
• More errors H1
• Higher confidence H3
• Shorter time H2 • Worse prediction H4
Domain familiarity will lead to:
• Less reuse H5
• Better adjustment H6
In general, users will:
• Adjust surface anchors better than deep ones H7
Domain Familiarity
• Difficult to study empirically
• Virtually impossible to assign subjects to
different levels of domain familiarity
• Parallel database schemata
• Within subject assignment
• Implications for problem-solving in general
Gene Ontology Domain
University Domain
U
ni
v
e
rs
it
y
Gene Ontology Domain
Parallel Questions
How many times have
professors employed
students from Arizona
(state='AZ')?
How many sponsors are
the primary sponsors for
courses that can be taken
for credits ranging from 1
to 3 (credit_range='1-3')?
How many definitions are
there for obsolete terms
(is_obsolete=1)?
How many species are the
primary species for gene
products with a symbol of
ADH2 (symbol='ADH2')?
Experimental Instrument
• Implemented as website
• Allows for presentation of example query
• Collects confidence information
• Allows dynamic query evaluation
• Automatic scoring
Question without Anchor
Question with Anchor
Subjects
• 157 Subjects
• 6 universities in United States
• Introduction to database management
Results for Anchoring
 H1 More Errors
P < .0001
Our sample: Half as likely to produce correct result
 H2 Shorter Time
P < .0001
Our sample: Less than half the time (40%)
 H3 Higher Confidence
P = 0.6981
Our sample: Only 0.6% difference
 H4 Worse Prediction
P < .0001
Our sample: More than 50% more overconfident
Results for Domain Familiarity
 H5 Less Reuse
P = 0.8875
Our sample: Only 0.5% difference
 H6 Better Adjustment
 H6a Surface Anchors
P = 0.7757
P < .0001
Counter
Significance
Our sample: 25% more likely for low familiarity
 H6b Deep Anchors
P = . 0148
Our sample: 7% more likely for high familiarity
Surface vs. Deep Anchors
 H7 Users will adjust surface anchors
better than they will deep anchors
P < .0001
Our sample: Subject almost three time as likely to
correctly adjust surface-level anchors than deepstructure anchors
Implications
• Given an opportunity, query writers will
reuse existing queries
• Reusing queries can lead to higher error rate
• Reusing queries can lead to more
overconfidence
• Adjustment bias is not entirely mitigated by
domain familiarity
• Query writers focus on surface-level changes
when modifying queries
Questions
and
Discussion