Transcript Slide 1

BiasTrust: Teaching Biased Users About Controversial Topics
V.G.Vinod Vydiswaran, ChengXiang Zhai, Dan Roth
University of Illinois at Urbana-Champaign
Peter Pirolli
Palo Alto Research Center
CIKM 2012 Research Poster: Information Retrieval Track
Interface variants
Exposing conflicting viewpoints Research Question
Is it healthy to drink milk?
Yes
No
Milk contains nine essential nutrients…
Dairy products add significant
amounts of cholesterol and
saturated fat to the diet...
The protein in milk is high quality, which
means it contains all of the essential amino
acids or 'building blocks' of protein.
Milk proteins, milk sugar, and saturated fat
in dairy products pose health risks for
children and encourage the development
of obesity, diabetes, and heart
disease...
It is long established that milk supports
growth and bone development
rbST [man-made bovine growth
hormone] has no biological
Drinking of cow milk has been linked to ironeffects in humans. There is no
deficiency anemia in infants and children
way that bST [naturally-occurring
bovine growth hormone] or rbST One outbreak of development of enlarged breasts in
in milk induces early puberty. boys and premature development of breast buds in
girls in Bahrain was traced to ingestion of milk from
a cow given continuous estrogen treatment by its
owner to ensure uninterrupted milk production.
Every coin has two sides
 People tend to be biased
 They may be exposed to
only one side of the story
 Presence of Confirmation
bias
 Effects of filter bubble
Challenges in claim verification
 Understand how human biases
affect learning of controversial topics
 Study user interface factors that help
in such learning
BiasTrust Task setup
 Subjects learn more about a
“controversial” topic
 Subjects are shown quotes
(documents) from experts on the topic
 Expertise varies, is subjective
 Subjects are asked to judge if quotes
are biased, informative, interesting
 Pre- and post-surveys measure extent
of learning
Factors studied in BiasTrust
 Does contrastive display help or hinder
learning?
 Do multiple documents per page affect
learning?
 Does sorting results by topic help?
 What is the effect of display of source
expertise on
readership?
which documents subjects consider
biased, novel, or agree with?
(B) Single document, with
contrasting viewpoint
(A) Single document,
with option to view
contrasting viewpoint
(C) Single vs. multiple
documents; each with
contrasting viewpoints
BiasTrust: User Study details
Expertise
Source
Evidence
Examples of Controversial Topics
Pre-survey
 Is milk good for you?
 Is organic milk healthier?
 Does milk cause early puberty?
 Are alternative energy sources viable?
 Nuclear? Solar? Clean coal?
Post-survey
 40 study sessions from 24 participants
 Average age of subjects: 28.6 ± 4.9 years
 Time to complete one study session: 45 min
Agreement
Novelty
Particulars
Bias
Show similar Show contrast Quit
Overall
Milk
Energy
Number of documents read
18.6
20.1
17.1
Number of documents skipped
12.6
13.0
12.1
Time spent (in min)
26.5
26.5
26.6
Study phase
B. Readership higher for expert documents
User Study findings
120.00
[KDD’11, ECIR’12]
Source
Are sources
trustworthy?
How to build
trust models
that make use
of evidence?
How to assign
truth values to
textual claims?
Claim
How to
present
evidence?
Evidence
Users
How to
address
user bias?
Data/Language Understanding
How to find
relevant
pieces of
evidence ?
[KDD-DMH’11]
Data
What kind of
data can be
utilized?
ClaimVerifier
 This work address the challenge of how users perceive
credible information.
 Studying user interactions will help us design better
claim verification systems.
Acknowledgments
100.00
Contrastive
80.00
60.00
 Significant improvement (at p = 0.05 level)
in mean knowledge rating for Milk domain
Readership (in %)
Algorithmic/Computational Issues
A. Contrastive display improves readership
Readership (in %)
HCI
Issues
Particulars Total
20.00
Expertise rating (in “stars”)
Single
Documents rated uniformly at random
0.00
Area Under Curve
Single
display
Contrastive
display
Top 10 pairs
45.00 %
64.44 %
Contrast docs only
22.00 %
64.44 %
Readership change
Absolute Relative
 Subjects tend to not read contrastive viewpoint, if it is
not shown by default.
 Subjects read more when both viewpoints are shown.
Change
Milk
9
7 2
+12.3 % *
Energy
13
8 5
+ 3.3 %
 Significant improvement (at p=0.05 level)
in bias rating for both domains
40.00
|1:P 11:C | 2:P 2 2:C | 3:P 3 3:C | 4:P 4 4:C | 5:P 5 5:C | 6:P 6 6:C | 7:P 77:C |8:P 8 8:C | 9:P 9 9:C |10:P1010:C |
Document position
C. Learning improves with
contrastive display
Expertise rating
Documents rated 1
or 3
 Higher expertise documents tend to be read more.
 Lower expertise documents are read less if ratings
were shown than when ratings were not shown.
Additional findings
 Showing multiple documents per page increases
readership.
 Subjects learned more about the topics they did
not know about.
This research was supported by the Multimodal Information Access and Synthesis
(MIAS) Center at the University of Illinois at Urbana-Champaign, part of CCICADA, a DHS Science and
Technology Center of Excellence, and grants from Boeing, as part of the Information Trust Institute. Some part
of this research was done at Palo Alto Research Center, with grant support from the Office of Naval Research.
Contact
details
[email protected],
[email protected],
[email protected],
[email protected]
Particulars Total
Change
Milk
11
2 9
- 31.0 % *
Energy
7
2 5
- 27.9 % *
Conclusions
 User study helped demonstrate need to show
multiple viewpoints for claim verification.
 Knowledge of expertise rating helps users.
 Showing contrastive viewpoints helps
subjects reduce strongly-held biases.
 This study helped us design better claim
verification systems.
 See additional results in Vydiswaran, V., Zhai,
C., Roth, D., and Pirolli, P., ASIS&T 2012