Transcript On the Epistemic Value of Reputation

On the Epistemic Value of Reputation:
The Place of Ratings and Reputational Tools in Knowledge
Organization
11th International ISKO Conference
Rome, February 23-26 2010
Gloria Origgi & Judith Simon
Institut Jean Nicod
ENS-EHESS-CNRS
Paris, France
REPUTATION: Overview
•
•
•
•
Background
Introduction
Reputation as Evaluative Social Information
A Rational Model for the Epistemic Use of
Reputation
• Reputational Tools on the Web
• Problems with the Epistemic Use of Reputation
• Conclusions
REPUTATION: Overview
• Guiding Questions
– How to use reputation for epistemic purpose?
– What’s the epistemic value of reputation?
– Is this a good thing or a bad thing?
REPUTATION: Background: LiquidPub
Different methods of
quantifying, assessing &
propagating reputation
Further Information on:
http://project.liquidpub.org
http://liquidpub.wordpress.com
REPUTATION: Background
Thesis:
Types of
Epistemic Sociality
REPUTATION: Overview
•
•
•
•
Background
Introduction
Reputation as Evaluative Social Information
A Rational Model for the Epistemic Use of
Reputation
• Reputational Tools on the Web
• Problems with the Epistemic Use of Reputation
• Conclusions
REPUTATION: Introduction
Hero
Sinner
Drunkard
Reputation as Heuristic:
Source:
http://en.wikipedia.org/wiki/File:Matteson_Scarlet_Letter.jpg
http://en.wikipedia.org/wiki/The_Scarlet_Letter
Reputation as a way to classify social types
within the community that will allow its
member to manage their relations with
others, to make inferences and predictions
about their behavior, i.e. to construct a basic
"social map" that will help them orient in their
society.
REPUTATION: Introduction
Hero
Sinner
Drunkard
Reputation as Heuristic:
Source:
http://en.wikipedia.org/wiki/File:Matteson_Scarlet_Letter.jpg
http://en.wikipedia.org/wiki/The_Scarlet_Letter
Reputation as a way to classify social types
within the community that will allow its
member to manage their relations with
others, to make inferences and predictions
about their behavior, i.e. to construct a basic
"social map" that will help them orient in their
society.
REPUTATION: and Knowledge Organization
• Reputation as social information about the value of
people, systems and processes that release
information.
• Focus: relationship between this special form of
social information and the processes of knowledge
organization and evaluation.
• More precisely, we argue not only that
– (1) we make use of other people's reputations to evaluate
information in various ways
– (2) within systems, like the Web, that make possible the
easy and dynamic organization and re-organization of
knowledge, we also use our own rankings to determine new
content and generate new categories.
REPUTATION: Overview
•
•
•
•
Background
Introduction
Reputation as Evaluative Social Information
A Rational Model for the Epistemic Use of
Reputation
• Reputational Tools on the Web
• Problems with the Epistemic Use of Reputation
• Conclusions
REPUTATION: as Evaluative Social Information
• Reputation is the informational track of our past
actions, it is the credibility that an agent or an item
earn through repeated interactions.
• Reputational Cues are indicators/proxies of
reputation where quality of objects or agents cannot
be directly assessed
• Relying on reputational cues is an efficient way of
shaping the too rich informational landscape around
us by creating new relevant categories.
REPUTATION: as Evaluative Social Information
• In an information-dense environment, where sources
are in constant competition to get attention and the
option of the direct verification of the information is
often simply not available at reasonable costs,
evaluation and rankings are epistemic tools and
cognitive practices that provide an inevitable shortcut
to information
• Modest Prediction: The higher the uncertainty on the
content of information, the stronger is the weight of
the opinions of others in order to establish the quality
of this content.
REPUTATION: Overview
•
•
•
•
Background
Introduction
Reputation as Evaluative Social Information
A Rational Model for the Epistemic Use of
Reputation
• Reputational Tools on the Web
• Problems with the Epistemic Use of Reputation
• Conclusions
REPUTATION: A Model for the Epistemic Use of Reputation
Lehrer & Wagner (1981) “Rational Consensus in
Science and Society”
• Proposes a model for rational decision making
processes in science, society and the arts that
makes epistemic use of reputation
• It rests upon the employment of consensual
probabilities, utilities and weights
• For decision making processes to be rational, it is
central that all evidence or empirical information
available for the topic of concern has to be used
• Experimental + Social Information
REPUTATION: A Model for the Epistemic Use of Reputation
• Social information = information about the expertise of
other experts on issues at hand = Reputation
• Example: Expert Dilemma: Do we need to vaccinate
large parts of the population to prevent a pandemie?
– Step 1: each expert gives a weights other experts’
competency
– Step 2: weights are laid open
– Step 3: revision of own weights taking the others‘ assessment
into account depending on the weights assigned to them
– Repeat cycle till consensus is achieved…
• Once these consensual weights are achieved, they
can be applied to answering the question of concern
by weighting each member’s votes on the issue with
their consensual personal weight.
REPUTATION: A Model for the Epistemic Use of Reputation
• Lehrer & Wagner propose a model of how to
rationally reach consensus that rests upon the
epistemic use of reputation
• This implies that reputational information, i.e. social
information about other people that is evaluative, is
epistemically useful.
• Do we need such formal models?
• Epistemic use of reputational cues does not have to
follow such a formal method. But on the Web,
models similar to this one are embedded and hidden
within different applications.
REPUTATION: Overview
•
•
•
•
Background
Introduction
Reputation as Evaluative Social Information
A Rational Model for the Epistemic Use of
Reputation
• Reputational Tools on the Web
• Problems with the Epistemic Use of Reputation
• Conclusions
REPUTATION: Reputational Tools on the Web
http://www.briansolis.com/2008/08/introducing-conversation-prism/
REPUTATION: Reputational Tools on the Web
• What the Web makes possible today is an algorithmic treatment
of methods of gathering social information to extract knowledge.
Ratings and rankings on the Web are the result of collective
human registered activities with artificial devices.
• However, the control of the heuristics and techniques that
underlie this dynamics of information may be out of sight or
incomprehensible for the users who find themselves in the very
vulnerable position of relying on external sources of information
through a dynamic, machine-based channel of communication
whose heuristics and biases are not under their control.
• Thus, the reputational tools that are proliferating on the Web
should be scrutinized by epistemically responsible users who do
not want to accept too naïvely the outcome of a process they do
not control.
REPUTATION: Reputational Tools on the Web
REPUTATION: Reputational Tools on the Web
REPUTATION: Reputational Tools on the Web
REPUTATION: Reputational Tools on the Web
Interestingness!
“There are lots of elements that make
something 'interesting' (or not) on Flickr.
Where the clickthroughs are coming from; who
comments on it and when; who marks it as a
favorite; its tags and many more things which
are constantly changing. Interestingness
changes over time, as more and more
fantastic content and stories are added to
Flickr.”
http://www.flickr.com/explore/interesting/
REPUTATION: Reputational Tools on the Web
Interestingness!
“There are lots of elements that make
something 'interesting' (or not) on Flickr.
Where the clickthroughs are coming from; who
comments on it and when; who marks it as a
favorite; its tags and many more things which
are constantly changing. Interestingness
changes over time, as more and more
Interestingness is a new
category based on reputational fantastic content and stories are added to
Flickr.”
mechanisms, making use of
different proxies whose weight
and combination is not obvious!
http://www.flickr.com/explore/interesting/
REPUTATION: Reputational Tools on the Web
• Reputational tools get more and more central on the Web
• Rankings and Ratings provide new arrangements of information
• Early years of 2000: focus on personalized information (MyFeatures)
• Now: trend towards systems of shared preferences, were
people can rely on others’ preferences and rankings to construct
there own access to and categorization of information
• Examples:
– Flickr’s Interestingess
– Twitter-Logic of Followers and Leaders
– LiquidJournal: people or groups create their own journals by selecting
(existing) content and making it available via their selection
REPUTATION: Problems for the Epistemic Use of Reputation
So, all is well, or?
REPUTATION: Problems for the Epistemic Use of Reputation
So, all is well, or?
Well, not quite…
REPUTATION: Overview
•
•
•
•
Background
Introduction
Reputation as Evaluative Social Information
A Rational Model for the Epistemic Use of
Reputation
• Reputational Tools on the Web
• Problems with the Epistemic Use of Reputation
• Conclusions
REPUTATION: Problems for the Epistemic Use of Reputation
1. The danger of misuse of reputation: danger of
epistemic injustice (Fricker 2007), judging epistemic
credibility and social identity (Alcoff 2001)
–
Using proxies that are not valid to assess the reputation
and epistemic credibility of epistemic agents (gender, race,
nationality, institutional background,…)
“testimonial injustice occurs when prejudice causes a
hearer to give a deflated level of credibility to a speaker’s
word” ((Fricker 2007) 1)
REPUTATION: Problems for the Epistemic Use of Reputation
2. Limits of the epistemic usefulness of reputation itself
–
–
–
–
–
How to calculate reputational values in the first place?
What are the pros and cons of different methods: e.g. peer
review versus Amazon-type ratings?
Which proxies should be used and how should they be
combined?
Stability of reputation over time?
Transferability of reputation over domains?
3. Lack of transparency of reputational algorithms and
metrics
–
–
–
How should users be responsible knowers if they do not
understand the functioning, the strengths and weaknesses
of different mechanisms?
How to detect biases, if the mechanisms are not laid open?
Need to make these mechanisms visible and
understandable
REPUTATION: Overview
•
•
•
•
Background
Introduction
Reputation as Evaluative Social Information
A Rational Model for the Epistemic Use of
Reputation
• Reputational Tools on the Web
• Problems with the Epistemic Use of Reputation
• Conclusions
REPUTATION: Conclusions
• Ratings and reputational tools in knowledge
organization have epistemological, practical as well
as ethical implications.
– Epistemological questions: How epistemically warranted is the use
of these tools?
– Practical questions: How to develop these mechanisms? Which
proxies to use, how to combine and weigh them? What’s the status
of these new types of classes, such as interestingness? Can
ratings and ranking serve as middle-ground categorizations?
– Ethical and political question: Epistemic injustices & lack of
transparency: Once reputation mechanisms become formalized
and are embedded within tools, there is a clear danger that
epistemic injustices are inscribed in and reinforced by technology.
REPUTATION: Conclusions
• What is the epistemic values of reputation? Is it useful? Or
rather dangerous?
REPUTATION: Conclusions
• What is the epistemic values of reputation? Is it useful? Or
rather dangerous?
• Both - it is useful and dangerous. But either way, reputational
information, different reputational proxies and methods of
quantifying and combining them are being used extensively on
the Web and elsewhere.
REPUTATION: Conclusions
• What is the epistemic values of reputation? Is it useful? Or
rather dangerous?
• Both - it is useful and dangerous. But either way, reputational
information, different reputational proxies and methods of
quantifying and combining them are being used extensively on
the Web and elsewhere.
• An additional problem on the Web concerns the lack of visibility:
for the users the metrics and algorithms behind different
reputation tools are often unknown.
REPUTATION: Conclusions
• What is the epistemic values of reputation? Is it useful? Or
rather dangerous?
• Both - it is useful and dangerous. But either way, reputational
information, different reputational proxies and methods of
quantifying and combining them are being used extensively on
the Web and elsewhere.
• An additional problem on the Web concerns the lack of visibility:
for the users the metrics and algorithms behind different
reputation tools are often unknown.
• There is an epistemic duty of epistemologists and knowledge
organization scholars to thoroughly analyze these different
reputational practices from epistemological, ethical and political
perspectives.
REPUTATION
Thank you for your attention!
Contact
Judith Simon
Institut Jean Nicod
Ecole Normale Supérieure
29, rue d'Ulm
F-75005 Paris
email: [email protected]
www: http://www.institutnicod.org
tel: +33 (0) 1 443 22 6464
fax: +33 (0) 1 443 22 699
REPUTATION: Problems for the Epistemic Use of Reputation
• Two major problems of using reputation for
epistemic purpose
1) the use of reputation to assess content can be
epistemically beneficial while being morally
questionable
2) limits of the epistemic usefulness of reputation itself
REPUTATION: as Evaluative Social Information
• We want to explore the epistemic value of reputation,
while being aware of the ethical and political
problems that might come with using it for epistemic
purpose.
• Using the judgment on past records to classify an
agent or an item can be epistemologically useful in
the absence or - as is especially relevant today overabundance of information. But it has to be and
remain open to constant scrutiny and revision to be
epistemically useful and ethically just.
REPUTATION: Reputational Tools on the Web
• Early years of 2000: focus on personalized information (MyFeatures)
• Now: trend towards systems of shared preferences, were
people can rely on others’ preferences and rankings to construct
there own access to and categorization of information
• Examples:
– Flickr’s Interestingess
– Twitter-Logic of Followers and Leaders
– LiquidJournal: people or groups create their own journals by selecting
(existing) content and making it available via their selection
REPUTATION: Background
• Epistemic Use and Value of Reputation as ongoing
inquiry by two authors fuelled by different sources
REPUTATION: Background
Two authors Two perspectives
REPUTATION: Conclusions
• What is the epistemic values of reputation? Is it useful? Or
rather dangerous?
• Both - it is useful and dangerous.
REPUTATION: Conclusions
• What is the epistemic values of reputation? Is it useful? Or
rather dangerous?
• Both - it is useful and dangerous. But either way, reputational
information, different methods or reputational cues of assessing
it are being used extensively on the Web and elsewhere.