Transcript Department of Distributed Information Systems SINTEF

Some ideas around PIADA:
(Picture Indexing: Affect, Description and Availability)
Diana Santos
Information and Communication Technologies
1
PIADA in a nutshell
 Context: making sense of images
 High-level concerns:
 Purpose
 Interaction with text/context
 Cultural factors
 Jokes, irony, metaphor, affect, humour
 Practical setup: Web archive, image repositories, Wikimedia
 Expected outcomes
 Picture ontologies (as well as integration with other NLP ontologies)
 Reasoning with images
 Cross-cultural understanding of image differences
Information and Communication Technologies
2
Presentation outlook
 A short description of the main ideas of PIADA
 5 things not yet solved
 How to go about it? Data; 3 kinds of reasoning
 SINTEF’s LEG (Language Engineering Group)
 PIADA in a Norwegian context
Information and Communication Technologies
3
1. Find images with feelings
 Most image descriptions are about “objective” features like who
(names), where (places), dates, colours and objects
 Asterix sad – no way, and we know there exist these pictures
 Try Hillary happy and Hillary angry and you get mostly the same
pictures: those of Hillary Clinton together with texts that may report
whatever about happiness or sadness of her competitors or fans
Information and Communication Technologies
4
2. Find culturally implicit information
 Japanese people in the underground
(tube): most probably posted by
Japanese people, not with captions
saying that people are Japanese...
 Try Google images and give up
 Man reading. This can be a good
enough caption in an European
museological context, but certainly not
in an Asian, or African context
Information and Communication Technologies
5
3. Find synergy between images and text
 why this caption or this illustration?
why do they work together? creative
use of pictures in text: or: how many
of these images of Sócrates are jokes?
 from the obvious: a happy baby in a
diaper’s advertisement to the
provocative or very very subtle
humour (was it?)
Information and Communication Technologies
6
4. Find the picture’s purpose
http://staff.science.uva.nl/~marx/
 Why is this picture here?
 Reasoning about a multimedia world...
finding the cause for humour and the
important details
How my students end up....
 Vi roser Anne!
Information and Communication Technologies
7
5. Help discuss or describe pictures for different
purposes
Properties of pictures are often mentioned in some contexts
 (didactical, antropological, documentary, scientific) to focus on
particular details: see the character behind Jesus, see the tree on the
left, note the tool on his hand, look at the back wings, at the dark
clouds, at the tumor ...
 (police or law courts) to explain why the picture was taken: revolver
under the table, near the corpse
 (artistic setting): in sunlight, in rainy wather, with a special lins...
Information and Communication Technologies
8
In a nutshell, we believe that
 there are many aspects of picture description and reasoning that have
not received any or enough attention, namely
 emotions
 creativity and humour
 crosscultural differences
 intertextuality with pictures
 a natural language processing angle is the right way to attack them
Information and Communication Technologies
9
Real applications
 Significantly enhancing image bank providers activity
 Helping professionals that need images and text
 teachers
 museum staff
 encyclopedia authors (the Wikipedia community)
 other multimedia content providers: for games, educational CDs, textbooks
 advertisers
 historians and biographers
 Common multilingual image search
Information and Communication Technologies
10
How to go about it? Data
 Image collections to study
 Image ontologies and folksonomies available
 “Text and image” collections
 Wikipedia
 Web pages (Internet archive)
 Special sites and or multimedia products (guided tours)
 Image search logs
 Elicitation collections: sets of stories about image search
 Game results: eliciting similarities or associations among images
Information and Communication Technologies
11
How to go about it? Reasoning 1
 If you want to find a picture of a strong healthy man, or of a genius,
you probably find an instance of a person that illustrates these
qualities (such as Johny Weissmuller or Einstein and look for them)
 If you want to find a picture of Asterix angry, you can look for more
“objective” descriptions such as Asterix shouting or Asterix beating
 If you want to illustrate the property black you probably look for
concepts where black is stereotypical, such as coffee...
Information and Communication Technologies
12
How to go about it? Reasoning 2
 You need to know the context of the annotation or text to know what
should be implicit and what should be probably commented out
 searches in .pt for Sócrates are probably about the prime-minister but
in .no, after the philosopher, comes the Brazilian football player 
 the stereotypical image for man or woman is obviously different
depending on the gender of the beholder, no matter sexual orientation,
and the same for places and cultures
 a “typical” restaurant (and its food) varies widely
 pictures captioned Lisbon (or Dublin, or Helsinki) are most often by
tourists. People who live in Lisbon give the precise names of what
they take the picture of, or don’t even bother to specify location
Information and Communication Technologies
13
How to go about it? Reasoning 3






Why are the images chosen?
What is the kind of connection?
What is their import?
Which other associations -- interpictuality -- they bring?
Why can they be considered offensive or funny?
Do they feel old-fashioned? Do they feel modern?
Information and Communication Technologies
14
Language engineering at SINTEF






Question answering
Ontologies
Geographical reasoning
Contrastive studies
Information extraction
Corpus search
We believe all these pieces will help us to address the image search
and indexing issue.
Information and Communication Technologies
15
PIADA in a Norwegian context
We want to develop specific knowledge on images in Norwegian
 The vocabulary of images and image search in Norway
 Demo collections
 Study of user behaviour: what do people ask for, what do they want to
see?
 Picture reasoning ontologies
 We hope that by cooperating with commercial actors we will do
something useful not only for research purposes
Information and Communication Technologies
16
Specific proposal
 brukerstyrte innovasjonsprosjekter (BIP)
OR
 kompetanseprosjekter med brukermedvirkning (KMB)
 Scanpix Norge as prime contractor
 SINTEF writes most of the proposal
 ABM-utvikling is also involved
 Other Norwegian actors also contacted
Information and Communication Technologies
17