Transcript content - CORDIS

Human-Centric
Multimedia Research:
Research Opportunities
Nuria Oliver, PhD
Telefonica Research
Multimedia and Data Mining & User Modeling
Scientific Director
Explosion of Digital Data
Information Created, Captured and Replicated
6-Fold Growth
in Four Years
2010
988 Exabytes
2006
161 Exabytes
Source: IDC, 2007
Exabytes?
All words ever
spoken by human
beings (5 exa)
Photo
Print Collection
US Library of Congress
Book
Movie
All printed
material
(multimedia)
Who will create all this data?
2010 Data
988 Exabytes
563 Exabytes
User*
Generated
Content
692 Exabytes
*Consumers and Workers
creating, capturing or replicating
personal informaton
Organizational**
Touch
Content
296 Exabytes
**Transported, hosted
managed or secured
Human-Centric Multimedia
Search &
Discovery
Media
Consume
Media
Produce
Media, tags,
ratings, comments
A FEW RESEARCH
CHALLENGES AND
OPPORTUNITIES
Multi-Modality: Content + context
Multi-modal approaches are needed to
construct novel methodologies to fuse multimodal content and context information
Multi-modal Multimedia
Content Analysis:
Feature Extraction
Similarity Metrics
Ontology definition
Indexing schemes….
Multimedia Context:
Higher level knowledge
generated by users
(tags, comments…)
User interaction data
Wisdom of the crowd
Fusion of content and context-based features
Creation of large collections of labeled training data
Noise filtering by aggregation of contextual
information
Improved search results
Multimedia Tagging
Paris
France
Vacation
Eiffel Tower
June 2009
Multimedia Tagging
• User generated content is rarely annotated  really
difficult, if not impossible to later find it
• When annotated, it is typically done in batch, per
session, not per item
• Tags significantly improve search results alone or
combined with content-based techniques
• Need for novel interfaces to encourage users to
annotate content
– Games with a Purpose
– Annotations at the time of capture
• More research on tag expansion and automatic
tagging
Multimedia Information Overload
Multimedia Information Overload
• Retrieval accuracy is not sufficient due to vast amounts of
available information  too many relevant results
• New orthogonal dimensions need to be used to extend
the notion of relevance and improve retrieval
performance  e.g., aesthetics
• User generated content is of varying quality
• Need for user centric approaches
• Multi-disciplinary approaches:
– Computer scientists, psychologists, human-computer
interaction researchers
– Computer Vision, Pattern Recognition, Machine Learning,
Human Perception, Human Activity Recognition
Example: Near-Duplicate Videos
Video and Audio Feature extraction
Signature Creation
Duplicate Detection
How do users perceive nearduplicate videos?
Do they care about them?
Which features are important
when defining near-duplicates?
M. Cherubini, R. de Oliveira, and N. Oliver, “Understanding near-duplicate videos,” in
Proceedings of ACM MM’09, (Beijing, China), pp. 35–44, ACM Press, October 19-24 2009.
Example: Multimedia Aesthetics
“The interest that a photograph, video or audio piece
generates when perceived by human observers, and that
incorporates both objective and subjective factors”
The Importance of Aesthetics
“Paris Louvre Night”
The Importance of Aesthetics
• User generated content has a wide range of quality and
aesthetic value for the same content
• Aesthetics influence our perception of content
• Highly disregarded in state-of-the-art multimedia retrieval
systems
• Need for computational models of the aesthetic value of
multimedia content
• Need for ground truth databases on image, audio and video
aesthetics
• Need for deeper understanding of
–
–
–
–
The role of aesthetics on user preferences and satisfaction
Universal vs personal aesthetics
Domain-dependent aesthetics
Quality vs aesthetics
Personalization, Recommendation and
Exploratory Search
Personalization, Recommendation and
Exploratory Search
• Future multimedia search and retrieval systems will need
to take into account the user’s preferences, interests and
task at hand in order to return relevant content
• Huge amounts of multimedia data
– Need for recommendations rather than direct search
– Automatic discovery of relevant information to the users
• More research should be devoted to user modeling,
personalization and recommendations of multimedia
content
• Untapped research challenge: Role that the task at hand
plays in determining the optimal multimedia content to
retrieve for the user
Multimedia Storytelling
“The conveying of events with words, images and sounds, often with embellishement. “
Stories or narratives have been shared in every culture and in every land as a means of
entertainment, education, preservation of culture and in order to instill moral values.
Crucial elements of stories and storytelling include
plot and characters, as well as the narrative point of view.
Multimedia Storytelling
• Despite capturing large amounts of digital multimedia
content, most users rarely access the content again
• Sharing the multimedia content is one of the main
reasons why users capture it
• Lack of efficient and scalable tools for browsing, finding
and selecting the desired content
• Multimedia Storytelling: User-friendly, semi-automatic
and scalable (space and time) multimedia tools that
enable users to
– Easily retrieve desired multimedia content
– Create and share the story they want to create from their
content
Exemplary Workflow for MM Storytelling
Multimedia
Analysis Tools
Create Story Slideshow:
“Madrid Christmas 2009”
Flickr e-mail communication
With story slideshow
Storytelling
UI
Storytelling
Algorithms
Face Detection
Face Recognition
Smile Detection
Aesthetics Reranking
Clustering
Near-duplicate Detection
Tag expansion
Automatic Classification
Identify main Actors
Identify main Chapters
Complement content
with external content
Select images based on
*Story Length
*Target Audience
*Target Device
New Multimedia Experiences
New Multimedia Experiences
• Users are increasingly seeking new ways to
experience multimedia content
• Research opportunities combining:
– Music + Video: High-quality visual musical experiences
– Video + 3D reconstruction: 3D video
– Images + Context : Mobile Augmented Reality
• Research challenges in:
– Multimedia analysis: Machine learning, pattern
recognition, computer vision
– User Modeling
– Human-computer interaction
http://research.tid.es/multimedia
[email protected]