Transcript ***w***w***x***x***x***x***x***x***x***x***x** x** x***x***x** x***x

Social Multimedia Analytics
A Case Study on Microblog Sentiment Prediction
Rongrong Ji
Director, Intelligent Multimedia Laboratory
Dean Assistant, School of Information Science and Technology
Outline

Introduction



Background
Research Interests
Visual Sentiment Ontology



Motivation
Design Principle
Demo
Outline

Introduction



Background
Research Interests
Visual Sentiment Ontology



Motivation
Design Principle
Demo
Background
1. Research Interest and Collaborators

My work is about…

Visual Big Data
Mobile Visual Search
Social Multimedia Analysis
Scene Understanding (3D/Depth/Saliency…)

Collaborators

Professor Shih-Fu Chang and Steven Feiner (Columbia University)
Professor Wen Gao (Peking University) and Hongxun Yao (Harbin Institute of Technology)
Dr. Xing Xie and Dr. Yong Rui (Microsoft Research)
Professor Qi Tian (University of Texas at San Antonio)






Outline

Introduction



Background
Research Interests
Visual Sentiment Ontology



Motivation
Design Principle
Demo
Research Interest
1. Mobile Visual Search

Mobile Visual Search and Compact Descriptor

Scenarios



Issue

Query delivery latency in mobile visual search
Solution

Directly exact and send compact visual descriptor from the mobile end

Learning to compress the original descriptor based on the mobile context
Pub.

IJCV, TIP, TMM, CVPR 12, ACM MM 11, and IJCA 11
Research Interest
1. Mobile Visual Search

Interactive Query Formulation

Active Query Sensing in Mobile Visual Search


Scenario
 Mobile Location Recognition
Point
 How the offline scene analysis can help to guide online user to take as few
query as possible to find the target location
 Pub.

ACM Multimedia 2011 Best Paper

ACM Trans. Multimedia Computing
Research Interest
2. Massive Scale Visual Search

My interests about “Visual Big Data”

Large-Scale Visual Search and Recognition

Hierarchical Vector Space Quantization Error Compensation for location search

Quantization Tree Transfer Learning across datasets


Embedding Semantics into Visual Feature Space Quantization


CVPR 2009, IEEE Multimedia 2011
CVPR 2010 (Oral), TIP
Supervised Hashing with Kernels

CVPR 2012 (Oral)
Research Interest
3. Social Multimedia Analysis

Landmark Mining from Blogs and Flickr


HITS based canonical view selection
Sparse Representation based canonical view selection

Twitter/Weibo Sentiment Analysis

Pub.

ACM Multimedia 2013 (Brave New Idea Track), ACM Multimedia 2009 (Oral)
Research Interest
4. Scene Understanding

Single Image Depth Estimation

Discriminative Scene Parsing and Depth Estimation

Joint Depth and Semantic Parsing with Structure SVM

Multi-User Semantic-Aware Mobile Augmented Reality

Visual Saliency
Pub.


CVPR 2013, CVPR 2012 (Oral)
Outline

Introduction



Background
Research Interests
Visual Sentiment Ontology



Motivation
Design Principle
Demo
Recent Work
1. Visual Sentiment Ontology

Motivation

Massive and Ever Increasing Social MultiMedia


Images (300 million photos uploaded to Facebook every day)
Videos (4 billion videos watched per month on YouTube)
Recent Work
1. Visual Sentiment Ontology

Motivation

One of the most important purpose of social media is to express
the user opinion
Recent Work
1. Visual Sentiment Ontology

Motivation

But…

Text based emotion/sentiment/opinion analysis tool is troublesome
when facing microblogs
@BarackObama: Four more years.
@Brynn4NY: Rollercoaster at sea.
Recent Work
1. Visual Sentiment Ontology

Motivation

So…


Can we reliably detect sentiments and/or emotions within images?
Web + big data + computer vision + psychology
A picture is worth one thousand words, but what
words should be used to describe the sentiments and
emotions conveyed in the increasingly popular social
multimedia?

We share a Visual



Sentiment Ontology to the community
1200-dim SentiBank Detector
Ontology Dataset
Code
http://visual-sentiment-ontology.appspot.com
Recent Work
1. Visual Sentiment Ontology

Design Principle
Question:
What is the sentiment(opinion) that the users
want to express through photo uploading
The Idea:
Design a concept dictionary to discover the
positive/negative sentiments (if any) of images

Each concept should



have a strong correlation with sentiment reflected in the image
be interpretable by human and understandable by machine
The outputs of all concepts should


ensure a good coverage of potential emotions and concepts in images
a middle-level representation to predict the sentiment of an image
Recent Work
1. Visual Sentiment Ontology

Initialization: Plutchik's Wheel of Emotion model
Recent Work
1. Visual Sentiment Ontology

Step 1

data mining to discover visual sentiments in social media

Crawl initial words by Plutchik’s “Wheel of Emotion” categories from YouTube
and Flickr
Initial concept dictionary
Crowdsourced Websites
Emotion Keyword Queries
Recent Work
1. Visual Sentiment Ontology

Step 1

data mining to discover visual sentiments in social media

Identify frequent photo tags related to emotions
Detectable nouns are not emotion-related!
Emotion-related adjectives are not detectable!
So which 1000 concepts to focus in pictures?
Recent Work
1. Visual Sentiment Ontology

Step 1

data mining to discover visual sentiments in social media


Adjective (268): needed for expressing emotions



frequent positive Adj: beautiful, amazing, cute
frequent negative Adj: sad, angry, dark
Nouns (1187): feasible for computer vision


Adjective-Noun Pair
Noun categories: people, places, animals, food, objects, weather
Standard steps


remove named entities like “hot dog” via wikipedia
Choose sentiment rich ANP concepts by tools “Senti‐WordNet”
“SentiStrength”
Recent Work
1. Visual Sentiment Ontology

Step 1

data mining to discover visual sentiments in social media

Adjective-Noun Pair
Recent Work
1. Visual Sentiment Ontology

Browser: http://visual-sentiment-ontology.appspot.com
Recent Work
1. Visual Sentiment Ontology

Browser: http://visual-sentiment-ontology.appspot.com
Recent Work
1. Visual Sentiment Ontology

Step 2

Concept Detector Training and Filtering
Visual Detectors
Performance
Filtering
Final Concept
Dictionary
Recent Work
1. Visual Sentiment Ontology

Step 2

Concept Detector Training and Filtering
 LibSVM, 5‐fold cross validation
 Features





RGB Color Histogram (3x256 dim.)
GIST descriptor (512 dim.)
Local Binary Pattern (52 dim.)
SIFT Bag‐of‐Words (1,000 codewords, 2‐layer spatial pyramid,
max pooling)
Classemes descriptor (2,659 dim.)
Recent Work
1. Visual Sentiment Ontology

Step 2

Concept Detector Training and Filtering
Good Results
Not Good Results
Recent Work
1. Visual Sentiment Ontology

Application

Live Twitter Stream Sentiment Prediction
True stuff. I have mad respect for all the
ladies that DO NOT give in to abortion
Ouch mr police man
#groundzero #hurricanesandy
#newjersey
Recent Work
1. Visual Sentiment Ontology

Application

Live Twitter Stream Sentiment Prediction


2000 tweets with images
Two-way (positive/negative) prediction
Recent Work
1. Visual Sentiment Ontology

Demo
Thank You!
imt.xmu.edu.cn/