Transcript Semantic Models for Multimedia Database Searching and Browsing

Multimedia Databases
Prepared by Chengcui Zhang
Lab: KDDM www.cis.uab.edu/kddm
Email: [email protected]
www.cis.uab.edu/zhang
2010 Spring
1
Trends in Internet, Mobile Phones,
Mobile Internet




Smart phones!
40 million of these mobile phone users in
Europe are mobile multimedia users.
The total Western European mobile market
is worth 120 billion ECU per year in 2010.
The mobile multimedia segment of this
Western European market are worth 30
billion ECU in 2010.
8
Introduction

Multimedia system:


A variety of information sources (text, voice, image, video,
audio, animation, etc.)
Characteristics:


Requirements:



All the different media are brought together into one single
unit, all controlled by a computer
Management and delivery of extremely large bodies of data
at a very high rate
Real-time constraints …
Challenges:


Synchronization…
Semantic heterogeneity
9
10
Problems of Relational
Database Model


Conventional data modeling techniques lack the ability to
manage the composition of multimedia objects in a
heterogeneous multimedia database environment.
Relational database system is only good to manage textual
and numerical data.




Retrieving data is often based on simple comparisons of text or
numerical values.
Relational data model has limited capabilities in modeling the
structural and behavioral properties of real-world objects.
Relational data model has difficulty to model time-dependent
multimedia data (video or audio).
BLOBs (Binary Large Objects) are incapable of interactively
accessing various portions of objects since a BLOB is treated
as a single entity in its entirety.
11
Problems of Object-Oriented
Model


It provides a better facility for managing the
multimedia data.
Good features:




Inheritance
Information hiding
Can include image data
Composite object (an object consisting of other
objects) provides the capability to handle the
structural complexity of the data
12
Problems of Object-Oriented
Model (cont.)



Lack of facilities for the management of
spatio-temporal relations.
Still, the O-O DBMS is not designed to
support multimedia information
management.
Multimedia extension is needed to handle
the mismatch between multimedia data and
conventional O-O database management
systems.
13
Important Characteristics of
Multimedia Objects (MO)






MO are complex and therefore less than
completely captured in an MDBMS.
MO are audiovisual in nature and are
amenable to multiple interpretations.
MO are content sensitive.
Queries looking for MO are likely to use
subjective descriptions that are often fuzzy
in their interpretation.
MO may be included in fuzzy classes.
…
14
15
Requirements for Modeling
Multimedia Data
1.
2.
3.
4.
5.
Specify incomplete information
Extend the definition of some individual
documents beyond the definitions of its type
Integrate data from various databases and
handle them uniformly
Describe structural information
Distinguish between internal modeling and
external presentation of objects
16
Requirements for Modeling
Multimedia Data (cont.)
6.
7.
8.
9.
Share data among multiple
documents
Create and control versions
Include appropriate operations
Handle document access control
17
New trend: SMELL!
18
http://staff.science.uva.nl/~gevers/master2007/PDF/lecture1_small_2007.pdf
Multimedia Database
Applications







Education: CAI (Computer Assisted
Instruction)
Internet search (e.g., Google
image/video search)
Medical Imaging
Surveillance Systems
Biometrics databases
Video-on-demand
Game …
20
Application: Image search
engines – Goggle!
http://www.google.com/mobile/goggles/#landmark
22
Application: Fingerprint
Matching and retrieval
23
Application: real-time skin
detection for human recognition

Are HP computer webcams really racist?

http://blogs.consumerreports.org/electronics/2009/12
/racist-hp-webcam-video-blog-consumer-reportsresponse.html
24
25
Application: real-time object
recognition and tracking
26
Application: Surveillance
http://www.nydailynews.com/ny_local/2010/01/08/2010-0108_new_jersey_man_arrested_over_security_breach_at_newark_liberty_airpo
27
rt.html
Content-Based Image
Retrieval

An picture is worth a thousand words!
28
Text-based
Retrieval
29
Content-Based Image
Retrieval

Content-Based Image Retrieval (CBIR)



Image databases can be huge, containing
hundreds of thousands or millions of images.
In most cases they are only indexed by
keywords that have to be decided upon and
entered into the database system by a human
categorizer.
However, image can be retrieved according to
their content, where content might refer to color
distributions, texture, region shapes, or object
classification.
Image Database Examples


IBM: Query by Image Content (QBIC)

Retrieves images based on visual content, including such
properties as color percentage, color layout, and texture.
Virage, Inc.

Virage search engine can retrieve images based on color
composition, texture, and structure.

Google Image search.

National Library of Medicine provides a database of x-rays,
CT scans, MRI images, and color cross-sections, taken at
very small intervals along the bodies of male and female
cadaver.
The NASA collects huge databases of images from its
satellites and makes them available for public acquisition.
(for free )

State-of-the-Art in MDBMS




First wave – query by text
In a second wave, commercial systems
were proposed which handle multimedia
content by providing complex object
types for various kinds of media.
Broadly used commercial MMDBMSs are
extensible Object-Relational DBMS
(ORDBMSs).
Oracle 10g, IBM DB2, and IBM Informix.
34
DB2 Image Extender

DB2 Image Extender defines the distinct
data type DB2IMAGE with associated
user-defined functions for storing and
manipulating image files


(http://www306.ibm.com/software/data/db2/extenders/ ).
The DB2 Image Extender provides
similarity search functionality based on
the QBIC technology

(http://wwwqbic.almaden.ibm.com/ )
35
Query By Example (QBE)

The image DB user should be able to:




show the system a sample image, or
Paint one interactively on the screen, or
Just sketch the outline of an object.
The system should then be able to
return similar images or images
containing similar objects.
36
IBM-QBIC

The Hermitage Web site was voted the
best in Russia. It uses the QBIC
engine for searching archives of worldfamous art.



http://www.hermitagemuseum.org/fcgibin/db2www/qbicSearch.mac/qbic?selLa
ng=English
Color percentage
Color layout
37
A sample query

SELECT CONTENTS(image),
QBScoreFROMStr(`averageColor=
<255,0,0>’, image) AS SCORE
FROM signs ORDER BY SCORE
38
Photobook System
Figure 1. The texture retrieval of PhotoBook system
(http://web.media.mit.edu/~tpminka/photobook/).
Search Using Sketch
Sketch entry

Results of search

ImageScape System
Figure 2.5 The interface of ImgeScape visual query
system (http://skynet.liacs.nl/imagescape/).
Relevance Feedback in CBIR

Motivation:


Human perception of image similarity is
subjective, semantic, and task-dependent.
The CBIR based on the similarities of pure
visual features are not necessarily perceptually
and semantically meaningful.


Each type of visual feature tends to capture only one
aspect of image property and it is usually hard for a
user to specify clearly how different aspects are
combined …
Relevance Feedback is introduced to address
these problems.

It is possible to establish the link between high-level
concepts and low-level features.
Relevance Feedback RF
(cont.)

RF is a supervised active learning
technique used to improve the
effectiveness of information systems.

Main idea: use positive and negative
examples from the user to improve
system performance.
Initial query
results
Query Image
Collect user’s
feedback
Real-time
learning
Refine query
results
Initial Query Results
User Relevance Feedback
Query Results After User Feedback
Training System Interface
Object-based Image Retrieval

Object-based CBIR:

Motivation
1.
The basic unit of user interests usually is individual
objects.
2.
Images are segmented into homogeneous regions,
and the image features are extracted for each region.
3.
Image similarity is then measured in term of region
similarity.
Spatial Indexing
Object-Based Image Retrieval with
Relevance Feedback

Techniques used:




Image segmentation
Neural network
Multiple instance
learning
…