Transcript ZhiBin

Content-based Image Retrieval
Zhibin Huang
CSCI 8810
Three requirements



Given a consultant image, find similar
images in an image database;
Find images of flowers;
Find images of babies with smiles in
their faces;
Outline





History of image retrieval
Content-based image retrieval
Feature Extraction
Demo
Q&A
History of Image Retrieval

Traditional text-based image search engines


Manual annotation of images
Use text-based retrieval methods
Water lilies

E.g.
Flowers in a pond
<Its biological name>
Limitations of text-based approach

Problem of image annotation



Problem of human perception



Large volumes of databases
Valid only for one language – with image retrieval this
limitation should not exist
Subjectivity of human perception
Too much responsibility on the end-user
Problem of deeper (abstract) needs

Queries that cannot be described at all, but tap into
the visual features of images.
Outline





History of image retrieval
Content-based image retrieval
Feature Extraction
Demo
Q&A
MileStone

In 1992, the National Science Foundation of the
United States organized a workshop on visual
information management systems to identify new
directions in image database management systems.

A more efficient and intuitive way to represent and
index visual information would be based on properties
that are inherent in the images themselves.
What is CBIR?


Images have rich content.
This content can be extracted as various
content features:



Mean color , Color Histogram etc…
Take the responsibility of forming the
query away from the user.
Each image will now be described by its
own features.
CBIR - A sample search query

User wants to search for, say, many rose images





He submits an existing rose picture as query.
He submits his own sketch of rose as query.
The system will extract image features for this
query.
It will compare these features with that of other
images in a database.
Relevant results will be displayed to the user.
CBIR System
Example
Query Image
Retrieved Images
Image Database
Similarity Assessment
Feature Space
Outline





History of image retrieval
Content-based image retrieval
Feature Extraction
Demo
Q&A
Feature Extraction

Image Content Descriptor

Visual content


color, texture, shape, spatial relation
Semantic content

Can be obtained either by textual annotation or
by complex inference procedures based on
visual content.
Color

Color is the most extensively used
visual content for image retrieval.



Color Space
Color Moment
Color Histogram
Color Space

HSV space is widely used in computer
graphics. The three color components
are hue, saturation (lightness) and
value (brightness).

The hue is invariant to the changes in
illumination and camera direction and
hence more suited to object retrieval.
Color Moments

Color moments have been proved to be
efficient and effective in representing color
distributions of images
1
E

p
 First order(mean)

N
N
i


j 1
ij
Second order(variance)
1
 i  
N

( pij  Ei ) 

j 1

1
2
Third order(skewness)
1
si  
N

( pij  Ei ) 

j 1

1
3
N
2
N
3
Color Histogram


The color histogram is easy to compute
and effective in characterizing both the
global and local distribution of colors in an
image.
Robust to translation and rotation about
the view axis and changes only slowly with
the scale, occlusion and viewing angle.
Example - Search by color
Images courtesy : Yong Rao
Texture

Structural methods

Describe texture by identifying structural primitives and
their placement rules.




morphological operator
adjacency graph
Effective when applied to textures that are very regular.
Statistical methods

Statistical distribution of the image intensity



Markov random field
Gabor and wavelet transform
Used frequently and have proved to be effective in
content-based image retrieval
Example - Search by texture
Images courtesy : Ming Zhao
Shape (feature)



Compared with color and texture features, shape
features are usually described after images have
been segmented into regions or objects.
The use of shape features for image retrieval has
been limited to special applications where objects
or regions are readily available.
The state-of-art methods for shape description can
be categorized into either boundary-based
(Fourier-based shape descriptors) or region-based
methods (statistical moments).
Example - Search by shape
Images courtesy : Yong Rao
Spatial Information

spatial location of regions (or objects)
or the spatial relationship between
multiple regions (or objects) in an
image is very useful for searching
images.
Example - Query by sketch
Images courtesy : Yong Rao
Outline





History of image retrieval
Content-based image retrieval
Feature Extraction
Demo
Q&A
Demo

QBIC(TM) - IBM's Query By Image Content


Perceptual Shape-Based Image Retrieval


http://www.hermitagemuseum.org/fcgi-bin/db2www/qbicSearch.mac/qbic?selLang=English
http://torch.cs.dal.ca/~xzheng/ipami/
SIMBA – Search IMages By Appearance

http://simba.informatik.uni-freiburg.de/
Open source project

Lire


An Open Source Java Content Based Image
Retrieval Library
http://www.semanticmetadata.net/lire/
Outline





History of image retrieval
Content-based image retrieval
Feature Extraction
Demo
Q&A
Q&A
Thank you !
Questions ?