Transcript week8

Image Retrieval Part I
(Introduction)
Image Understanding Functions
• Image indexing
• similarity matching
• image retrieval (content-based method)
2
Images containing similar colors
3
Images containing similar content
4
Image Database
5
Image Database
6
Variety of Similarity
Degree of difficulty
7
• Similar color distribution
Histogram matching
• Similar texture pattern
Texture analysis
• Similar shape/pattern
Image Segmentation,
Pattern recognition
• Similar real content
Life-time goal :-)
Image Indexing
8
9
Query by Example I
10
Query by Example 2
“Find all paintings of
this shape”
11
Retrieval by Sketch
PictureFider
search engine
12
QBIC - Query By Image Content
First and best known commercial image retrieval system
• IBM Almaden Research Center
• Commercial product
Search by shape, color, texture, keyword
Queries can be based on:
• example images,
• user-constructed sketches and drawings,
• selected color and texture patterns
User can:
• Select colors and color distributions from a color wheel,
• Select textures from a predetermined selection,
• Adjust relative weights among shape features
13
Content-based image retrieval
• simple features are not semantic entities
• as opposed to words in a text
• not easy to extract semantic content
• choice of indexing elements is unclear
• low-level features
• color, texture, shapes
• high-level features
• a person on bicycle, a windsurfer, vase
• abstract features
• triumph, sadness, happiness
14
15
Content of Image
• Image = array of pixel intensities
•
•
•
•
16
b/w: 1 bit
gray: 8 bits
colour: 24 bits
code: index in colour lookup table
Image Retrieval by Intensity
• pair-wise comparison
• histogram methods
• central moments
17
Pair-wise comparison
• feature vector = raw array of pixel intensities
• needs model of pixel intensity distance dC
• compared pictured must be same size
• considers spatial arrangement
• not concise
18
Histogram methods
• quantise intensity space into N bins H(i) is pixel proportion
with intensity of bin i
• distance of images I and Q (query example):
histogram intersection
• simple, concise method
• global feature: locality is lost
• Exercise: why must the histograms be normalized?
19
Histogram Encoding
20
Center Image Histogram
21
Window Histogram Encoding
22
Window + center encoding
23
Intensity central moments
• instead of a histogram compute three statistical numbers
per image (or window)
• mean (average gray level)
• variance (squared standard deviation)
• 3rd central moment (skewness of the grey level distribution)
24
Intensity central moments
• mean, variance and 3rd central moment:
25
Intensity central moments
• in order to have numbers which are comparable in size for
each moment we use the corresponding root
• we get a 3-dim feature vector:
26
Moments + spatial coherence
• moments are a global feature
• spatial coherence can be added by
• windows
• centering
• windows+centering
27
summary (retrieval by intensities)
• paiwise comparison
• histograms
• central moments
• balance in locality through
• windows, centering, windows+centering
28
Retrieval by color
• RGB color model
• color histograms
• color moments
• clustering in color space
29
The RGB color model
• mixture of intensities of red, green and blue
• for active devices such as monitors
• some examples
30
color histograms
• simple methods: three 1-d histograms; one 3d histogram
• create a 1d histogram for each color component each: eg, one
for R, one for G, one for B
R
G
B
• divide 3d color space into 3d bins for histograms:
31
Color moments
• feature vector:
9 numbers per image or window
• 3 moments per color channel:
32
Clustering in color space
• cluster the pixels in 3d color space
• features: centroids and population of clusters
• similarity measure
• compares color centriods and population
• must take different cluster number into account
33
Summary (retrieval by color)
• RGB color model (or HSV color model)
• two color histograms
• moments of color
• clustering in color space
34
Texture
• perceptual phenomenon
• local region quality, not point quality
• depends on the scale
• texture = repeated pattern
• local variations in image intensity
• too fine as to be considered as own objects
• Texture feature
• Co-occurence matrix
• Gabor Wavlet Filter coeffiecients
35
Texture Examples
36
Segmentation of Images
• texture analysis needs segmentation of the still image to be
meaningful
• separate objects
• ignore background
• assign texture feature per object
• object segmentation is a difficult task; edge detection can be
useful for this
37
Texture+Color Segmentation
38
Shape Analysis
• Shape=
class of geometric objects invariant under
• information preserving description (for compression)
• non-information preserving (for retrieval)
• boundary based (ignore interior)
• region based (boundary+interior)
• Shape features
• Chaincodes (boundary based)
• Fourier Descriptor (boundary based)
• ect
39
Examples of shapes
40
Chaincodes
• contour description by a string of directions
• Freeman chain code
• f1f2f3...
• eight possible directions for each pixel
• translation invariant, not rotation invariant
41
Chain code histograms
• used for distance measure
• same histogram represents many shapes
• percentage histogram is scale invariant
42
Retrieval by shape
43
Retrieval by shape
44
Content-Based Image Retrieval
• many different low-level features can be computed, why did
we do this?
• to describe a complex multimedia object
• to automatically annotate mm objects with salient properties
(eg, color shape, texture)
• to be able to compute similarities of mm objects
• to make mm objects searchable by content
• high-level feature are not (yet) accessible, but research
attempts to construct high-level concepts from low-level
features
45
Content-Based Image Retrieval
• Which features are best for searching? Depends on the
information need:
• looking for sunset holiday picture in your digital shoebox? Use
color histograms
• want to build a wallpaper customer database? Use
color+texture
• want to build a b/w sketch database for technical industrial
designs? Use shape descriptors
• Not sure which features are best for a query (eg, if you also
have abstract features such as Fourier coefficients)? Deploy
relevance feedback and let the system learn the relevant
feature for this query...
46
Content-based Image Retrieval
• Summary: we looked at
•
•
•
•
•
retrieval by intensity
color models
retrieval by color
retrieval by shape
retrieval by texture
• these features can be used in a general framework for
content-based retrieval as discussed earlier
47
Experiment: Setting up CBIR search engine
• Create directory c:/Experiments/database
• copy database file “texturedatabase” into the directory
• copy the following two subdirectory into
c:/Experiments/database
• ImageDatabase
• ShapeColor
• Open database (“java COM.cloudscape.tools.cview”)
• install the database “texturedatabase”
• Close database
48
Compile file.Java & deploy application
• open cmd and type the following command
• javac *.java
• run J2ee server
• run deploytool
• deploy your application to include the following Web
Components:
•
•
•
•
49
index.jsp
MyLocalesRbf2
MyDateJose2
and all the jpeg images stored in the directory called
“ImageDatabase”.
Summary of Fundamental CBIR
• Content-based image retrieval (CBIR)
• CBIR Algorithm:
• First step --- Image Indexing
• Second step --- content matching
50