Transcript Image Mining Presentation

Multimedia/Image Mining
Mazen AbouKhamis
Department of Computer Science and Engineering
Southern Methodist University
Outline of Presentation


Challenges Facing Image Mining
MultiMediaMiner:



Major Components
Major Functional Modules
Phases


Harvesting
Preprocessing
•







Median Filter Technique
Storing
Creating a Data Cube
Mining
Discovering
Image Retrieval
Conclusions
Future Work
2
Challenges Facing Image Mining



Representing the image objects clearly and efficiently because image
objects are hard to define. Thus, we have to break the image object
into meaningful components such as color, texture, shape, etc…
Querying the image objects after representing them to retrieve the
discovered knowledge. Below are some questions to ask...
1. how to compose an image query object? Can we use
Keywords to compose a query?
2. How to compare the query image object to the objects in
the database because unlike numeric or text data, exact
match can be rarely found between two image objects.
Visualization techniques to view the image components in a meaningful
way such as DATA CUBES.
3
MultiMediaMiner


It is a system prototype responsible for mining multimedia
information and knowledge from large multimedia
databases.
It allows the users to control, combine and manipulate
different types of Media.
4
MM - Major Components




Image excavator: responsible to extract images and videos from the
multimedia sources.
Pre-processor: responsible to extract image features and to store
pre-computed data in the database.
User interface: Used for querying.
Search engine: Responsible to match queries with the image and
video features in the database
5
MM - Major Modules

MM-Characterizer : It provides users with a multi-level view of the
data in the database with roll-up and drill-down capabilities. The figure
describes the characteristics for two dimensions: the internet domain
from which the media was extracted and the size of the media in bytes.
6
MM - Major Modules


MM-Associator : This module finds a set of association rules from
the relevant sets of data in an image database.
Example: “What are relationships among still images, the frequent
colors used in them, their size and the keyword sky. One possible
association rule would be “if image is big and is related to sky, it is
blue with possibility of 68%” or “if image is small and is related to
sky, it is dark blue with a possibility of 55%. The figure shows a
visualization of some association rules.
7
MM – Major Modules

MM-Classifier : This module classifies multimedia data based on
some provided class labels. The result is an elegant classification of
a large set of multimedia data and a characteristic description of
each class. The figure below shows an output of the classifier
module where a classification of images and frames based on their
topic, with reference to the distribution of image format, is made for
a given web site.
CS 785 Data Mining
Chapter 9 Mining Complex Types of Data
8
MM - Phases


Harvesting: Images are retrieved from the Web along with information
about the web pages in which they are found.
The World-Wide Web has been chosen to be the repository of images in
the experiments done to build MM for the following reasons:
1.
2.
3.
4.

Free
Available
Has a large collection of images
Legal issues !!!
CT scans is an interesting application for the discovery of association rules
based on colors in these scans but getting access to CT scans from
hospitals is not easy due to privacy issues.
9
MM - Phases

Pre-Processing:


Images are processed to extract the following features:
 Color
 Texture
 Size
 Length
 Width
 Duration
 Format
Web Pages are processed to extract the following features:
 Internet Domain
 Image Popularity
 Page Richness
 Keywords
10
MM - Phases

Keywords extracted from the web pages are “cleaned” and organized into
concept hierarchy.




Unnecessary words are elimimated.
Words are normalized.
WordNet is used to validate words and build the Concept Hierarchy.
 WordNet , a semantic network, in version 1.6 has 95,600 different
word forms organized into 71,000 word meanings interconnected
with links representing subsumptions. It gives relationships
between meaning of words.
 WordNet can be enriched with some specific words that are not
in its list such as : “Boeing 747” or “fighter F15”.
The built concept Hierarchy of words is used to browse images and select
image data sets for mining.
11
MM - Phases
Example:
By selecting a keyword “Boeing” from the hierarchy of the keywords, which is
on the left of this figure, all the images associated with the word “Boeing” and its
descendents are selected.
12
MM – Phases

Images also have to be preprocessed in order to remove or to reduce the
noise which occur mainly during the image capture. The Median filter
technique proves to perform well. Let’s Consider this example:
Steps:
123
125
126
130
140
122
124
126
127
135
118
120
125
134
119
115
119
123
133
111
116
110
120
130
150
1. The pixel value 150 does not represent its surroundings.
2. We have to sort neighbors of 150 in a numerical order.
If we choose to use 3*3 square neighborhood, the neighborhood
values in sorted order will be:
115, 119, 120, 123, 124, 125, 126, 127, and 150.
3. Calculate the median and replace 150 by the median value.
13
MM - Phases

The figures of the coins below shows the image of coins before
and after applying the median filter to it.
Before
After
14
MM - Phases

Storing: Information or Meta-data extracted from the images and the
web pages containing these images are stored in the database.

The image is not stored by itself in the database. Only its feature
descriptors are stored in the database.

Since the World-Wide Web is the source of images and because it has
dynamic structure ; i.e. some new images may appear and some images
may disappear. Thus, the descriptors of those images who disappear are
discarded from the database.
15
MM - Phases



Creating a Data Cube: A Multimedia data cube is created based on
different dimensions (features stored in
database).
In reality it is impossible for the physical data cube to hold more than a
given number of dimensions because the size of the cube grows
exponentially with the number of dimensions. That is, each time a
dimension is added, the size of the cube is multiplied by the number of
distinct values in the new dimension.
For example, the number of dimensions of color by itself = 256 and there
are other features having large number of dimensions. So, how the data
cube addresses this problem of dimensionality? Compromise is the solution.
16
MM - Phases
The third column in the table below represents the compromise.
17
MM - Phases

The number of dimensions is still too large to be handled by a data
cube. So, many data cubes needed in the implementation of MM-Miner:




1st cube holds the color and the texture dimensions.
2nd cube holds mainly the size, length, and width dimensions.
3rd cube holds the dimensions of the remaining features.
It is impossible to discover the correlation, if it exists, between different
dimensions in different cubes such as the color and the size because
data mining algorithms work at one cube at a time. What is the
solution? Overlap. To have an overlap:


4th cube needed having dimensions from the above 3 cubes.
The internet domain and the size dimensions have to exist in all the
4 cubes.
18
MM - Processes



Mining: Data Mining algorithms and OLAP technology are used to discover
implicit knowledge like classification of images based on their multimedia
features, correlation between multimedia features, etc…
Discovering: Classification, Association, and Summarization of image data is
discovered. Moreover, Slicing, Dicing, Drilling-down and Rolling-up allow
identifying specific images by clusters.
Image Retrieval: After the data mining process filters out the interesting
images, these images and the web pages containing them can be retrieved
now.
19
MM – Phases (Summary)
Media
Descriptors
WWW
Discoveries
Database
Mining Engine
Data Cube
Dimensions
20
Conclusions



The user interface of the 3 major modules of MultiMediaMiner
allows interactive mining.
Tradeoff exists to implement the data Cube.
 Keywords are not represented in the data cube.
 Choose only the most frequent values of some features
such as color, texture, etc… in order to reduce the
number of dimensions.
 Designing many data cubes instead of just one data
cube.
Noise must be removed before storing the descriptors of the
image in the database.
21
Future Work



A new model for data cube materialization is under study.
Adding a clustering module which would group images into different
clusters based on their features.
Enhancing the system to work not only with images and videos but with
audio data as well.
22
THE END
23