Transcript Multimedia_Search_and_Retrieval

Multimedia Search and Retrieval:
New Concepts, System
Implementation, and Application
Qian Huang, Atul Puri, Zhu Liu
IEEE TRANSACTION ON CIRCUITS AND SYSTEMS FOR
VIDEO TECHNOLOGY, AUGUEST 2000
Outline







Introduction
Media Types and Their Properties
Multimedia Data Abstraction and Retrieval
System Architecture
System Implementation
An Application: Multimedia Broadcast News
Summary
Introduction



MPEG-7 focuses on the standardization of a
common interface for describing multimedia
materials,supporting pull and push information.
Our proposed multimedia information retrieval
system cover system architecture, data storage
and representation, database and its remote
access, and human-machine interfaces.
The concept of physical and logical layers and
events.
Media Types and Their
Properties

Audio



Audio events can be defined as continuous
audio segments that have some distinct
acoustic characteristics
Visual
Text
Multimedia Data Abstraction
and Retrieval


A physical layer addresses the media
events that are time continuous and
better described using the specific
characteristics of a media signal.
A logical layer addresses multimedia
events that present integrated behavior
of other events across media.
Multimedia Data Abstraction
and Retrieval(cont.)

Physical Program and Physical Event

Physical program


a continuous stream of multimedia data that
begins and ends due to the physical scene edits
and cuts decided by human operators
Physical event

A continuous segment of multimedia data,
representing a coherent and meaningful event
within a physical program
Multimedia Data Abstraction
and Retrieval(cont.)

Logical Event



A virtual view of the multimedia data
constructed from a set of physical programs or
events, or recursively from other physical or
logical events.
Application dependent
Semantic Content Structure



Conceptual and virtual
Provide a seamless interface to user
Table of Content (ToC)
Relationship among different event layers
System Architecture
System Implementation

Server Implementation


Content Hierarchy Extraction
Layered Content Storage



Store lower level information in a file system and
higher level information in standard database systems
When raw data is stored in a file system, different
events in the data stream can be accessed using time
based indexing
Database Content Access

Employ JDBC to provide connectivity to our database
for query and update
Multimedia data storage in combined file system and database
Example of event indexing from database to a file
Database access
System Implementation(cont.)

Client Browser Implementation

Java applet-based interface



Remote Database Query


RMI (Remote Method Invocation)
Browsing Interface




Platform independence
support for a variety of coded media
Table of Content: a conceptual content structure
TimeLine: a content structure in time
Event Presentation: presentation and playback
Content Playback with a JMF Player
Remote database access
Browser interface for a news program with both ToC and timeline
representations of the semantic structure within the program
An Application: Multimedia
Broadcast News

Server End

Content Hierarchy Extraction



Broadcast news data is well structured.
Lower levels – extract primitive physical events
in each individual media stream.
Higher semantic level – merge different events
into semantically meaningful logical events.
ex: Headline stories and news summaries.
Content hierarchy of broadcast news programs

Content Storage and Access



In the file system, for each physical broadcast
news program, store the timetables for audio,
video,and text,individually.
The tables in the database system will index
into these timetables for the actual retrieval
and data access.
Physical events in each media are recorded in
separate database tables at the physical event
layer.
Relationship between logical event and physical
event objects
Database design in broadcast news application
Client End

Remote Database Query


The physical program ID is uniquely determined by
the channel and the date of the program.
Content Presentation

Content Structure Presentation





ToC and TimeLine
Static(single click) and playback(double click)
Story Presentation
News Summary Presentation
Streaming Playback
Browser interface for a news program with both ToC and timeline
representations of the semantic structure within the program
Visual representation for story “El Nino”
Representation for news summary of the day
Playback interface with synchronized
audio/video/text
Summary


We have introduced a new layered
approach to multimedia content
representation and storage for search and
retrieval.
The concepts are implemented in a
client/server multimedia information
retrieval system designed for broadcast
news search and retrieval.