Transcript VICTORY by Petros Daras

Information Society
Technologies
Dr. Petros Daras,
Informatics & Telematics Institute
NAVS Spring 2008 Concertation Meeting - CERTH/ITI
1
VICTORY Overview
• VICTORY is a 30-month STREP
• Start: Jan 2007
• End: Jun 2009
• 9 partners
• CERTH/Informatics & Telematics Institute
• University of Ljubljana
• Politecnico di Torino
• Alcatel – Lucent Deutschland AG
• TELETEL S.A.
• HYPERTECH S.A.
• EMPOLIS GmbH
• TWT GmbH
• LivingSolids GmbH
NAVS Spring 2008 Concertation Meeting - CERTH/ITI
Greece
Slovenia
Italy
Germany
Greece
Greece
Germany
Germany
Germany
2
VICTORY Objectives
 To develop an innovative, distributed search engine that will introduce
MultiPedia search and retrieval capabilities to a standard (PC-based) and
a mobile P2P network. The 3D search engine will be based on:
•
•
content, which will be extracted taking into account low-level geometric
characteristics and
context, which will be high-level features (semantic concepts) mapped
to low-level features.
 To bridge the gap between low and high-level information through
automated knowledge discovery and extraction mechanisms.
•
•
Annotation mechanisms
Relevance feedback
NAVS Spring 2008 Concertation Meeting - CERTH/ITI
3
VICTORY Objectives
 To achieve delivery of audio-visual content on low-power mobile
terminals as to enable their integration in a PC network, where the
missing resources will be provided collaboratively by other participants.
 To develop a multimodal personalized user interface. Different
MultiPedia search and retrieval modalities will be supported:
• Text (annotation)
• 2D images (taken by the user's mobile device)
• Sketches (made by the user)
• 3D objects
 To ensure that the shared MultiPedia content will not be distributed
uncontrollably against the owner's will and to protect the intellectual
property rights of the content owners.
•
•
DRM (Digital Right Management)
3D Object Watermarking
NAVS Spring 2008 Concertation Meeting - CERTH/ITI
4
Achievements so far
System Architecture
P2P Network
3D search & retrieval algorithms
Relevance feedback algorithms
Annotation propagation framework
NAVS Spring 2008 Concertation Meeting - CERTH/ITI
5
The overall VICTORY
architecture
NAVS Spring 2008 Concertation Meeting - CERTH/ITI
6
Super peer
module (broker)
Superpeers are alwayson, always connected machines, with a static IP
addresses associated.
Super-peers are trustworthy machines for the point of view of security.
A super-peer manages a set of client peers that are “directly connected” to it.
Each superpeer knows all the services and contents available on the
controlled clients.
Each super peer maintains and updates a certain number of metadata
catalogues describing clients directly connected to it. Moreover, a table
containing IP addresses of other super-peers directly connected is stored
on each superpeer.
Super-peers play a basic role in processing user queries in order to: search for
Multipedia Objects, negotiate resources (QoE), and manage VICTORY's
services.
They also serve as relays thus enabling the clients that are ‘hidden’ behind
NATs and firewalls to successfully traverse them and establish connections
to other client peers.
NAVS Spring 2008 Concertation Meeting - CERTH/ITI
7
Edge-peers
Each edge-peer may be occasionally off or disconnected from the network;
When it reconnects to the network, its IP address may not be the same as
before.
When an edge-peer gets connected to the network, it should register (or create
a “connection”) with one and only one superpeer,
It should provide it with all the information regarding the contents and services
available, plus other general information.
The edge-peer may have a list of more than one superpeer to which connect,
in order to tolerate superpeer failures.
NAVS Spring 2008 Concertation Meeting - CERTH/ITI
8
P2P networking
VICTORY island groups
users that share common
interest (like automotive
group or tourist group)
VICTORY island may
provide some specific
services (collaboration),
while other services (search
and download) are common
to whole VICTORY
community
NAVS Spring 2008 Concertation Meeting - CERTH/ITI
Victory island
Victory island
Super peer
Super peer
Victory island
Super peer
Client peer
Client peer
Client peer
9
Search architecture
Offline
3D object
Repository
Browsing / Selecting
existing 3D object
Submitting new
3D object
Low-level feature
extraction
Geometric
Descriptors
Spherical Harmonics
Ellipsoidal Harmonics
Medial Surface
Feature
… Retrieval
Feature
Matching
Results
Low-level feature
extraction
Online
NAVS Spring 2008 Concertation Meeting - CERTH/ITI
10
Low-level feature
extraction Module
• Input:
3D Object
• Output: Feature vector with geometrical
descriptors
• Goals: Discriminative power
Robustness
Invariance to geometric transformations
• Formats: VRML, OFF, OBJ, 3DS, X3D
NAVS Spring 2008 Concertation Meeting - CERTH/ITI
11
Pre-processing
Sub-module
Format-specific
Parser
Input 3D object
(VRML, OFF, 3DS,
OBJ, X3D)
Uniform 3D Mesh
Represetation
(points, triangles)
Mesh-to-Volume
Transformer
Voxel Model
NAVS Spring 2008 Concertation Meeting - CERTH/ITI
12
Mesh-to-Volume
Transformer
(Voxelization)
3D model
Voxel model
Binary Volume
Function
- Creation of the bounding cube, which is partitioned in equal
cubed shape voxels
- Creation of the Binary volume function f (x)
NAVS Spring 2008 Concertation Meeting - CERTH/ITI
13
Preprocessing
• Translation
• Scaling
• Rotation (PCA)
NAVS Spring 2008 Concertation Meeting - CERTH/ITI
14
Descriptor Extraction
Sub-module
Low-level Feature Extraction
Algorithms
Generalized 3D Radon Transforms
– 3D Radon Transform
– RIT
– EnRIT
– SIT
Voxel Model
Spherical Trace Transform
– "Mutated" RIT
– 1D Fourier Transform
– Radon Transform
– Polar-Fourier Transform
– Hu Moments
– Zernike Moments
– Krawtchouk Moments
– Spherical Harmonics
Weighted 3D Krawtchouk
Moments
NAVS Spring 2008 Concertation Meeting - CERTH/ITI
Geometric Descriptor
Vectors
3D Radon – based
Descriptors
Spherical Trace –
based Descriptors
3D Krawtchouk –
based Descriptors
15
Combining Topology
& Geometry (1)
• The model is segmented into “meaningful parts”
based on the segmentation of medial surface
• A meaningful part is a component that can be
perceptually distinguished from the remaining
object.
NAVS Spring 2008 Concertation Meeting - CERTH/ITI
16
Combining Topology
& Geometry (2)
•
•
•
•
•
Every part is approximated with a Super Quadratic surface and described
using the novel 3D Distance Field Descriptor
3D Distance Field Descriptor (3D DFD) is a measure of the difference
between the surface of the quadratic and the surface of the part.
An attributed graph is formed, where the attributes are the Super Quadric
parameters and the 3D DFD
The matching is based on state-of-art Approximate Attributed Graph
Matching.
The matching approach is capable of partial matching
NAVS Spring 2008 Concertation Meeting - CERTH/ITI
17
Ellipsoidal Harmonics
• The 3D object is described using
Ellipsoidal Harmonics.
• Ellipsoidal Harmonics are solutions of
Laplace’s equations in ellipsoidal
coordinates
• Ellipsoidal Harmonics offer:
– Compact 3D object description
– Better 3D object approximation
NAVS Spring 2008 Concertation Meeting - CERTH/ITI
18
Impact Descriptor(1)
• The key idea of Impact Descriptor is the indirect description of
the 3D object’s geometry, by computing a descriptor that
describes the impact of the 3D object in the surrounding space.
• Every object is treated as a distributed mass and the
gravitational impact is described
– Assuming that the time-space is following the rules of Euclidean
geometry, generalizations of Newton’s Laws are utilized in order to
compute the Potential and the Density of the surrounding gravitational
field
– Assuming that the time-space is following the rules of Riemannian
geometry, Einstein’s General Relativity Laws are adopted in order to
estimate the curvature of the surrounding time-space (or, how the 3D
object curves the surrounding time-space).
NAVS Spring 2008 Concertation Meeting - CERTH/ITI
19
Impact Descriptor(2)
• Histograms of the Riemannian curvature and
Newtonian fields are forming the descriptors
NAVS Spring 2008 Concertation Meeting - CERTH/ITI
20
Watermarking
• A watermarking approach for copyright protection based
on spheroidal harmonic analysis
• The object surface is segmented into patches.
• Patches that are appropriate for watermarking are
selected
• Every selected patch is mapped on a spheroid
• Spheroidal harmonic coefficients are appropriately
modified (based on the watermark sequence)
NAVS Spring 2008 Concertation Meeting - CERTH/ITI
21
Relevance Feedback
Objective:
• To refine the retrieved results
• To give a “human-centric” approach to the
system, since it will be based on the user’s
individual actions and choices.
Methods:
•
Semantic Force Relevance Feedback
Coulomb-like forces are applied between query
and the objects
•
Ranked-based Relevance Feedback
Rank lists are calculated for all objects
NAVS Spring 2008 Concertation Meeting - CERTH/ITI
Relevance Feedback Steps
Step 1: The user defines a query
object
Step 2: The system compares the
query with all database objects
Step 3: The system returns the
most similar objects to the query
Step 4: The user marks the
degree of relevance of the
retrieved results
Step 5: The user feedback is used
to train the system
22
Relevance Feedback Results
NAVS Spring 2008 Concertation Meeting - CERTH/ITI
23
Annotation and annotation
propagation mechanisms
Objective:
• To model and develop a mechanism for the manual
annotation of 3D objects.
• To develop a method for the automatic annotation
propagation to non-annotated 3D objects.
Method:
•
Geometry exploitation
•
Fuzzy logic, neural networks, active learning, relevance feedback
•
Every object i maintains 2 vectors:
– Geometric features vector Gi
– Probabilities vector Si = <pi1, … , piK>
pij : Probability for object Oi to have attribute Aj
K : Number of Attributes
NAVS Spring 2008 Concertation Meeting - CERTH/ITI
24
Annotation Propagation
Step 1: Object that provides
maximum knowledge gain is
selected for manual annotation
Step 2: User annotates the object
Geometric Features
Probabilities
Input
Geometric
characteristics of i
Knowledge Gain Estimator
Step 3: Probability vector of object
is updated
Step 4: User marks similar and
non-similar objects
Step 5: Feature vector of marked
objects is updated
Step 6: The annotation is used to
train the neural network
Step 7: Probability vectors of all
non-annotated vectors are
updated
Relevance Feedback
Annotator
NeuroFuzzy Controller
Output
Probability for i to
have specific
attributes
NAVS Spring 2008 Concertation Meeting - CERTH/ITI

Expected output
User annotation
(Probability 
{0,1})
25
Spherical Harmonics
Spherical Trace
Annotation Propagation
Results
Automatic Annotation on ITI DB
NAVS Spring 2008 Concertation Meeting - CERTH/ITI
Automatic Annotation on SHREC DB
26
Access
Over the Project Website (http://www.victory-eu.org) / Results / Victory Search
NAVS Spring 2008 Concertation Meeting - CERTH/ITI
27
Publications
[1] A.Mademlis, P.Daras, D.Tzovaras and M.G.Strintzis: "3D Volume Watermarking Using 3D
Krawtchouk Moments", 2nd International Conference on Computer Vision Theory and Applications
(VISAPP 2007), Barcelona, Spain, March 2007.
[2] G. Kordelas and P. Daras: "Recognizing 3D Objects Using Ray-Triangle Intersection Distances",
IEEE International Conference on Image Processing (ICIP 2007), San Antonio, Texas, USA,
September 2007.
[3] A.Mademlis, P.Daras, D.Tzovaras and M.G.Strintzis: "On 3D Partial Matching of Meaningful Parts",
IEEE International Conference on Image Processing (ICIP 2007), San Antonio, Texas, USA,
September 2007.
[4] D.Zarpalas, P.Daras, A.Axenopoulos, D.Tzovaras, and M.G.Strintzis:"3D Model Search and
Retrieval Using the Spherical Trace Transform", EURASIP Journal on Advances in Signal
Processing Volume 2007.
[5] A.Mademlis, P.Daras, A.Axenopoulos, D.Tzovaras, and M.G.Strintzis :"Combining Topological and
Geometrical Features for Global and Partial 3D Shape Retrieval", IEEE Transactions on
Multimedia, Accepted for Publication
[6] E.Onasoglou and P.Daras: "Semantic Force Relevance Feedback, Content-Free 3D Object
Retrieval and Annotation Propagation: Bridging the Gap and Beyond", SPRINGER Multimedia
Tools and Applications Journal (MTAP), Special Issue on Multimedia Semantics, Accepted for
Publication
[7] A.Mademlis, P.Daras, D.Tzovaras and M.G.Strintzis: "3D Object Retrieval based on Resulting
Fields", 29th International conference on EUROGRAPHICS 2008, workshop on 3D object
retrieval, 15 April 2008, Crete, Greece.
[8] M.Lazaridis and P.Daras: "A Neurofuzzy Approach to Active Learning based Annotation
Propagation for 3D Object Databases", 29th International conference on EUROGRAPHICS 2008,
workshop on 3D object retrieval, 15 April 2008, Crete, Greece
NAVS Spring 2008 Concertation Meeting - CERTH/ITI
28
Under review
[1] J. Constantinides, A.Mademlis, P.Daras, and M.G.Strintzis: “Blind Robust 3D
Mesh Watermarking based on Oblate Spheroidal Harmonics", IEEE
Transactions on Multimedia
[2] P. Daras, M. Lazaridis, Timotheos Kastrinogiannis,Vasileios Karyotisand
Christos Malavazos, “A Novel Annotation Propagation Framework for 3D
Object Databases Combining Geometric Features and Relevance
Feedback Mechanisms”, IEEE Transactions on Multimedia
Special Issue on Integration of Context and Content for Multimedia
Management
[3] A.Mademlis, P.Daras, D.Tzovaras and M.G.Strintzis: “3D Object Retrieval
using a 3D Shape Impact Descriptor” IEEE Transactions on Multimedia
[4] A.Mademlis, P.Daras, D.Tzovaras and M.G.Strintzis: “Ellipsoidal Harmonics
for 3D Shape Description” IEEE Transactions on Multimedia
NAVS Spring 2008 Concertation Meeting - CERTH/ITI
29
Future Work
• 3D low-level feature extraction algorithms:
– Refinement & Integration
• P2P Network
– Intraconnectivity through different Victory Islands
– Support to some Victory island specific services, like
rendering and collaboration
– Support for mobile clients – Mobile Gateway
• Annotation and annotation propagation mechanisms
– Integration
• Relevance feedback algorithms
– Integration
• Ontology based retrieval, QoE, Visualization on mobile
devices, integration
NAVS Spring 2008 Concertation Meeting - CERTH/ITI
30