Transcript No Slide Title - start [kondor.etf.rs]

IPSI Belgrade Ltd.
ARTSHOP GALLERY
[email protected]
[email protected]
http://www.ipsi.co.yu
This material was developed with financial help of the WUSA fund of Austria.
1/50
Authors
Marinkovic Ivan
Stojanovski Aleksandar
Skundric Nikola
Radakovic Miroslav
Nikezic Gavro
Milutinovic Darko
Toskov Ivan
Vujovic Ivana
Milutinovic Veljko
Anucojic Goran
2/50
Introduction – IPSI Bgd
IPSI Belgrade is a company jointly founded by
German and Serbian capital
Partners:
- IPSI Fraunhofer, Darmstadt, Germany
- Telecom Italia Learning Services, Italy
- NYU, School of Continuous Professional Studies, USA
- Instituto Tecnologico de Durango, Mexico
3/50
Introduction – IPSI Bgd
- Workspaces of the Future
- Environments for Cooperative Working and Learning
- Virtual Information and Knowledge Environments
- Mobile Interactive Media
- Open Adaptive Information Management Systems
- Publication Engineering and Technology
- Hardware Design and Operating Systems
- Networks and WWW
4/50
Introduction – IPSI Bgd
Products:
• Advanced Virtual Gallery
• Semantic Web tutorial and book development
• The injection cache, the STS cache, VLSI Detection for
Internet/Telephony Interfaces, Genetic Search
with Spatial/Temporal Mutations, Browser Acceleration,
Technology transfer, Testing the Infrastructure for EBI,
Socratenon Distant Web Educating Machine, e-Tourism…
5/50
Problem Statement
- Creating Web based art gallery with
“look and feel” of the real world exhibitions
- Visitor moves through the gallery by
“walking with options”
6/50
Existing Solutions
- Musee national des Arts asiatiques
http://www.museeguimet.fr/tour-guimet/index.html
- Web Server of the Galleria degli Uffizi in Florence
http://www.uffizi.firenze.it
- The Distributed Interactive Virtual Environment (DIVE)
http://www.sics.se/dive/
- The Web3D Repository
http://www.web3d.org/vrml/artgal.htm
7/50
Proposed Solution
- Virtual reality gallery
- Advanced search capabilities
- Visitor’s criteria based room generation
8/50
Why is it better?
- Dynamically generated gallery
- Content based search engine
- User satisfaction
9/50
Conditions and Assumptions
- PC
- Internet connection
- Internet Explorer 5.0 or higher
- Netscape 7.0
- Cortona VRML plug-in for IE
10/50
Analysis and Implementation
- Application is written in ASP.NET using C# as code-behind,
and ADO.NET for database access.
- Database server is SQL Server 2000.
- Communication with the database is entirely made through XML
(using SQLXML3.0 framework).
- Queries are made in XPath, while adding,
changing and deleting of the records is done through UpdateGrams.
- Application is optimized for Internet Explorer 5.0 or higher,
at the 1024x768 screen resolution. Netscape 7.0 or higher is also supported.
- 3D gallery is completely generated on the server side (dynamically)
using VRML.
11/50
Track 1
Track Requirements:
• To stand as the integrative part for the other two tracks
• To provide:
– User interaction
– Database connectivity (database independent)
– Search functions
(simple and advanced using Track3 output)
– Information brokering between artists and buyers
– Administration tools
– Artworks management tools, etc.
– Thin client (3D scene generation on server side)
12/50
Track 1
Development Tools:
• Application server platform:
– Windows XP Professional
– IIS 5.1
– MS SQL Server 2000
• Development platform:
– ASP.NET
– C# as code-behind.
• Communication with the underlying database:
– XML & XSD using XPath queries (DB independent)
– Currently using SQLXML3.0 add-on for ADO.NET
13/50
Track 1
Database Design:
14/50
Track 1
Administrator Tools:
• Separate entry point:
http://<server_address>/artshop/admin
15/50
Track 1
Users & Exhibitors:
• Entry point:
http://<server_address>/artshop/index.htm
16/50
Track 1
Interesting Details:
• Native XML DBMS under development at IPSI Fraunhofer
• Practical testing of the XML/XPath database access
• Dynamic addition (to the system) of new multimedia types
• 3D view of search results
17/50
Track 1
Interesting Details:
• Application that can connect on the fly to any DBMS which
supports XML/XPath is an interesting and possibly useful idea
(user just has to set one XML file containing local field mapping,
and one XSD to map the database fields to the pre-defined scheme)
• Cons:
– XPath queries are lot less powerful then standard SQL queries
– Inherently, loss of speed
(one complex SQL query had to be simulated with couple of
XPath queries and additional processing in the code).
– For now, SQLXML3.0 does not support complete
XPath standard.
18/50
Track 1
Errors Made:
• Initially, content analysis, picture processing, and adding data to
database were completely separated (as specified in the contract),
with the idea of later (partial) integration.
• Turned out to be a bad idea
(required a lot of intervention
from the ArtShop system administrator when adding artworks).
19/50
Track 1
Lessons Learned:
• Problem solved by complete integration of forementioned tasks
into the one system process which monitors input directory,
automatically schedules picture processing and content analysis,
and takes care of updating of all necessary fields in all required
databases.
• With that, we achieved maximum automation,
reduced time needed for artwork addition,
and reduced amount of data transferred through the Internet
(between the administrator’s machine and the application host).
20/50
Track 2
Image-Content-Oriented Search
Track requirements:
•
•
•
•
•
Images used for extracting objects are artistic paintings
Image analyses
Extraction of the features
Create XML file for each image
Fetch the database with the features
21/50
Track 2
Algorithm:
Load parameters, input and output directory
Open the picture
Determine the filter value
Put the picture into the reduced matrix
Determine histogram
Create objects
Merge objects into bigger objects
Create sorted array of objects
Create database objects, prepare them,
and put them in XML file and tables in database
22/50
Track 2
Determine histogram:
Create general histogram
Sort histogram
Remove zero values
Remove redundancies
Sort histogram
Refresh matrix
23/50
Track 2
Regions for
processing matrix:
P8
P8
P3
P5
P4
P6
P5
P3
the first colon
P8
P3
P5
P4
P8
P8
the rest of the matrix
the last row
the last element in the last row
the last colon
24/50
Any pixel left in the current region?
false
true
p8 = the current pixel
Creating
objects:
p8.oi == 0 (it doesn’t belong to any object)
false
true
create new object
Any neighbour pixel left?
false
true
px = the neighbour pixel
p8.oi != px.oi (don’t belong to the same object)
false
true
call matrix.unite_pixels method
take the next neighbour pixel
take the next pixel
25/50
Is Picture_Objects list empty?
true
false
q = true; Edge[0] = Picture_Objects[0]
false
Merge
objects
into
bigger
objects:
q==true (are there any objects inside Border which colors after translation
are the same as the color of Core object)
true
i < number of objects inside Edge list
true
false
Take ith object
Find all neighbour objects, which colors after translation are the same as the color of Core object,
and put them into Border
i = i+1
Move all pixels from objects inside Edge to Core object
Remove objects that are inside Edge from Picture_Objects list
Is Border empty
false
true
Move all objects inside Border to Edge
q=false; add Core object to big list; Remove pixels belonging to Core object from
Picture_Objects
26/50
Track 2
Tools used in development:
• C# programming language – the chosen tool
– Advantage: Includes the best properties
from other programming languages (C++, Java, Visual Basic)
– Disadvantage: slower processing speed than C++,
which is not necessary in this application
• C++ - the best alternative tool
– Advantage: faster processing speed (unnecessary)
– Disadvantage: more complicated code,
50% of all bugs due to use of pointers
27/50
Track 2
Original picture:
28/50
Track 2
Picture after applying
histogram values:
29/50
Track 2
Picture represented
through extracted objects:
30/50
Track 2
3D HSL space => 1D histogram
Index of histogram array
Hue
Saturation
Luminance
Description
0
any
any
<=30
black
1
any
any
>=lummax
white
2
any
<20
the darkest grey
...
…
…
>30
<30+lum
…
ilmax-1
any
<20
>=30+(ilmax-3)*lum
<30+(ilmax-2)*lum
the lightest grey
ilmax
<=hue/2
>=240hue/2
<=hue/2
>=240hue/2
…
>20
<20+1*sat
>30
<30+lum
the darkest red with the
smallest saturation
>=20+1*sat
< 20+2*sat
>30
<30+lum
the darkest red with smaller
saturation
…
…
…
<=hue/2
>=240hue/2
<=hue/2
>=240hue/2
<=hue/2
>=240hue/2
…
>=20+(isat-3)*sat
< 20 +(isat-2)*sat
>30
<30+lum
the darkest red with the
biggest saturation
>20
<20+1*sat
>=30+lum
< 30+2*lum
darker red with the smallest
saturation
>=20+1*sat
< 20+2*sat
>=30+lum
< 30+2*lum
darker red with smaller
saturation
…
…
…
<=hue/2
>=240hue/2
<=hue/2
>=240hue/2
>=20+(isat-3)*sat
< 20 +(isat-2)*sat
>=30+lum
< 30+2*lum
darker red with the biggest
saturation
>20
<20+1*sat
>=30+2*lum
< 30+3*lum
dark red with the smallest
saturation
ilmax+1
…
ilmax+isat-2
ilmax+1*(isat-1)
ilmax+1*(isat-1)+1
…
ilmax+1*(isat-1)+isat-2
ilmax+2*(isat-1)
…
31/50
Track 2
3D HSL space => 1D histogram
ilmax+2*(isat-1)+1
<=hue/2
>=240-hue/2
>=20+1*sat
< 20+2*sat
>=30+2*lum
< 30+3*lum
dark red with smaller
saturation
…
…
…
…
…
ilmax+(ilmax-3)*(isat-1)+1
<=hue/2
>=240-hue/2
>=20+1*sat
< 20+2*sat
>=30+(ilmax-3)*lum
< 30+(ilmax-2)*lum
the brightest red with
smaller saturation
…
…
…
…
…
ilmax+(ilmax-3)*(isat-1)+(isat-2)
<=hue/2
>=240-hue/2
>=20+(isat-3)*sat
< 20 +(isat-2)*sat
>=30+(ilmax-3)*lum
< 30+(ilmax-2)*lum
the brightest red with the
biggest saturation
ilmax+1*(ilmax-2)*(isat-1)
> hue/2
< 3*hue/2
>20
<20+1*sat
>30
<30+lum
ilmax+1*(ilmax-2)*(isat-1)+1
> hue/2
< 3*hue/2
>=20+1*sat
< 20+2*sat
>30
<30+lum
the darkest orange-red
with the smallest
saturation
the darkest orange-red
with smaller saturation
…
…
…
…
…
ilmax+1*(ilmax-2)*(isat-1)+(ilmax-3)*(isat1)+(isat-2)
> hue/2
< 3*hue/2
>=20+(isat-3)*sat
< 20 +(isat-2)*sat
>=30+(ilmax-3)*lum
< 30+(ilmax-2)*lum
the brightest orange-red
with the biggest saturation
ilmax+2*(ilmax-2)*(isat-1)
>=3*hue/2
< 5*hue/2
>20
<20+1*sat
>30
<30+lum
ilmax+2*(ilmax-2)*(isat-1)+1
>=3*hue/2
< 5*hue/2
>=20+1*sat
< 20+2*sat
>30
<30+lum
the darkest red-orange
with the smallest
saturation
the darkest red-orange
with smaller saturation
…
…
…
…
…
ilmax+(ihmax-1)*(ilmax-2)*(isat-1)+1
>=(2*ihmax1)*hue/2
<(2*ihmax+1)*
…
hue/2
>=20+1*sat
< 20+2*sat
>30
<30+lum
the darkest magenta-red
with smaller saturation
…
…
…
>=(2*ihmax1)*hue/2
<(2*ihmax+1)*
hue/2
>=20+(isat-3)*sat
< 20 +(isat-2)*sat
>=30+(ilmax-3)*lum
< 30+(ilmax-2)*lum
the brightest magenta-red
with the biggest saturation
…
ilmax+(ihmax-1)*(ilmax-2)*(isat-1)+(ilmax3)*(isat-1)+(isat-2)
32/50
Track 2
Lessons learned:
• It is impossible to extract objects using only colors as a criterion
• It is impossible to extract objects,
even using textures, edges, different transformations as criteria
• Semantics should be used in segmentation
• Colors are the most important features in artistic paintings
33/50
Track 3
Track requirements:
•
•
•
•
•
Possibility of moving through 3D galleries
Automatic generation of 3D galleries based on user’s query
Manual generation of 3D galleries
User interface for image zooming
Application for image processing
34/50
Track 3
Underlying algorithms:
•
•
•
•
Dynamic creation of gallery
Creation of static galleries
Algorithm for picture zooming
Algorithm for picture processing
35/50
Track 3
Creation of galleries:
•
•
•
•
•
Validation of the created gallery
Forming VRML files depending on users query
Determining the number of pictures in the gallery
Drawing a 2D floorplan based on the 3D gallery
“Forest fire” algorithm for filling the floorplan with color
36/50
Track 3
Picture processing:
•
•
•
•
Loading image into memory
Clone image into different-size copies
Filtering of copies
Parting of copies
37/50
Track 3
Development tools:
• C# in .NET Framework for programming image processing
• Macromedia Dreamweaver for programming zoom tool
• VRML Pad v2.0
38/50
Track 3
Flowchart:
39/50
Track 3
Creation of galleries:
• Making files based on users data
• Putting data on server
so it can be available for artists
• Artist chooses which gallery
he/she will be using for exhibition
• User can move through 3D world
• Selecting the textures for gallery
• Selecting the starting position of the user
40/50
Track 3
NF filter details:
• If the new picture is smaller,
every pixel is one pixel of the old picture.
• If the new picture is bigger,
pixels are calculated based on the pixels surrounding the current.
41/50
Track 3
Errors made:
• Requests were not precise,
so there was a gap at the end of the project between wanted and done
• Better results could be done with better using of ASP
and XML
42/50
Track 3
Lessons learned:
• Every member of the team gets a part
where his experience is dominating
• More planning at the start
reduces a lot of work later
• Good communication between programmers
can save a lot of time
43/50
Demo
44/50
Future Plans
• Improving the existing 3D dynamic gallery
• Improving search engine capabilities
• Improving feature extraction algorithms
and objects recognition
45/50
Future Track 1
3D Multimedia Showroom Environment:
• Implementing a general
Web based 3D Multimedia Showroom Environment
• Exhibiting various MM data types:
images, 3D objects, videos, audio, etc.
• Set of MM data types should be extendable
46/50
Future Track 2
MM Object Feature Extraction:
• Implementing algorithms and software components
for extracting features from MM data types
(images, videos, 3D objects),
in order to enable content based search
• System should be extendable (“plug-in”)
47/50
Future Track 3
Semantic Abstraction of MM Feature Spaces:
• Developing methods and SW components
which derive mapping
from extracted features of MM objects to semantic concepts
• Using intelligent classification algorithms
(Neural Networks, Fuzzy Classifier)
• Developing semantic query engine
(answering questions,
which could previously only be answered by humans)
48/50
Usability
• Art Galleries
• Museums
• Exhibition Fairs
49/50
Instead of Conclusion
IPSI Belgrade, [email protected]
http://www.ipsi.co.yu
50/50