Transcript ifact - start [kondor.etf.rs]

Multimedia Research
in
E-Education
Veljko Milutinović, Fellow of the IEEE
An Overview of the Ongoing Projects
http://galeb.etf.bg.ac.yu/~vm
e-mail: [email protected]
Major R&D Bottlenecks
•
•
•
•
Integrated Educational Systems
Concept Understanding and Interactive Cooperation
Intelligent Search and Retrieval for Educational Purposes
Innovative Applications
Page 2 of 79
U. of California at Berkeley
•
•
•
•
Integrated Systems: Intel MES Grant
Concept Understanding: WISE
Intelligent Search: GIST
Innovative Applications: BMRC
Page 3 of 79
MIT
•
•
•
•
Integrated Systems: Microsoft I-Campus
Concept Understanding: OXYGEN
Intelligent Search: NCAM
Innovative Applications: NREN
Page 4 of 79
Stanford University
•
•
•
•
Integrated Systems: Challenge 2000 Multimedia
Concept Understanding: Media X
Intelligent Search: CBIR
Innovative Applications: SUMMIT, CERAS, MM Collab, Shakespeare
Page 5 of 79
Project Name (University)
•
•
•
•
Project Leader(s)
Project URL
Project Essence in ASCII
Project Essence in JPEG/MPEG
Page 6 of 79
Intel MES Grant (Berkeley)
• James Demmel
• http://www.intel.com/pressroom/archive/releases/CO081897.HTMA
• RISC approach to MES
Page 7 of 79
WISE (Berkeley)
• Eric Baumgartner
• http://wise.berkeley.edu/pages/intro/wiseIntro01.html
• Risk approach to K12
Page 8 of 79
GIST (Berkeley)
• Joe Hellerstein
• http://now.cs.berkeley.edu/AM
• Risk approach to generalized search tree for secondary and
multimedia storage; supports any lookup over that data
Page 9 of 79
BMRC (Berkeley)
• Larry Rowe
• http://www-plateau.cs.berkeley.edu/
• Risk approach to contents and technology management
Page 10 of 79
Microsoft I-Campus (MIT)
• Thomas L. Magnanti
• http://web.mit.edu/newsoffice/tt/1999/oct06/microsoft.html
• Risk approach to $25 million
Page 11 of 79
OXYGEN (MIT)
• Anant Agarval, John Ancorn, Krste Asanovic, Rodney Brooks, …
• http://oxygen.lcs.mit.edu/
• Risk approach to bringing abundant computation, multimedia,
and communication naturally into people's lives, through an
infrastructure of mobile and stationary devices connected by a
self-configuring network
Page 12 of 79
NCAM (MIT)
• Geoff Freed
• http://www.ncddr.org/du/researchexchange/v06n03/multi.html
• Risk approach to accessible multimedia and distant education
Page 13 of 79
NREN (MIT)
• MIT, ARPA, DOE, NASA, NSF
• http://www.ccic.gov/pubs/blue94/section.3.2.html
• Risk approach to gigabit communications infrastructure for
e-research and e-education
Page 14 of 79
Challenge 2000 MM (Stanford)
• Shari Golan, Barbara Means, Bill Penuel
• http://ctl.sri.com/projects/displayProject.jsp?Nick=ch2000mm
• Risk approach to the next decade challenges in MES
Page 15 of 79
Media X (Stanford)
• John Perry
• http://mediax.stanford.edu/about/education.html
• Risk approach to multimedia courses in interdisciplinary major
undergraduate and graduate programs
Page 16 of 79
CBIR (Stanford)
• Jia Li, James Z. Vang, Gio Wiederhold
• http://www-db.stanford.edu/IMAGE/
• Risk approach to contents based image retrieval: semanticssensitive integrated matching for picture libraries, wavelet-based
image indexing and searching
Page 17 of 79
SUMMIT (Stanford)
• Parvati Dev
• http://www-smi.stanford.edu/projects/summit.html
• Risk approach to MES in medicine
Page 18 of 79
Potentials of R&D in MES
through
Italy/Serbia Cooperation
An Overview of Belgrade University Projects
for High-Tech Computer Industry
in the USA and EU (IPSI)
Some of the Recent Projects
NCR, Compaq, SUN, Intel,
Comshare, Zycad, QSI, Virtual,
TechnologyConnect, BioPop, eT, MainStreetNetworks,
Salerno, Pisa, Siena, L’Aquila, ...
Page 20 of 79
Response: Industry
Page 21 of 79
Flynn, M. J., Computer Architecture, Jones and Bartlett, USA (96)
position 1 (12 citations)
Response:
Academia
Bartee, T. C., Computer Architecture and Logic Design, McGraw-Hill, USA (91)
position 1 (2 citations)
Tabak, D., RISC Systems (RISC Processor Architecture), Wiley, USA (91)
position 1s (6 citations)
Stallings, W., Reduced Instruction Set Computers (RISC Architecture), IEEE CS Press, Los Alamitos, California, USA
(90)
position 1s (3 citations)
Heudin, J. C., Panetto, C., RISC Architectures, Chapman-Hall, London, England (92)
position 3s (2 citations)
van de Goor, A. J., Computer Architecture and Design, Addison Wesley, Reading, Massachusetts, USA (2nd printing, 91)
position 4s (3 citations)
Tannenbaum, A., Structured Computer Organization (Advanced Computer Architecures), Prentice-Hall, USA (90)
position 5s (4 citations)
Feldman, J. M., Retter, C. T., Computer Architecture, McGraw-Hill, USA (94)
position 7s (2 citations)
Stallings, W., Computer Organization and Architecture, Prentice-Hall, USA (96)
position 9s (3 citations)
Murray, W., Computer and Digital System Architecture, Prentice-Hall, USA (90)
position >10s (2 citations)
Wilkinson, B., Computer Architecture, Prentice-Hall, USA (91)
position >10 (2 citations)
Decegama, A., The Technology of Parallel Processing (Parallel Processing Architectures), Prentice-Hall, USA (90)
position >10s (2 citations)
Baron, R. J., Higbie, L., Computer Architecture, Addison-Wesley, USA (92)
position >10s (1 citation)
Tabak, D., Advanced Microprocessors (Microcomputer Architecture), McGraw-Hill, USA (95)
position >10s (1 citation)
Zargham, M. R., Computer Architecture, Prentice-Hall, USA (96)
position >10s (1 citation)
Hennessy, J. L., Patterson, D. A., Computer Architecture: A Quantitative Approach, Morgan-Kaufmann, USA (96)
na (0 citations)
Hwang, K., Advanced Computer Architecture, McGraw-Hill, USA (93)
na (0 citations)
Kain, K., Computer Architecture, Addison-Wesley, USA (95)
na (0 citations)
Page 22 of 79
N.B.
ERRORS
MADE
&
LESSONS
LEARNED
Page 23 of 79
1
Page 24 of 79
2
Page 25 of 79
3
Page 26 of 79
Summary
The world’s best journals – IEEE (50):
A European record in ICT
Books with 7 Nobel Laureates:
Kenneth Wilson, Ohio (North-Holland)
Leon Cooper, Brown (Prentice-Hall)
Robert Richardson, Cornell (Kluwer-Academics)
Jerome Friedman, MIT (Kluwer-Academics)
Herb Simon, CMU (IOS)
Arno Penzias, AT&T (IOS)
Harold Kroto, England (IOS)
Page 27 of 79
Multimedia Internet Gallery
Funding:
Fraunhofer IPSI, Darmstadt, Germany
Implementation:
IPSI, Belgrade, Serbia
Project Termination Date:
August 31, 2003.
Users:
Germany, Serbia, USA, Canada,…
Demo:
www.ipsi.co.yu
Page 28 of 79
IPSI Belgrade, Ltd.
3D@3D MULTIMEDIA
ARTSHOP GALLERY
[email protected], [email protected]
http://galeb.etf.bg.ac.yu/vm, http://www.ipsi.co.yu
Page 29 of 79
Authors
Marinkovic Ivan
Stojanovski Aleksandar
Radakovic Miroslav
Skundric Nikola
Nikezic Gavro
Milutinovic Darko
Toskov Ivan
Vujovic Ivana
Milutinovic Veljko
Page 30 of 79
Anucojic Goran
Introduction – IPSI Belgrade
IPSI Belgrade is a company jointly founded by German and Serbian capital
Partners:
• IPSI Fraunhofer, Darmstadt, Germany
• Telecom Italia Learning Services, Italy
• NYU, School of Continuous Professional Studies, USA
• Purdue University, School of Technology, USA
Page 31 of 79
Introduction – IPSI Belgrade
- Multimedia Workspaces of the Future
- Multimedia Applications for the Web
- Environments for Cooperative Working and Learning
- Virtual Information and Knowledge Environments
- Mobile Interactive Media
- Open Adaptive Information Management Systems
- Publication Engineering and Technology
- Hardware Design and Operating Systems
- Networks and WWW
- Semantic Web and Datamining
Page 32 of 79
Introduction – IPSI Belgrade
Products:
• Advanced Multimedia Virtual Gallery
• Tools for B2B Matchmaking on the Web
• Web security for P2P, The injection cache, The STS cache,
Genetic Search with Spatial/Temporal Mutations, Customer
Browser Satisfaction Web Search, Browser Acceleration,
Technology Transfer, Testing Infrastructure for EBI,
Distant Web Educating Machine, e-Tourism, …
Page 33 of 79
MMAG: Problem Statement
- Creating Web based art gallery with
“look and feel” of the real world exhibitions
- Visitor moves through the gallery by
“walking with options”
- 2D on 3D and 3D on 3D,
with multimedia options
Page 34 of 79
Existing Solutions
- Musee national des Arts asiatiques
http://www.museeguimet.fr/tour-guimet/index.html
- Web Server of the Galleria degli Uffizi in Florence
http://www.uffizi.firenze.it
- The Distributed Interactive Virtual Environment (DIVE)
http://www.sics.se/dive/
- The Web3D Repository
http://www.web3d.org/vrml/artgal.htm
Page 35 of 79
Proposed Solution
- Virtual reality gallery: Multimedia In Action
- Advanced search capabilities: Dream Search
- Artist’s criteria room generation: Do It By Yourself
Page 36 of 79
Why is it better?
- Dynamically generated/exploitable gallery
- Content based search engine
- User satisfaction
Page 37 of 79
Conditions and Assumptions
- PC
- Internet connection
- Internet Explorer 5.0 or higher
- Netscape 7.0
- Cortona VRML plug-in for IE
- Basic multimedia tools and standards
Page 38 of 79
Analysis and Implementation
- Application is written in ASP.NET using C# as code-behind,
and ADO.NET for database access.
- Database server is SQL Server 2000.
- Communication with the database is entirely made through XML
(using SQLXML3.0 framework).
- Queries are made in XPath, while adding,
changing and deleting of the records is done through UpdateGrams.
- Application is optimized for Internet Explorer 5.0 or higher,
at the 1024x768 screen resolution. Netscape 7.0 or higher is also supported.
- 3D gallery is completely generated on the server side (dynamically)
using VRML.
Page 39 of 79
Track 1
Track Requirements:
•
•
To stand as the integrative part for the other two tracks
To provide:
– User interaction
– Database connectivity (database independent)
– Search functions
(simple and advanced using Track3 output)
– Information brokering between artists and buyers
– Administration tools
– Artworks management tools, etc.
– Thin client (3D scene generation on server side)
Page 40 of 79
Track 1
Development Tools:
•
•
•
Application server platform:
– Windows XP Professional
– IIS 5.1
– MS SQL Server 2000
Development platform:
– ASP.NET
– C# as code-behind.
Communication with the underlying database:
– XML & XSD using XPath queries (DB independent)
– Currently using SQLXML3.0 add-on for ADO.NET
Page 41 of 79
Track 1
Database Design:
Page 42 of 79
Track 1
Administrator Tools:
•
Separate entry point:
http://<server_address>/artshop/admin
Page 43 of 79
Track 1
Users & Exhibitors:
•
Entry point:
http://<server_address>/artshop/index.htm
Page 44 of 79
Track 1
Interesting Details:
• Native XML DBMS under development at IPSI Fraunhofer
• Practical testing of the XML/XPath database access
• Dynamic addition (to the system) of new multimedia types
• 3D view of search results
Page 45 of 79
Track 1
Interesting Details:
•
•
Application that can connect on the fly to any DBMS which supports XML/XPath is an interesting
and possibly useful idea
(user just has to set one XML file containing local field mapping, and one XSD to map the
database fields to the pre-defined scheme)
Cons:
– XPath queries are lot less powerful then standard SQL queries
– Inherently, loss of speed
(one complex SQL query had to be simulated with couple of XPath queries and additional
processing in the code).
– For now, SQLXML3.0 does not support complete
XPath standard.
Page 46 of 79
Track 1
Errors Made:
•
•
Initially, content analysis, picture processing, and adding data to database were completely
separated (as specified in the contract),
with the idea of later (partial) integration.
Turned out to be a bad idea
(required a lot of intervention
from the ArtShop system administrator when adding artworks).
Page 47 of 79
Track 1
Lessons Learned:
•
•
Problem solved by complete integration of forementioned tasks
into the one system process which monitors input directory, automatically schedules picture
processing and content analysis, and takes care of updating of all necessary fields in all
required databases.
With that, we achieved maximum automation,
reduced time needed for artwork addition,
and reduced amount of data transferred through the Internet
(between the administrator’s machine and the application host).
Page 48 of 79
Track 2
Image-Content-Oriented Search
Track requirements:
•
•
•
•
•
Images used for extracting objects are artistic paintings
Image analyses
Extraction of the features
Create XML file for each image
Fetch the database with the features
Page 49 of 79
Track 2
Algorithm:
Load parameters, input and output directory
Open the picture
Determine the filter value
Put the picture into the reduced matrix
Determine histogram
Create objects
Merge objects into bigger objects
Create sorted array of objects
Create database objects, prepare them,
and put them in XML file and tables in database
Page 50 of 79
Track 2
Determine histogram:
Create general histogram
Sort histogram
Remove zero values
Remove redundancies
Sort histogram
Refresh matrix
Page 51 of 79
Track 2
P8
Regions for
processing matrix:
P8
P3
P5
P4
P6
P5
P3
the first colon
P8
P3
P5
P4
P8
P8
the rest of the matrix
the last row
the last element in the last row
Page 52 of 79
the last colon
Any pixel left in the current region?
false
true
p8 = the current pixel
p8.oi == 0 (it doesn’t belong to any object)
Creating
objects:
false
true
create new object
Any neighbour pixel left?
false
true
px = the neighbour pixel
p8.oi != px.oi (don’t belong to the same object)
false
true
call matrix.unite_pixels method
take the next neighbour pixel
take the next pixel
Page 53 of 79
Is Picture_Objects list empty?
true
false
q = true; Edge[0] = Picture_Objects[0]
false
q==true (are there any objects inside Border which colors after translation
are the same as the color of Core object)
Merge
objects
into
bigger
objects:
true
i < number of objects inside Edge list
true
false
Take ith object
Find all neighbour objects, which colors after translation are the same as the color of Core object,
and put them into Border
i = i+1
Move all pixels from objects inside Edge to Core object
Remove objects that are inside Edge from Picture_Objects list
Is Border empty
false
Move all objects inside Border to Edge
q=false; add Core object to big list; Remove pixels belonging to Core object from
Picture_Objects
Page 54 of 79
true
Track 2
Tools used in development:
• C# programming language – the chosen tool
– Advantage: Includes the best properties
from other programming languages (C++, Java, Visual Basic)
– Disadvantage: slower processing speed than C++,
which is not necessary in this application
• C++ - the best alternative tool
– Advantage: faster processing speed (unnecessary)
– Disadvantage: more complicated code,
50% of all bugs due to use of pointers
Page 55 of 79
Track 2
Original picture:
Page 56 of 79
Track 2
Picture after applying
histogram values:
Page 57 of 79
Track 2
Picture represented
through extracted objects:
Page 58 of 79
Track 2
3D HSL space => 1D histogram
Index of histogram array
Hue
Saturation
Luminance
Description
0
any
any
<=30
black
1
any
any
>=lummax
white
2
any
<20
the darkest grey
...
…
…
>30
<30+lum
…
ilmax-1
any
<20
>=30+(ilmax-3)*lum
<30+(ilmax-2)*lum
the lightest grey
ilmax
<=hue/2
>=240hue/2
<=hue/2
>=240hue/2
…
>20
<20+1*sat
>30
<30+lum
the darkest red with the
smallest saturation
>=20+1*sat
< 20+2*sat
>30
<30+lum
the darkest red with smaller
saturation
…
…
…
<=hue/2
>=240hue/2
<=hue/2
>=240hue/2
<=hue/2
>=240hue/2
…
>=20+(isat-3)*sat
< 20 +(isat-2)*sat
>30
<30+lum
the darkest red with the
biggest saturation
>20
<20+1*sat
>=30+lum
< 30+2*lum
darker red with the smallest
saturation
>=20+1*sat
< 20+2*sat
>=30+lum
< 30+2*lum
darker red with smaller
saturation
…
…
…
<=hue/2
>=240hue/2
<=hue/2
>=240hue/2
>=20+(isat-3)*sat
< 20 +(isat-2)*sat
>=30+lum
< 30+2*lum
darker red with the biggest
saturation
>20
<20+1*sat
>=30+2*lum
< 30+3*lum
dark red with the smallest
saturation
ilmax+1
…
ilmax+isat-2
ilmax+1*(isat-1)
ilmax+1*(isat-1)+1
…
ilmax+1*(isat-1)+isat-2
ilmax+2*(isat-1)
Page 59 of 79
…
Track 2
3D HSL space => 1D histogram
ilmax+2*(isat-1)+1
<=hue/2
>=240-hue/2
>=20+1*sat
< 20+2*sat
>=30+2*lum
< 30+3*lum
dark red with smaller
saturation
…
…
…
…
…
ilmax+(ilmax-3)*(isat-1)+1
<=hue/2
>=240-hue/2
>=20+1*sat
< 20+2*sat
>=30+(ilmax-3)*lum
< 30+(ilmax-2)*lum
the brightest red with
smaller saturation
…
…
…
…
…
ilmax+(ilmax-3)*(isat-1)+(isat-2)
<=hue/2
>=240-hue/2
>=20+(isat-3)*sat
< 20 +(isat-2)*sat
>=30+(ilmax-3)*lum
< 30+(ilmax-2)*lum
the brightest red with the
biggest saturation
ilmax+1*(ilmax-2)*(isat-1)
> hue/2
< 3*hue/2
>20
<20+1*sat
>30
<30+lum
ilmax+1*(ilmax-2)*(isat-1)+1
> hue/2
< 3*hue/2
>=20+1*sat
< 20+2*sat
>30
<30+lum
the darkest orange-red
with the smallest
saturation
the darkest orange-red
with smaller saturation
…
…
…
…
…
ilmax+1*(ilmax-2)*(isat-1)+(ilmax-3)*(isat1)+(isat-2)
> hue/2
< 3*hue/2
>=20+(isat-3)*sat
< 20 +(isat-2)*sat
>=30+(ilmax-3)*lum
< 30+(ilmax-2)*lum
the brightest orange-red
with the biggest saturation
ilmax+2*(ilmax-2)*(isat-1)
>=3*hue/2
< 5*hue/2
>20
<20+1*sat
>30
<30+lum
ilmax+2*(ilmax-2)*(isat-1)+1
>=3*hue/2
< 5*hue/2
>=20+1*sat
< 20+2*sat
>30
<30+lum
the darkest red-orange
with the smallest
saturation
the darkest red-orange
with smaller saturation
…
…
…
…
…
ilmax+(ihmax-1)*(ilmax-2)*(isat-1)+1
>=(2*ihmax1)*hue/2
<(2*ihmax+1)*
…
hue/2
>=20+1*sat
< 20+2*sat
>30
<30+lum
the darkest magenta-red
with smaller saturation
…
…
…
>=(2*ihmax1)*hue/2
<(2*ihmax+1)*
hue/2
>=20+(isat-3)*sat
< 20 +(isat-2)*sat
>=30+(ilmax-3)*lum
< 30+(ilmax-2)*lum
the brightest magenta-red
with the biggest saturation
…
ilmax+(ihmax-1)*(ilmax-2)*(isat-1)+(ilmax3)*(isat-1)+(isat-2)
Page 60 of 79
Track 2
Lessons learned:
• It is impossible to extract objects using only colors as a criterion
• It is impossible to extract objects,
even using textures, edges, different transformations as criteria
• Semantics should be used in segmentation
• Colors are the most important features in artistic paintings
Page 61 of 79
Track 3
Track requirements:
•
•
•
•
•
Possibility of moving through 3D galleries
Automatic generation of 3D galleries based on user’s query
Manual generation of 3D galleries
User interface for image zooming
Application for image processing
Page 62 of 79
Track 3
Underlying algorithms:
•
•
•
•
Dynamic creation of gallery
Creation of static galleries
Algorithm for picture zooming
Algorithm for picture processing
Page 63 of 79
Track 3
Creation of galleries:
•
•
•
•
•
Validation of the created gallery
Forming VRML files depending on users query
Determining the number of pictures in the gallery
Drawing a 2D floorplan based on the 3D gallery
“Forest fire” algorithm for filling the floorplan with color
Page 64 of 79
Track 3
Picture processing:
•
•
•
•
Loading image into memory
Clone image into different-size copies
Filtering of copies
Parting of copies
Page 65 of 79
Track 3
Development tools:
• C# in .NET Framework for programming image processing
• Macromedia Dreamweaver for programming zoom tool
• VRML Pad v2.0
Page 66 of 79
Track 3
Flowchart:
Page 67 of 79
Track 3
Creation of galleries:
• Making files based on users data
• Putting data on server
so it can be available for artists
• Artist chooses which gallery
he/she will be using for exhibition
• User can move through 3D world
• Selecting the textures for gallery
• Selecting the starting position of the user
Page 68 of 79
Track 3
NF filter details:
• If the new picture is smaller,
every pixel is one pixel of the old picture.
• If the new picture is bigger,
pixels are calculated based on the pixels surrounding the current.
Page 69 of 79
Track 3
Errors made:
• Requests were not precise,
so there was a gap at the end of the project between wanted and done
• Better results could be done with better using of ASP
and XML
Page 70 of 79
Track 3
Lessons learned:
• Every member of the team gets a part
where his experience is dominating
• More planning at the start
reduces a lot of work later
• Good communication between programmers
can save a lot of time
Page 71 of 79
Demo
Page 72 of 79
Future Plans
• Improving the existing 3D dynamic gallery
• Improving search engine capabilities
• Improving feature extraction algorithms
and objects recognition
Page 73 of 79
Future Track 1
3D Multimedia Showroom Environment:
• Implementing a generalized
Web based 3D Multimedia Showroom Environment
• Exhibiting various MM data types:
images, 3D objects, videos, audio, etc.
• Set of MM data types should be extendable
Page 74 of 79
Future Track 2
MM Object Feature Extraction:
• Implementing algorithms and software components
for extracting features from MM data types
(images, videos, 3D objects),
in order to enable content based search
• System should be extendable (“plug-in”)
Page 75 of 79
Future Track 3
Semantic Abstraction of MM Feature Spaces:
• Developing methods and SW components
which derive mapping
from extracted features of MM objects to semantic concepts
• Using intelligent classification algorithms
(Neural Networks, Fuzzy Classifier)
• Developing semantic query engine
(answering questions,
which could previously only be answered by humans)
Page 76 of 79
Usability
• Art Galleries
• Museums
• Exhibition Fairs
Page 77 of 79
Instead of Conclusion
IPSI Belgrade, [email protected]
http://www.ipsi.co.yu
Page 78 of 79
http://galeb.etf.bg.ac.yu/~vm/
e-mail: [email protected]
Page 79 of 79