Transcript MegaStore - Database Services for Music Industry

Databases and Web-Based Applications
The MegaStore Application
Advanced Internet-based Electronic
Commerce Technologies for Music Industry
Ammar Benabdelkader
University of Amsterdam
December 1, 2003
UvA, Amsterdam
A. Benabdelkader ©UvA, 2002
Motivation
• Large number of emerging e-commerce applications.
These applications comprise large multimedia data, for
which:
– Data classification and cataloguing are required
– Related information needs to be properly inter-linked
– Efficient storage, and access of large multimedia data is supported
– Short response time for on-line requests must be guaranteed
– High data transfer rates must be provided
• Furthermore, public information must be separated from
the private information that need to be securely kept at the
place where it belongs.
A. Benabdelkader ©UvA, 2002
Approach
• An efficient architecture must be designed addressing the need
to provide the user of electronic commerce with an environment
through which he can experience as sufficiently close to real life
shopping environment. The approach considers the following
aspects:
– Security for data access and users authentication
– Suitable user friendly interfaces
– Performance issues must be addressed
– Private data need to be protected
– Public data is made available to the Internet users
• The MegaStore system aims at the design and set-up of the
necessary database structure and platform architecture for
advanced e-commerce applications
A. Benabdelkader ©UvA, 2002
Problem Analysis
• From the analysis of music industry application, the following aspects
are identified:
– The data is geographically distributed over the network
– Information about music is classified into two main categories:
• The general information stored at the directory services (database catalogue)
and accessible by all the Internet users
• The raw music data that can only be accessible by the music storekeepers at
music centers or burning towers
– Depending on the user profile and authorization, only a part of the
information can be accessed and users need not to know about the data
distribution
– The real music data must be securely transferred through a dedicated
network
– High bandwidth connection is necessary to handle raw music data that
need to be transferred between the music storage centers and the burning
towers
– Low latency network connection is necessary to support the huge number
of users expected to connect to the system
A. Benabdelkader ©UvA, 2002
Database Design
SONG
Album
Order
Song ID: Short
Song Title: String
Album ID: Short
AlbumOrders
0..* Order ID: Short
Song Type: String
SongOfAlbum 0..* Album Title: String
Order Date: Date
Song Genre: String
Album Publisher: String 1..*
OrderAlbums
1..* AlbumSongs
Release Date: Date
Album Cover: Image
Song Lyrics: String
Album Price: Float
0..*
Song Price: Real
Album PubYear: Date
Status Code: Char
0..*
Song Label: String
OrderedBy CustOrders
Song Barcode: Short
AlbumArtists
ArtistAlbums
Sort Code: Char
Song Prefix: String
1..1
1..*
Song Suffix: String
Song Flags: String
Customer
ARTIST
Song Duration: Time
SungBy
Song Language: String
Cust ID: Short
1..* Artist Name: String
Song Stream: Audio
Cust Name: String
1..*
Artist Bio: String
ArtistSongs
Song Video: Video
Cust Faithful: Short
Artist Language: String
Cust Company: String
0..*
Artist Image: Image
Cust Address: String
Uses
Used In
Cust Country: String
Cust Email: String
0..*
Instrument
Composer
Performer
InstName: String
InstCat: String
A. Benabdelkader ©UvA, 2002
The MegaStore System Architecture (1)
The designed system architecture involves the following
components:
– The back-end system, including the database engine
and the predefined networking connection between the
MegaStore system components
– The front-end system, including:
• The Internet–Shop interface, where a user from home (or
work place) can search for music, listen/watch to the
audio/video clips, and order CDs, and
• The Shop-in-a-Shop interface, where the music storekeeper
can fetch on-line the real music data from its original source in
order to burn at run-time the requested music CDs
A. Benabdelkader ©UvA, 2002
The MegaStore System Architecture (2)
Parallel Distributed
Database Server
WWW
Browser
Remote
Applications
Web Server
Shop-in-a-shop
Server
Web Server
Internet
Local
database
Internet Shop
Directory Services
Interface
Intranet
Internet Shop
Interface
Local
database
Database Catalogue
Local
database
Local
database
LAN
Web Server
Web Server
Internet
WWW
Browser
Shop-in-a-shop Shop-in-a-shop
Interface
Interface
High bandwidth
medium bandwidth- medium latency
Remote Client
Low latency
A. Benabdelkader ©UvA, 2002
Activity Diagram for the InternetShop Interface
User
Login
Internet
User
S
Level 1
Artist
Entry
Song
Entry
Album
Entry
Show
Help
Level 2
Show
Artists
Show
Songs
Show
Albums
Order
albums
Level 3
Artist
Songs
Song
Artists
Album
Songs
Custom
Orders
A. Benabdelkader ©UvA, 2002
Internet Web Interface (1)
A. Benabdelkader ©UvA, 2002
Internet Web Interface (2)
(a) Song Search
(b) Album Search
A. Benabdelkader ©UvA, 2002
Dynamic Browsing of Interface
(A) Artist Songs
(B) Album Songs
(A)
(B)
A. Benabdelkader ©UvA, 2002
MegaStore Ordering System
User
Logging
Logged
Order
Created
Submitted
Order
Handling
Handled
Order
Archived
State Diagram for Orders
(a)Standard Order
(b)Custom Order
A. Benabdelkader ©UvA, 2002
System Implementation
• The database schema is implemented on top of the Matisse object-oriented
database system
• The Matisse back-end database runs on a cluster composed of 20 nodes.
• The MegaStore Internet interface is implemented using an NT front-end
machine, that is in turn connected to the underlying back-end database.
• The Internet-shop server prototype is implemented using a combination of
the following software technologies:
– JavaScript and Vbscript for tips programming,
– Active Database Objects (ADO) for database access, and
– HTML for text formatting
• the server implementation is made possible using the Active Server Pages
(ASP) environment that allows the combination of different software
technologies in one single environment.
A. Benabdelkader ©UvA, 2002
Information Management Strategy
F- A
F- B
F- C
DBA
Program
Legacy systems
(Flat-files and Databases)
Database access
Adapter
Adapter
Adapter
- ODL
- OIF
C++
Access independent Framework
JAVA
ODBMS
SQL
User
Interfaces
SQL3
Object-oriented
Facilitating Applications
- ODBC
- XML
- JDBC, Java
Databases
- ODL is used to support the portability of database schemas across conforming ODBMSs.
- OIF is used to exchange objects between databases and provide database documentation.
Applications
- Universal data access through standards and middle ware solutions (XML, ODBC, JDBC, JAVA, etc.).
A. Benabdelkader ©UvA, 2002
Management of Large
Multi-Media Data
• An efficient architecture must be designed for
manipulation of large data sets. Therefore,
Data
management mechanisms must be addressed in such a way
that:
– Data needs to be properly searched, retrieved, published, interlinked, and compared to other data,
– Information security is preserved,
– Performance issues are improved,
– Private data is protected, while published data is made available to
the outside users.
A. Benabdelkader ©UvA, 2002
Data Storage Approaches
• File System Approach and HFS (Hierarchical File System)
• External Data Link Approach
– database catalogue refers to NFS (network file system)
• One-Database Storage Approach
• Parallel/Distributed Database Server
• etc.
A. Benabdelkader ©UvA, 2002
1- File System Approach
• Inefficient in:
– Maintaining the link between the inter-related pieces of information,
– Comparing related data in different applications,
– Supporting the ability to query and modify the data using appropriate
query languages,
– Searching the stored information and supporting efficient access to
data items,
– Preserving the system coherency, data is scattered in various files of
different formats.
• Applications are based on direct access to local file system: they are
hard to maintain and to extend.
Disk
Application
A. Benabdelkader ©UvA, 2002
2- External Data Link Approach
• A database catalogue is used together with the file system:
– Database provides references to all objects stored locally or remotely
– This approach solves problems related to database overload and improves
DB performance
– Data is distributed so that it is physically located closest to intensive usage
sites
– Result files can be archived at (or close to) the point where they are
generated
– Proper data distribution reduces access bottlenecks at individual sites
Link
Database
Catalogue
Disk
Application
A. Benabdelkader ©UvA, 2002
2- External Data Link Approach (Cont.)
Two problems are faced when using this approach:
• Database catalogue consistency: referenced binary objects can be
updated/removed without notifying the database catalogue maintainer
– Solution: a specific module that automatically and periodically checks
the availability of the referenced objects against the database catalogue
• Security issue: referenced objects, which are usually stored in a
public location, are not secure.
– Solution: development of a remote file server, through which, the file
access and user authentication are controlled based on the database
catalogue information
A. Benabdelkader ©UvA, 2002
3- One-Database Storage Solution
• In ideal case, if a database is being created to manage the meta-data,
then that database must also store the large scientific data
– This approach solves the problem of keeping the meta-data
synchronized with the archive
• However, since the scientific binary data is of huge size:
– It is very costly to store/access the data as large binary objects within
the database itself
– Access mechanisms to binary objects require extra encoding/decoding
facilities
Database
Application
A. Benabdelkader ©UvA, 2002
4- Parallel/Distributed Database Server
• This approach uses a database repository to store the general
information, and a distributed database server to store large objects:
– The database repository is better exploited for cataloguing, indexing,
and searching facilities
– The database server enforces the issues related to security for access,
concurrency control, and information visibility rights.
User Queries
Query
Processor
Transaction
Manager
Storage
Manager
Application
DBMS Software
DB
Repository &
Metadata
DB
Database Server
DB
DB
Database System
A. Benabdelkader ©UvA, 2002
Parallel/Distributed Database Server Example
Database
Server
WWW
Browser
LAN
Internet
Web-based
Application
Web
Interface
Internet
Storage
Manager
Database
Catalog
WWW
Browser
Database
Server
Remote User
End Users
Common
Access Point
Database
Server
Parallel/distributed
database Server
A. Benabdelkader ©UvA, 2002
Distributed Parallel Server Extension
The parallel/distributed database framework provides the MegaStore
web server with efficient access to the raw music data. The nodes
(music stores) of the distributed MegaStore server are interconnected, making it possible for specific users to connect to any
node in the distributed server and to request an object, without the
need to know where that object actually resides
The distributed database supports the following required
functionalities:
–
–
–
–
Provides a way for managing huge amount of data
Data is securely kept at geographically distributed music centers
Data is stored only at the point(s) where it belongs
Data is visible from any node (music center) within the cooperation
community
– Data is efficiently transferred between the nodes in short response
time
A. Benabdelkader ©UvA, 2002
The MegaStore Prototypes
- The FRS Application
- The LuisterPaal Interface
- Sheet Music Server
A. Benabdelkader ©UvA, 2002
Achievement
• Design and development of an e-MegaStore
application that seamlessly fits several emerging
applications and supports their model of operations
• Design Methodology:
– A conceptual model
– A computer system architecture
– An Implementation
– Prototype
A. Benabdelkader ©UvA, 2002
Conceptual Model
Content Suppliers
Partners
Value
Adding
Partners
Information
Integration and Value
Adding
Retailer Shops and home
users
A. Benabdelkader ©UvA, 2002
System Architecture
Multimedia
Database
Suppliers
Major
Content
Suppliers
Retail Value Adding
FRS
NL
FRS
BE
Van
Leest
MSI
Musi
c
Hemi
spher
e
Catalogue
External Links
Additional Inf.
FireWall
Upload & Storage
Interne
t
Shop Network
Internet Users
Advanced In-Shop Systems
A. Benabdelkader ©UvA, 2002
The FRS-Pilot Application
Multimedia Database Server
Music Content
Multimedia
Database
Catalogue
Additional Inf.
Back Office
SQL Server
Internet
Shop
Database
Catalogue
LAN
Interne
t
Cache
Music Center
Shops
Catalogue
WWW
Browser
Internet User
A. Benabdelkader ©UvA, 2002
Security for the Multimedia
DB
Multimedia
Database Server
Multimedia
Content
Short Clips
Audio
Data is Secure but
Not Encrypted
Medium Tracks High Quality
Audio/Video Audio Tracks
Data is Secure and
Encrypted
Data is Secure and
Highly Encrypted
A. Benabdelkader ©UvA, 2002
FRS Application
Main Interface
A. Benabdelkader ©UvA, 2002
FRS Application
LuisterPaal Interface
A. Benabdelkader ©UvA, 2002
Sheet Music Application Main
Interface
A. Benabdelkader ©UvA, 2002
Sheet Music Application
Subscription Interface
A. Benabdelkader ©UvA, 2002
Sheet Music Application
Music Book Search Interface
A. Benabdelkader ©UvA, 2002
Sheet Music Application
Composer Search Interface
A. Benabdelkader ©UvA, 2002
Sheet Music Application
Music Notes Example
A. Benabdelkader ©UvA, 2002
Conclusion
• The main idea behind the developed framework is to
design a comprehensive system to support advanced
web-applications with two specific characteristics:
– to hold large data sets and
– to manage multimedia information
• Thus, the MegaStore system can be considered as a
general implementation approach that proves the
validity of the proposed architecture and design.
• From this framework, other applications in biology,
medicine, and system engineering that share the same
characteristics can benefit.
A. Benabdelkader ©UvA, 2002
References
• Personnel Home Page:
– http://carol.wins.uva.nl/~ammar
• Cooperative Information Management (CO-IM) Home Page:
– http://carol.wins.uva.nl/~netpeer/
• Matisse DBMS:
– http://www.matisse.com
• Related Publications:
–
A. Benabdelkader, H. Afsarmanesh, L. O. Hertzberger. MegaStore: Advanced Internet-based
Electronic Commerce Service for Music Industry. In proceedings of 11th IEEE International
Conference on Database and Expert Systems Applications - DEXA'2000, Pages 869-878, London Greenwich, United Kingdom, 2000.
–
A. Benabdelkader, H. Afsarmanesh, L. O. Hertzberger. The Virtual MegaStore System
Implementation. Technical Report CS-99-05, Faculty of Science, Research Institute Computer
Science, University of Amsterdam, 1999.
–
A. Benabdelkader, H. Afsarmanesh, L. O. Hertzberger. The Virtual MegaStore System
Architecture: Analysis and Design. Technical Report CS-99-04, Faculty of Science, Research
Institute Computer Science, University of Amsterdam, 1999.
A. Benabdelkader ©UvA, 2002