Transcript AstroGrid and OGSA-DAI

A PPARC funded project
AstroGrid and OGSA-DAI:
Early experiences
Kona Andrews
Guy Rixon
Institute of Astronomy
University of Cambridge
Data Set Access: The DataCenter
DataCenter component
Web
Services
Data
Access
Plug-in
Data
Tomcat / Axis
OGSA-DAI UG Meeting
AstroGrid and OGSA-DAI: Early experiences
07-04-2004
DataCenter JDBC plug-in
DataCenter component
Web
Services
JDBC
DBMS
Tomcat / Axis
OGSA-DAI UG Meeting
AstroGrid and OGSA-DAI: Early experiences
07-04-2004
DataCenter custom plug-in
DataCenter component
Web
Services
File querier
FITS
files
Tomcat / Axis
OGSA-DAI UG Meeting
AstroGrid and OGSA-DAI: Early experiences
07-04-2004
DataCenter proxy plug-in
DataCenter component
Web
Services
Service
proxy
Vizier
Tomcat / Axis
OGSA-DAI UG Meeting
AstroGrid and OGSA-DAI: Early experiences
07-04-2004
What we imagined
DataCenter
Component
Web
Services
Grid
Service
Ports
OGSA-DAI
client
plug-in
OGSADAI
Tomcat / Axis
OGSA-DAI UG Meeting
AstroGrid and OGSA-DAI: Early experiences
Tomcat / OGSA
07-04-2004
Problem
DataCenter
Component
Web
Services
Grid
Service
Ports
OGSA-DAI
client
plug-in
OGSADAI
Tomcat / Axis
OGSA-DAI UG Meeting
AstroGrid and OGSA-DAI: Early experiences
Tomcat / OGSA
07-04-2004
Problem
DataCenter
Component
Web
Services
Grid
Service
Ports
OGSA-DAI
client
plug-in
OGSADAI
Tomcat / Axis
OGSA-DAI UG Meeting
AstroGrid and OGSA-DAI: Early experiences
Tomcat / OGSA
07-04-2004
Nasty solution
DataCenter
Component
Web
Services
OGSA-DAI
“client”
plug-in
Separate JVM
OGSA-DAI
client
Grid
Ports
OGSADAI
Tomcat / Axis
OGSA-DAI UG Meeting
AstroGrid and OGSA-DAI: Early experiences
Tomcat / OGSA
07-04-2004
Current use of OGSA-DAI activities
Data
Center
OGSA-DAI
Postgres
DBMS
ADQL
OGSA-DAI UG Meeting
AstroGrid and OGSA-DAI: Early experiences
07-04-2004
Current use of OGSA-DAI activities
Data
Center
ADQL
OGSA-DAI UG Meeting
SQL
OGSA-DAI
ADQL
to
SQL
Postgres
DBMS
AstroGrid and OGSA-DAI: Early experiences
07-04-2004
Current use of OGSA-DAI activities
Data
Center
ADQL
OGSA-DAI UG Meeting
SQL
OGSA-DAI
sqlQueryStatement
ADQL
to
SQL
SQL
Postgres
DBMS
AstroGrid and OGSA-DAI: Early experiences
07-04-2004
Current use of OGSA-DAI activities
Data
Center
ADQL
SQL
OGSA-DAI
sqlQueryStatement
ADQL
to
SQL
SQL
Postgres
DBMS
(to XML RowSet)
Result
rows
OGSA-DAI UG Meeting
AstroGrid and OGSA-DAI: Early experiences
07-04-2004
Current use of OGSA-DAI activities
Data
Center
ADQL
ADQL
to
SQL
SQL
OGSA-DAI
sqlQueryStatement
deliverFromURL
SQL
Postgres
DBMS
(to XML RowSet)
Result
rows
OGSA-DAI UG Meeting
AstroGrid and OGSA-DAI: Early experiences
07-04-2004
Current use of OGSA-DAI activities
Data
Center
ADQL
ADQL
to
SQL
SQL
OGSA-DAI
sqlQueryStatement
deliverFromURL
SQL
Postgres
DBMS
(to XML RowSet)
xslTransform
Result
rows
VOTable
OGSA-DAI UG Meeting
AstroGrid and OGSA-DAI: Early experiences
07-04-2004
Current use of OGSA-DAI activities
Data
Center
ADQL
ADQL
to
SQL
SQL
OGSA-DAI
sqlQueryStatement
deliverFromURL
SQL
Postgres
DBMS
(to XML RowSet)
xslTransform
Result
rows
VOTable
deliverToFile /
deliverToGFTP
OGSA-DAI UG Meeting
AstroGrid and OGSA-DAI: Early experiences
07-04-2004
Current use of OGSA-DAI activities
Data
Center
ADQL
ADQL
to
SQL
SQL
OGSA-DAI
sqlQueryStatement
deliverFromURL
SQL
Postgres
DBMS
(to XML RowSet)
xslTransform
Result
rows
VOTable
Deliver
results
OGSA-DAI UG Meeting
deliverToFile /
deliverToGFTP
AstroGrid and OGSA-DAI: Early experiences
07-04-2004
Comments
Astronomers often need LARGE result sets
Thousands to millions of rows
Memory usage is a big issue
OGSA-DAI 3.1 much better than 3.0
xslTransform activity hogging memory
OGSA-DAI UG Meeting
AstroGrid and OGSA-DAI: Early experiences
07-04-2004
OGSA-DAI speed penalty
40
30
20
10
0
1e3
1e4
1e5
2e5
4e5
1e6
2e6
Number of result rows

Rough figures only

Measured without xslTransform activity

Will likely vary with available memory
OGSA-DAI UG Meeting
AstroGrid and OGSA-DAI: Early experiences
07-04-2004
Planned extension: User tables
DBMS
Data
Center
Query
OGSA-DAI UG Meeting
OGSADAI
R/O
Datasets
Query results
AstroGrid and OGSA-DAI: Early experiences
07-04-2004
Planned extension: User tables
DBMS
Data
Center
OGSADAI
GDS to GDS
transfer
R/O
Datasets
Bulk load
OGSADAI
Query
OGSA-DAI UG Meeting
Query results
AstroGrid and OGSA-DAI: Early experiences
User
tables
Data flow
07-04-2004
Planned extension: User tables
DBMS
Data
Center
Data
Center
OGSADAI
DBMS
Table
Manager
Query
OGSA-DAI UG Meeting
R/O
Datasets
Query results
AstroGrid and OGSA-DAI: Early experiences
OGSADAI
User
tables
Data flow
07-04-2004
Planned extension: User tables
DBMS
Data
Center
Data
Center
Client
OGSADAI
R/O
Datasets
DBMS
Table
Manager
OGSADAI
User
tables
User data
Query
OGSA-DAI UG Meeting
Query results
AstroGrid and OGSA-DAI: Early experiences
Data flow
07-04-2004
Comments
Will be relying on sqlBulkLoadRowSet (!)
Table management facilities currently thin
Not using X.509 auth at user granularity
Still require W/S-based access to G/S
OGSA-DAI UG Meeting
AstroGrid and OGSA-DAI: Early experiences
07-04-2004
Our needs
Ability to handle very large result sets
- Don’t rely on loading whole result set into memory!
Suite of DB/table management activities
- List tables in DB
- Describe table column types / space usage
- Last access/modification time
- Etc etc etc
Ability to create and delete indices on tables
Ultra-simple installation procedures
OGSA-DAI UG Meeting
AstroGrid and OGSA-DAI: Early experiences
07-04-2004
The End
www.astrogrid.org
OGSA-DAI UG Meeting
AstroGrid and OGSA-DAI: Early experiences
07-04-2004