Transcript Astrogrid Architecture Review

Astrogrid Architecture Review
Presenters:
Elizabeth Auden
David Giaretta
Tony Linde
University of Leicester
Friday, 17 May 2002
Astrogrid Architecture Review
Clive Page
Guy Rixon
University of Leicester
17 May 2002
Architecture Review: Agenda
09:50
10:00
10:10
Introduction
Architecture Challenges
Core Use Cases
Tony Linde
Elizabeth Auden
Elizabeth Auden
10:20
11:00
11:15
11:45
12:10
Sequence Diagrams
Tea and Coffee
Domain Model
IAA
Workflow
Elizabeth Auden
12:25
12:45
13:30
14:00
Databases
Lunch (served in room)
Component Model
Technology
Clive Page
14:30
What Next?: Architecture,
Phase B, WP5, Training
Finish
Tony Linde
15:00
Astrogrid Architecture Review
David Giaretta
Guy Rixon
Tony Linde
Tony Linde
Tony Linde
University of Leicester
17 May 2002
Architecture Challenges
6 Architecture
Challenges:
 All use cases:
IAA, interoperability,
•
•
•
compute resources
 Most use cases:
Catalogue queries
 Some use cases:
Software interfaces
•
•
•
Astrogrid Architecture Review
Certification:
Identification, authorization, authentication
Catalogues:
Complex queries, searches, joins, correlation
Distributed Catalogues:
Distributed queries, distributed joins
Compute Resources:
Upload/execute code, scheduler, estimator
Interfaces:
Astronomical packages, client software
Interoperability:
Data, metadata
University of Leicester
17 May 2002
Core Use Cases
USE CASE
CHALLENGE
DISCIPLINE
•
Brown Dwarf Science Problem
Catalogue search and join
Optical
•
Deep Field Survey Correlation
Complex query, correlation
Radio, Optical
•
Galaxy Clustering
Catalogue search, upload code
Optical
•
High Z Quasars
Complex query and join
Optical, X-ray
•
Low Surface Brightness
Catalogue search, upload code
Optical
•
Magnetic Storm Onset
Complex queries, models
STP
•
Solar Coronal Waves
Complex queries, upload code
Solar
•
Solar / STP Event Coincidence
Distributed queries, join
Solar / STP
•
Solar / Stellar Flare Redshift
Distributed queries, upload code
Solar / Astro
•
Supernova Galaxy Environment
Catalogue search, upload code
Optical / IR
Astrogrid Architecture Review
University of Leicester
17 May 2002
Sequence Diagrams
3 Sample
•
Low Surface Brightness:
Optical astrophysics:
An astronomer wants to identify faint galaxies and
extended objects by comparing source flux with
background flux.
•
Solar Coronal Waves
Solar physics:
A solar physicist wants to examine solar activity in
the hours preceding coronal waves and flares to
identify causes of solar events.
•
Magnetic Storm Onset:
Solar terrestrial physics:
A solar terrestrial physicist wants to study the
morphology of the Earth's magnetosphere tail
during the onset of geomagnetic storms.
Use Cases:
 Low Surface
Brightness
 Solar Coronal
Waves
 Magnetic Storm
Onset
Astrogrid Architecture Review
University of Leicester
17 May 2002
Detail: Low Surface Brightness
Astronomer
Resource
Catalogue
Image
Catalogue
Compute
Resource
My Space
Search for image catalogues
with magnitude / depth
metadata
Return list of
catalogues
Search for images of point sources / bright
structures
Upload image data
Upload algorithm to remove point sources / bright structures
Upload smoothing algorithm
Upload algorithm to find extended sources
Return extended source list
Astrogrid Architecture Review
University of Leicester
17 May 2002
Detail: Solar Coronal Waves
Astronomer
Resource
Catalogue
Search for flare
catalogues
Catalogue
Source
Image /
Spectra
Compute
Resource
My Space
Return catalogue list
Complex Query: search flare
catalogue for time intervals given
an intensity inequality
Return time intervals
Search for catalogues with
spectral / image data for
intense flare time intervals
Return catalogue list
Search catalogue for spectra and images
based on time interval and location criteria
Upload images
and spectra
Upload user image difference algorithm
Upload user pattern-matching algorithm
Return list of difference images matching pattern
Astrogrid Architecture Review
Upload
images
University of Leicester
17 May 2002
Detail: Magnetic Storm Onset
Physicist
Resource
Catalogue
DST
Catalogue
Mag. Sphr.
Model
Spacecraft
Metadata
Instrument
Data
My
Space
Search for DST
catalogues
Return list of
DST catalogues
Query for time intervals
where DST index > X
Return time intervals
Search for models
of magnetosphere
Return list of
models
Search for
spacecraft with
magnetometers
Return list of
spacecraft
Complex Query: find regions observed during
time intervals while magnetometer is turned on
Return regions; tool converts coordinate system
Get magnetometer data by region and time interval
Run data through
magnetosphere model
Return result list
Astrogrid Architecture Review
University of Leicester
17 May 2002
Domain Model
Astrogrid Architecture Review
University of Leicester
17 May 2002
Low Surface Brightness
Together Sequence Diagram:
Astrogrid Architecture Review
University of Leicester
17 May 2002
Low Surface Brightness
Together Collaboration Diagram:
Astrogrid Architecture Review
University of Leicester
17 May 2002
Solar Coronal Waves
Together Sequence Diagram:
Astrogrid Architecture Review
University of Leicester
17 May 2002
Solar Coronal Waves
Together Collaboration Diagram:
Astrogrid Architecture Review
University of Leicester
17 May 2002
Magnetic Storm Onset
Together Sequence Diagram:
Astrogrid Architecture Review
University of Leicester
17 May 2002
Magnetic Storm Onset
Together Collaboration Diagram:
Astrogrid Architecture Review
University of Leicester
17 May 2002
Architecture Review: IAA
Guy Rixon
• Identification
• Authorization
• Identification
Astrogrid Architecture Review
University of Leicester
17 May 2002
Architecture Review: Workflow
Workflow:
7 workflow scenarios
illustrate interactions
between Astrogrid,
administrators, and
users.
Astrogrid Architecture Review
•
Create Action
•
Job Admin
•
Orchestrate Job
•
Program Admin
•
Resource ACL
•
Resource Admin
•
Run Job
University of Leicester
17 May 2002
Workflow: Create Action
Astrogrid Architecture Review
University of Leicester
17 May 2002
Workflow: Job Admin
Astrogrid Architecture Review
University of Leicester
17 May 2002
Workflow: Orchestrate Job
Astrogrid Architecture Review
University of Leicester
17 May 2002
Workflow: Program Admin
Astrogrid Architecture Review
University of Leicester
17 May 2002
Workflow: Resource ACL
Astrogrid Architecture Review
University of Leicester
17 May 2002
Workflow: Resource Admin
Astrogrid Architecture Review
University of Leicester
17 May 2002
Workflow: Run Job
Astrogrid Architecture Review
University of Leicester
17 May 2002
Database Technology
Database
Techonology:
Internet
DBMS
•
Aims and Elements
•
Queries
•
Resource Discovery
•
Interoperability
•
DBMS
•
Sky Indexing
Astrogrid
Applications
Astrogrid Architecture Review
University of Leicester
17 May 2002
Database Tech: Aims / Elements
Assumed Astrogrid Aims
•To federate existing data archives
•To provide data mining facilities (anything that can't be done in-situ)
•To devise a scaleable infrastructure for archives suitable for future projects like
WFCAM, VISTA...
Assumed Architectural Elements
•Data Portal - provides user interface and intelligence, knows about resource registry
at least.
•Resource Registry - replicated directory of archives - level of detail under debate.
•Data Warehouse - provides cpu power and disc space for operations not feasible
elsewhere.
Expect more than one of each of these, many examples will be co-located with main
archive sites.
Astrogrid Architecture Review
University of Leicester
17 May 2002
Database Tech: Queries
Types of Query
•Positional queries - specify (RA, DEC, radius) - or several of them. May query any of:
•Source catalogues
•Image repositories
•Observatory archives
•Bibliographical archives
•Non-positional queries
•Cross-matching sources in two (or more) catalogues - fuzzy join is basic step for
many data mining operations.
•Selection from tables by reference to properties - may require sequential scan
•Statistical operations on tables - ditto
•Data mining queries requiring special algorithms (clustering, time-series, density
functions, etc)
Astrogrid Architecture Review
University of Leicester
17 May 2002
Database Tech: Resource Discovery
Astrogrid, AVO, NVO: agreement on need for unique but replicated Astronomical
Resource Registry. NVO seems to be thinking of a human-managed registry. Could the
registry be updated by a robot which uses WSDL descriptions of each archive?
• UDDI has some of the right properties, but very business-oriented.
• Level of detail subject of active debate.
Possible Registry Information:
• Name of service
• URL
• Physical location
• Contact details of responsible person
• Web interfaces supported (CGI, ASU, SOAP, WSDL, ...)
• Holding type (source catalogues, images, observing logs, raw data, bibliographic info., …)
• Waveband (radio, IR, optical, UV, X-ray, ...)
• Sky coverage (limits in Dec or Galactic latitude, or details of each pointing?)
• Access restrictions (by date of observation, location of user, max download size)
• Spatial resolution (images), or positional error (catalogues)
• Epoch of observation
• Data volume
• Export formats supported (FITS, VOtable, CSV, PNG, ...)
Astrogrid Architecture Review
University of Leicester
17 May 2002
Database Tech: Interoperability
Implementation of resource discovery will rely on the adoption of common standards by
Astrogrid, AVO, NVO, and the others. The essential interfaces are mainly those to
databases.
• Standards for queries. SQL inadequacies include:
•Too batch-oriented, hard to save intermediate results to refine a query
•Hard to do positional queries
•Poor support for statistical functions
•Text output only in most cases - need binary, interfaces to visualisation packages, etc.
•Ignores or loses metadata.
• Astronomical Query Language:
• Develop our own? Interpreted by the portal or passed to the DBMS?
• Standards for results:
•VOtable for tabular results: v1.0 drafted.
•Need similar standards for images and bulk data: discussed on US-VO metadata lists.
• Standards for metadata: useful starts are FITS, HFWG, UCD, etc. Much more needed.
• Database Task Force developing Web Services wrappers around databases (using JDBC).
Astrogrid Architecture Review
University of Leicester
17 May 2002
Database Tech: DBMS
There are plans to evaluate several database management systems:
• MySQL - fast, basic, used by several astronomical archives sites already
• Postgres - object-relational, has R-tree indexing
• DB2 (IBM) - heavyweight, free/cheap for research use - may get help in testing
• Oracle - market leader, all possible features - may get help in testing
• Sybase - used by many astronomical data archive sites already
• Microsoft SQL Server - used by Sloan/JHU to replace Objectivity, easy to use;
however, available for Windows only.
Principal types of query to be evaluated:
• I/O intensive queries, e.g. scans of entire table, statistical queries.
• Index-intensive queries, e.g. join of two large catalogues (fuzzy join).
Other important features:
• Licence cost
• Ease of use
• Support for parallelism
• Easy upload of users' own data, easy download/interface with astronomical applications.
Astrogrid Architecture Review
University of Leicester
17 May 2002
Database Tech: Sky Indexing
Indexing the sky is surprisingly difficult to index in the 2-d plane, but it is even
harder to index spherical surface.
Two promising methods:
•Postgres and Oracle support R-tree indexes on rectangles - should evaluate.
•Alternatively can map 2-d surface to points on a line then use an ordinary Btree
•Suitable mapping functions include HTM and HEALPix, simpler ones
possible.
•Devised PCODE method which allows fuzzy join to be done efficiently.
Astrogrid Architecture Review
University of Leicester
17 May 2002
Interface Model
Astrogrid Architecture Review
University of Leicester
17 May 2002
Deployment Model
Astrogrid Architecture Review
University of Leicester
17 May 2002
Astrogrid Technology
•
Technology to implement Astrogrid challenges.
Astrogrid Architecture Review
University of Leicester
17 May 2002
What Next?
Astrogrid Architecture Review
•
Architecture
•
Phase B
•
WP 5
•
Training
University of Leicester
17 May 2002