Transcript No Slide Title - The Australian Virtual Observatory

Australian Virtual
Observatory
Pacific Rim Applications and Grid Middleware Assembly
The 4th Workshop
5th-6th June 2003 Monash University
David Barnes
School of Physics, The University of Melbourne
What is a Virtual Observatory?
• A Virtual Observatory (VO) is a distributed, uniform
interface to the data archives of the world’s major
astronomical observatories.
• A VO is explored with advanced data mining and
visualisation tools which exploit the unified interface
to enable cross-correlation and combined
processing of distributed and diverse datasets.
• VOs will rely on, and provide motivation for, the
development of national and international
computational and data grids.
Scientific motivation
• Understanding of astrophysical processes depends
on multi-wavelength observations and input from
theoretical models.
• As telescopes and instruments grow in complexity,
surveys generate massive databases which require
increasing expertise to comprehend.
• Theoretical modeling codes are growing in
sophistication to consume available compute time.
• Major advances in astrophysics will be enabled by
transparently cross-matching, cross-correlating and
inter-processing otherwise disparate data.
Sample multi-wavelength data for the galaxy IC5332 (Ryan-Weber)
Visible blue light - young hot stars
Infrared light - old cooler stars
H-alpha spectral line - star forming sites
HI spectral line - gas to form stars
HI velocity field - kinematics
HI velocity dispersion - gas
stability, parameters
Integrated HI spectrum total neutral gas mass,
distance from redshift
And this is just the data on
one object from three
Australian telescopes!
Fundamental VO Challenges
• Data description: multi-wavelength, multiresolution, multi-dimensional, multi-domain
(optical, radio, X-ray, …), world coordinate
systems, limited period ownership, …
• Data provision: distributed mass storage,
high-bandwidth networks, registries, …
• Data processing: high performance clusters
as grid nodes, data to code versus code to
data, mountains of legacy software!, …
• Interface: portals, visual data flow control,
analysis tools, display tools, …
Aus-VO structure 2003
• Phase A funded AUD 260K by a 2003 ARC grant:
– The University of Melbourne
– The University of Sydney
– CSIRO Australia Telescope National Facility
– Anglo-Australian Observatory
• Additional institutes participating w/o direct funding from the
ARC grant:
–
–
–
–
ANU, Mount Stromlo Observatory & APAC
CSIRO Mathematical and Information Sciences
University of Queensland
VPAC & GridBus (Melb)
• Lead investigator Rachel Webster (Melb)
• Project scientist David Barnes (Melb)
Aus-VO projects 2003
• Common format on-line archive projects:
– HIPASS catalog: HI Parkes All Sky Survey: neutral
Hydrogen spectral line survey, ~4,300 sources with 138
parameters and 1024-channel spectra
– SUMSS catalog: Sydney University Molonglo Sky Survey:
radio continuum survey at 843 MHz, >100,000 sources
– 2dFGRS QSO catalog: 2-degree Field Galaxy Redshift
Survey: optical spectra of >20,000 southern quasi-stellar
objects
– ATCA archive: Australia Telescope Compact Array archive:
all observations since 1988, circa 1.5 TB of more than
1,000 separate observing projects! Massive exercise in
describing data with metadata.
– MACHO archive: Massive Compact Halo Objects archive:
8yr lightcurves for >18M stars
Aus-VO projects 2003
• Server-based visualisation tools:
– client Java canvas for legacy software package
AIPS++ to draw on from a remote server (ATNF)
– grid-service implementation of distributed
volume rendering - remote data transferred to
remote cluster, with display canvas applet
supplied by coordinating portal
• Pipelines to enable on-line reprocessing of archived
raw or pre-processed telescope data:
– Molonglo Observatory Synthesis Telescope
• Interfaces: beta testers for the AstroGrid consortium
software
VO Interface & Portal
• Agreement with AstroGrid (UK e-Science project)
to be testers for their data publication and portal
creation code.
• Collecting the necessary resources and intend to
have an AstroGrid-based portal serving HIPASS
catalog data for demonstration at IAU General
Assembly in July 2003.
• Separately testing IBM Lotus Notes and Domino
Server for publication of astronomical catalogs.
Grid-based Visualisation
• ATNF will build a Java
PixelCanvas so that
AIPS++ visualisation
applications can be
deployed as WebService and GridService Java Applets
• AIPS++ is modern,
OpenSource software
for reducing (radio)
astronomy data, 1.6M
lines of code.
Grid-based Volume Rendering
• Agreement between Melbourne and AstroGrid to develop our
existing distributed-data volume rendering code into a fullyfledged Grid-Service. [see my talk at GridBus this Saturday]
• Challenge is to interactively render a multi-GB cube at the IAU
GA 2003, using GridFTP to transfer the data volume from a
remote data warehouse to a remote rendering cluster and
display and control the rendering from an applet.
Time to render 512x512 view of
1024x1024x1024 volume (seconds)
1000
100
10
1
0
10
20
number of nodes
30
40
The near future: data grids for Aus-VO
• Australian archives range from ~10 GB to
~10 TB in processed (reduced) size.
• providing just the processed images and
spectra on-line requires a distributed, highbandwidth network of data servers – that is,
a data grid.
• users may want some simple operations
such as smoothing or filtering, applied at the
data server. This is a virtual data grid.
The near future: compute grids for Aus-VO
• More complex operations may be applied
requiring significant processing:
– source detection and parameterisation
– reprocessing of raw or intermediate data
products with new calibration algorithms
– combined processing of raw, intermediate or
"final product" data from different archives
• These operations require a distributed, highbandwidth network of computational nodes
– that is, a compute grid.
2004 ARC LIEF grant
• 10 partners!
• more data archives on-line
• more tools developed with special
focus on server-based visualisation
• construction of the Australian
Astronomy Grid…
The Australian Astronomy Grid 2004
http://www.aus-vo.org