Transcript Progress in the Australian Virtual Observatory

Aus-VO:
Progress in the Australian
Virtual Observatory
Tara Murphy
Australia Telescope National Facility
Aus-VO: who we are


Science infrastructure effort, funded by ARC LIEF grants
The Aus-VO Partner Institutes are:
–
–
–
–
–
–
–
–
–
–

CSIRO Australian Telescope National Facility
The University of Melbourne
The University of Sydney
Anglo-Australian Observatory
Swinburne University of Technology
The University of Queensland
Monash University
The University of New South Wales
Victorian Partnership for Advanced Computing
ANU Mount Stromlo Observatory & Supercomputing Facility
Plus collaborators:
– CSIRO Mathematical and Information Sciences
– Australian Partnership for Advanced Computing
Tara Murphy
Australia Telescope National Facility
IVOA Small Projects Meeting, Pune 2004
APAC Astronomy Grid



Grid infrastructure effort
Funded by the Australian Partnership for
Advanced Computing (APAC)
Active members & partners are:
– Melbourne and Swinburne
– UNSW and Sydney
– ANU


Project Leader: David Barnes
Research Leader: Rachel Webster
Tara Murphy
Australia Telescope National Facility
IVOA Small Projects Meeting, Pune 2004
Aus-VO: our focus






Bring legacy astronomy archives on-line
Ensure future project compliance
Provide access to archived realizations of
simulations
Describe all data fully, and support a small and
well-chosen set of interoperability protocols
Develop tools and interfaces to find, acquire,
process and visualize data
Build national grid infrastructure to host the data,
tools and interfaces
Tara Murphy
Australia Telescope National Facility
IVOA Small Projects Meeting, Pune 2004
Aus-VO: what we do

Science content
–
–
–
–

providing data
providing software
metadata
registry content
Science services
–
–
–
–
–
source finding
coordinate transformations
catalogue matching
visualization
interfaces to legacy environments
Tara Murphy
Australia Telescope National Facility
IVOA Small Projects Meeting, Pune 2004
APAC Grid: what we do

Data grids
–
–
–
–

data store
data access protocols
authentication & authorisation
network facilities
Compute grids
– workflow
– job configuration, submission & monitoring
– code-to-data paradigms
Tara Murphy
Australia Telescope National Facility
IVOA Small Projects Meeting, Pune 2004
Aus-VO: current research

Data mining
– catalogue matching
– searches for new empirical relationships & rare objects

Legacy software integration
– using legacy code in VO services
– using VO services in legacy software environments
– transparent access to Web and Grid Services from the
desktop

Visualisation
– remote visualisation service (legacy s/w drawing to remote
“canvases” via Corba, SOAP, …)
– VOTable to VRML service
– volume rendering
Tara Murphy
Australia Telescope National Facility
IVOA Small Projects Meeting, Pune 2004
Aus-VO: projects 2004

Common format on-line archive projects:
– HIPASS catalog: HI Parkes All Sky Survey: neutral Hydrogen
spectral line survey, 4,300 sources with ~30 parameters and
1024-channel radio spectra
– SUMSS catalog: Sydney University Molonglo Sky Survey: radio
continuum survey at 843 MHz, 107,000 sources with ~15
parameters
– 2dFGRS QSO catalog: 2-degree Field Galaxy Redshift Survey:
optical spectra of >20,000 southern quasi-stellar objects
– ATCA archive: Australia Telescope Compact Array archive: all
observations since 1988, circa 1.5 TB of more than 1,000
separate observing projects! Substantial exercise in describing
data with metadata.
– MACHO, RAVE, Pulsar timing archives, Gemini archive and more
in 2004!
Tara Murphy
Australia Telescope National Facility
IVOA Small Projects Meeting, Pune 2004
Aus-VO: projects 2004



Server- and client-based visualisation tools:
– client canvas for legacy software package AIPS++ to draw
upon from a remote server.
– 3d visualisation of catalogues
Data reduction pipelines for:
– Molonglo Observatory Synthesis Telescope
– AT Compact Array archive
– Gemini, for Australian-used instruments
Portal for theoretical astrophysics jobs:
– configuration and execution
– monitoring
– result retrieval and initial analysis and display
Tara Murphy
Australia Telescope National Facility
IVOA Small Projects Meeting, Pune 2004
The Australia Telescope
Online Archive

Consists of:
1. an archive of all data observed with the
Australia Telescope Compact Array
2. a data reduction pipeline to process the data
for ‘quick and dirty’ images
3. a remote visualisation system to inspect the
images on the client machine

People:
–
Peter Lamb, Tara Murphy, Anil Chandra,
Praveena Tokachichu, Chris Owen
Tara Murphy
Australia Telescope National Facility
IVOA Small Projects Meeting, Pune 2004
1. The archive




All ATCA data since
1990
Total of 1.9 Tb covering
over 1000 projects
Hosted on a machine
with 2GB memory and
6TB of disk
Back end is a 20 node
Beowulf cluster
Tara Murphy
Australia Telescope National Facility
IVOA Small Projects Meeting, Pune 2004
1. The archive

Challenges:
– Unifying metadata to fully describe data
– Capturing the intentions of the
observers

determining the calibration strategy
– Dealing with the significant minority of
non-standard observations
Tara Murphy
Australia Telescope National Facility
IVOA Small Projects Meeting, Pune 2004
2. The pipeline




Automatically runs complete data
reduction process for ATCA data
Underlying work done by AIPS++
Methods exposed as web services
using Glish/Java
Uses metadata from archive to make
decisions about reduction
Tara Murphy
Australia Telescope National Facility
IVOA Small Projects Meeting, Pune 2004
2. The pipeline

Challenges:
– Dealing with legacy code
– Dealing with legacy data (via the archive)
– Getting the pipeline to make sensible
decisions about how to process the data
Tara Murphy
Australia Telescope National Facility
IVOA Small Projects Meeting, Pune 2004
3. Visualization system

A software system that allows scientists to view and analyse
astronomical data that are stored in remote locations.
User connects to
view data
RVS
Server
Dataset sent to
RVS
Visualisation Data
(Image) sent to
user
RVS response
Browse data
User
Tara Murphy
Australia Telescope National Facility
Data Centre
IVOA Small Projects Meeting, Pune 2004
3. Visualization system

Challenges:
– Performance
RVS Image data format
 Connection speeds between services

– Session Management
How to coordinate data centres and clients?
 Caching of images for future sessions

Tara Murphy
Australia Telescope National Facility
IVOA Small Projects Meeting, Pune 2004
ATOA Walkthrough
1.
2.
3.
4.
Query the archive
Query results
Automatic data reduction
Visualise data
Tara Murphy
Australia Telescope National Facility
IVOA Small Projects Meeting, Pune 2004
Aus-VO: links





www.aus-vo.org
www.atnf.csiro.au/vo
RVS will be public in ~2 months
The ATOA will be public in ~2 months
The prototype pipeline will be public in
~3 months
Tara Murphy
Australia Telescope National Facility
IVOA Small Projects Meeting, Pune 2004