Transcript Sussex AstroGrid Science Workshop

March 29-30, 2007
Queen’s University Belfast
AstroGrid Science Workshop
Meeting Goals
Silvia Dalla
Workshop goals
• Aim to work with you, the early users of
AstroGrid, to demonstrate its capabilities and
the potential of the Virtual Observatory (VO)
• Introduce you to the elements of the system and
how to do science with it
• Look at a science problem in your research
area and use VO tools to tackle it
Workshop goals (contd)
• Collect your feedback
 What you like/dislike
 What needs to be improved
 Which data you need
 Which applications you want to see running in AG
Workshop Plan
• Agenda:
http://wiki.astrogrid.org/bin/view/Astrogrid/AgBelfastWorkshopMar07
• Today: background talks on AstroGrid and how the system
works (am). Initial workgroup set up (pm)
• Tomorrow: workgroups work on science cases. Report and
feedback Session (pm)
• Don’t miss the dinner (Today)
• Group Wiki pages to record progress
What you can do with AstroGrid
AstroGrid Science Team
Silvia Dalla, Eduardo Gonzalez-Solares,
Anita Richards, Jonathan Tedds,
Nicholas Walton and Dugan Witherick
Outline
1. Background: Virtual Observatories (VOs) and
AstroGrid
2. Example of non-VO solution to a data retrieval
and processing task: making a movie of solar
images
3. The AstroGrid solution
4. Introduction to AstroGrid components
What is the VO?
VO = collection of integrated astronomical
data archives and software tools that utilize
computer networks to create an environment
in which research can be conducted
(from www.encyclopedia.com)
Thanks to Ohishi Masatoshi (NAOJ) for the
pointer…
VOs worldwide
• Similar efforts now in 15
countries:
– UK, USA, Canada, France,
Germany, Italy, Holland, Japan,
Australia, India, China, Russia,
Hungary, South Korea, ESO,
Spain
• Active collaboration
among projects
– Standards, common demos
– International VO roadmap being
developed
– Regular telecons over 10
timezones
Formal collaboration:
International Virtual
Observatory Alliance (IVOA)
AstroGrid: the UK’s VO
• Consortium of several UK
Universities, funded by
PPARC to build a VO for
the UK.
• Part of the Euro-VO
www.euro-vo.org
• Astrogrid release 2007.1:
www.astrogrid.org/launch
• Data discovery, access to
data and applications.
AstroGrid Goals
• Enable science by:
– Improving the quality, easy, speed of on-line
astronomy
– Making integration and comparison of data from
diverse sources transparent
– Removing data access barriers to multiwavelength
analysis
– Enabling access and manipulation of large datasets
The need for a VO
• Data volume doubles every year
• Increase of size and multiplex capabilities of
new instruments
–
–
–
–
WFCAM: 100Gb/night (100Tb/yr)
VISTA: 300Gb/night (300Tb/yr)
e-MERLIN/ALMA: 1Tb/day
SDO: 1 Tb/day
• We will have Petabytes of data by 2010
The need for a VO
• Hubble UDF
• Million second exposure
• 6000x6000 pix
• 11.5 sq. arcmin
• 10,000 galaxies
The need for a VO
• SWIRE ELAIS N1
• 9 sq. degrees (~3000 UDF)
• (moon ~0.2 sq. deg.)
• ~ 600,000 objects
The need for a VO
• Solar Dynamics Observer
• Launch 2008
• >1 TB/day
• Over 5 years: 1.8 PB
Making a solar movie – non VO
Web interface
to database of
SOHO/EIT
observations
Making a solar movie – non VO
Download images
to local machine
Making a solar movie – non VO
IDL SolarSoft required
to calibrate images and
make a movie. Routines
available, however each
user rewrites code calling
them to produce a movie.
Making a solar movie – non VO:
weaknesses
• If user wishes to make a movie for a different time period,
the above steps need to be repeated – by hand
• Similarly for astronomy data if one is interested in many
objects
• The entire archive is not ‘visible’ to the user – only the
downloaded subset can be processed
• Scripting eg with Python, Perl etc is possible though
code is different for different archives – not easy for the
‘general’ user / for many datasets. This limits feasibility of
multi-wavelength, multi-instrument work
Making a solar movie – with AG
Solar Movie Maker science
workflow.
Other science workflows
available:
Redshift Maker, Colour
Cutter, Cone Search,
SWIRE images
Making a solar movie – with AG:
under the hood
• Access to
database of
observations – via
AG DSA (DataSet
Access) software
• Requests are sent
using ADQL
(Astronomy Data
Query Language),
similar to SQL
AG Workbench/
workflow engine
DSA
Database of
observations
(at archive)
Input: ADQL query
Output: table of observations
satisfying the query, in
VOTable format
Making a solar movie – with AG:
under the hood: CEA
• Capability to call a
set of processing
routines that do
operations on data,
or a model (apps in
any programming
language can be
wrapped) – CEA
Application
AG workbench/
workflow engine
CEA
Application. Eg
software that
processes data,
model etc
(on apps server)
Input: CEA application
input parameters
Output: whatever the output
of the application is, delivered
to user’s Myspace
Making a solar movie – with AG:
under the hood: Workflow
• Workflow capability so that
queries to datasets and calls
to applications can be
managed
• Jobs are run remotely and
asynchronously
• Queries and workflows can
be re-used and shared
• AstroGrid is currently the
only VO project with a
workflow capability
AstroGrid: Workbench
• Workbench
gives access to
AG services
• Data discovery:
AstroScope and
HelioScope
• Task Launcher
– send a query
to a database or
launch a
application
AstroGrid: Workbench- contd
• Run a ready
made science
workflow
• Build your own
workflow
• View your files in
MySpace
AstroGrid: MySpace
• Virtual disk space where
you can store results,
temporary files, and new
things like query files
and workflow files, so
you can adjust and rerun jobs on a later day.
• Visible from any
computer.
Finding Information : the
Registry
• How do you find the data
you require?
• How do you decide which
resource (data, application,
information, disk, …) to
use?
• The registries are the yellow
pages for astronomical
resources
• All VO registries harvesting
each other: thus querying
any one returns full list of
globally held resources.
VOTable: an interchange format
• XML standard for the interchange of data represented
as a set of tables
• A table is an unordered set of rows, as specified in the
table metadata
• Each row is a sequence of table cells, each of them
containing a primitive data type or an array of such
primitives
• It bridges two ways to express structured data: XML
and FITS
• Use of Universal Column Descriptors (UCDs) to
express the content of each parameter.
<?xml version="1.0"?>
<VOTABLE version="1.1" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:noNamespaceSchemaLocation="http://www.ivoa.net/xml/VOTable/VOTable/v1.1">
<COOSYS ID="J2000" equinox="J2000." epoch="J2000." system="eq_FK5"/>
<RESOURCE name="myFavouriteGalaxies">
<TABLE name="results">
<DESCRIPTION>Velocities and Distance estimations</DESCRIPTION>
<PARAM name="Telescope" datatype="float" ucd="phys.size;instr.tel"
unit="m" value="3.6"/>
<FIELD name="RA" ID="col1" ucd="pos.eq.ra;meta.main" ref="J2000"
datatype="float" width="6" precision="2" unit="deg"/>
<FIELD name="Dec" ID="col2" "pos.eq.dec;meta.main" ref="J2000"
datatype="float" width="6" precision="2" unit="deg"/>
<FIELD name="Name" ID="col3" ucd="meta.id;meta.main"
datatype="char" arraysize="8*"/>
<FIELD name="RVel" ID="col4" ucd="src.veloc.hc" datatype="int"
width="5" unit="km/s"/>
<FIELD name="e_RVel" ID="col5" ucd="stat.error;src.veloc.hc"
datatype="int" width="3" unit="km/s"/>
<FIELD name="R" ID="col6" ucd="phys.distance" datatype="float"
width="4" precision="1" unit="Mpc">
<DESCRIPTION>Distance of Galaxy, assuming H=75km/s/Mpc</DESCRIPTION>
</FIELD>
<DATA>
<TABLEDATA>
<TR>
<TD>010.68</TD><TD>+41.27</TD><TD>N 224</TD><TD>-297</TD><TD>5</TD><TD>0.7</TD>
</TR>
<TR>
<TD>287.43</TD><TD>-63.85</TD><TD>N
6744</TD><TD>839</TD><TD>6</TD><TD>10.4</TD>
</TR>
<TR>
<TD>023.48</TD><TD>+30.66</TD><TD>N 598</TD><TD>-182</TD><TD>3</TD><TD>0.7</TD>
</TR>
</TABLEDATA>
</DATA>
</TABLE>
</RESOURCE>
</VOTABLE>
VOTable
Example
Table Metadata
Table data
(XML,
FITS)
Visualising results: Topcat
Visualising results: Topcat
Astrogrid Help Pages
www.astrogrid.org
->Science user
->Examples
->Documentation
Summary
• Workshop is an opportunity to see AG in action
• Current capability can be of use to a diverse
range of science problems
• New cases – perhaps emerging from this
workshop – may demand new science services:
input is welcome from you!