Transcript Virtual Observatory-India - Chinese Virtual Observatory

Hello!
Virtual Observatory - India
Ajit Kembhavi
Inter-University Centre for Astronomy and Astrophysics
Pune, India
A collaboration between
IUCAA and PSPL,
with a grant from the
Ministry of
Communications and
Information Technology
The City of Pune
IUCAA
Persistent Systems
Pvt. Ltd., Pune
The Alliance
Members of the IVOA
Interactions
The Data Avalanche
• An increasing number of telescopes and large
area detectors are producing immense amounts of
data through imaging and spectroscopic surveys.
• Terabytes of data are now available, and Petabytes
will soon be available from frequent all sky
imaging.
• Vast databases are also being produced through
simulations.
Data Storage and Retrieval
The Astronomer
Vermeer
1632-1675
The Library of Alexandria
3rd Century BC
Star Positions
Stars in the Milky Way
The Hertzsprung-Russell Diagram
Wavelength Coverage
• The data spans the electromagnetic spectrum
from the radio to the gamma-ray region.
• Obtaining, analysing and interpreting the
data in different wavebands involves highly
specialised instruments and techniques.
• The astronomer needs new tools for using
this wealth of data in multiwavelength
studies.
Virtual Observatories
• Manage vast data resources and provide
these on-line to astronomers and other
users.
• Provide tools for data analysis, visualization
and mining.
• Develop interoperability concepts to make
different databases seamless.
• Empower astronomers by providing
sophisticated query and computational
tools, and computing grids for producing
new science.
VO Schema
IVOA Technology Initiatives
The IVOA has identified six major
technical initiatives to fulfill the scientific
goal of the VO concept.
IVOA-LISTS
• REGISTRIES: These collect metadata about
about data resources and information
services into a queryable database. The
registry is distributed. A variety of industry
standards are being investigated.
• DATA MODELS: This initiative aims to
define the common elements of astronomical
data structures and to provide a framework to
describe their relationships.
• UNIFORM CONTENT DESCRIPTORS:
These will provide the common language for
for metadata definitions for the VO.
• DATA ACCESS LAYER: This provides a
standardized access mechanisms to
distributed data objects. Initial prototypes are
a Cone Search Protocol and a simple Image
Access Protocol.
• VO QUERY LANGUAGE: This will provide
a standard query language which will go
beyond the limitations of SQL.
• VOTable: This is an XML mark-up standard
or astronomical tables.
VO-India Projects
•
•
•
•
•
•
•
•
VOTable Parser in C++; Streaming data
Web based FITS browser
VOPlot
Plots and Statistics for VIZIER and Aladin
GALics data bases
Data mirror sites
Scientific data mining
Applications beyond astronomy
VOTable
• This is a new data exchange standard
produced through efforts led by
Francois Ochsenbien of CDS,
Strasbourg and Roy Williams of
Caltech.
• VOTable is in XML format. Physical
quantities come with sophisticated
semantic information.
VOTable
• The format enables computers to easily parse
the information and communicate it to other
computers.
• Federation and joining of information become
possible and Grid computing is easier.
• VOTable parsers have been developed in Perl,
Java and C++.
• Enhancements and extensions are being
considered.
Streaming Parser
Non-streaming Parser
Project Design
VTable
Metadata
Link Collection
Field Collection
Field
Table Data
Link
Link Collection
Link
Values
minimum
Row Collection
maximum
Row
Option Collection
Column Collection
Options
The VOPLOT Collaboration
Visualization and simple
statistics of catalogue data.
Integration with sky atlases.
The VOPlot Collaboration
• The idea for VOPlot was developed through a
discussion between VO-I and CDS.
• VOPlot was developed by PSPL and IUCAA and
was integrated into Vizier with the help of CDS.
• It was then decided to integrate VOPlot with
Aladin, an interface was agreed on, and the
integration is now complete.
• The collaboration was carried through a few short
visits, phone calls and E-mail.
• Main persons: Sonali Kale et al (PSPL), Pierre
Fernique and Francois Ochsenbien. (CDS).
VOPlot
FITS Manager
View, create and add to FITS files
Convert to other formats
Fits-manager
VOTable Java Streaming Writer
Acts on a data array in memory
to convert it to the VOTable form,
Which is streamed row by row
To an output file. Very large
VOTables can be written without
excessive memory.
VOTable-Java
Fast Computing
Four Alpha Server ES-45 machines
Each with 4 processors Alpha 21264C
RAM 3 x 8 Gb + 12 Gb
Fast, Low latency interconnect
Memory channel Architecture
(MCA)
1 Tera-byte SCSCI storage
Trucluster clustering environment
(Tru64 Unix, DecMPI, openMP)
High Volume Storage
Raid 5, 4 Terabyte
Thank You
IUCAA HPC Facility
Hercules
Co-proposed by :
Ajit Kembhavi
T. Padmnabhan
Tarun Souradeep
• Four Alpha Server ES-45 machines
HPC Team :
Sarah Ponthratnam
Sunu Engineer
Rajesh Nayak
Anand Sengupta
• Each with 4 processors Alpha (21264C)
•1.25 GHz clock speed
• Cache on chip: 64 Kb –I, 64 Kb-D
• Cache : 16 Mb ECC DDR
• RAM 3 x 8 Gb + 12 Gb
• Fast, Low latency interconnect
> 30 G flops
Preliminary HPL
benchmark
• Memory channel Architecture (MCA)
• High volume Storage
ES-45
• 1 Tera-byte SCSCI Specfp2000: 1327
Linpack(Tru64
1000x1000:
•Trucluster clustering environment
Unix, 6847
DecMPI, openMP)
Virtual Observatory - India
Persistent Systems
IUCAA
Caltech, Fermilab, JHU, NASA/HEARC,
Microsoft, NCSA/UIUC, NOAO, NRAO,
Raytheon ITS, SDSC/UCSD, SAO/CXC,
STScI, UPenn, UPitts/CMU, UWis, USC,
USNO, USRA, CVO
NVO-People
Terapix
Jodrell Bank
Registry and DIS
AVO Prototype
Demo
Astrogrid:
Astronomy
Catalogue
Extractor
AVO:
Aladin+SED
VO-India:VOPlot
VOTable Data
The data part in a VOTable may be represented using one of
three different formats:
– FITS : VOTable can be used either to encapsulate FITS
files, or to re-encode the metadata.
– BINARY : Supported for efficiency and ease of
programming, no FITS library is required, and the
streaming paradigm is supported.
– TABLEDATA : Pure XML format for small tables.
CVO Collaborations
• There are three major projects at the CVO involving
collaborations with other VO.
• CVO is collaborating with the German Astrophysical VO
to incorporate ROSAT X-ray data and catalogues into the
CVO system.
• CVO is collaborating with the Australian VO.to
incorporate 2Qz and 2DF galaxy spectra into the CVO
database.
• CVO is an associate member of NVO and is have put in
place some components of the NVO galaxy morphology
demo.
Science Initiatives
• Many IVOA projects have active Science Working
Groups consisting of astronomers from a broad
cross-section of the community representing all
wavelengths.
• The focus here is to develop a clear perception of
the scientific requirements of a VO.
• Projects within the working groups will develop
new capabilities for VO based analysis.
• This will enable the community to create new
research programs and to publish their data and
research in a more pervasive and scientifically
useful manner.
Australian –VO Collaborations
• The distributed volume renderer (dvr) software, is a
tool for rendering large volumetric data sets using the
combined memory and processing resources of
Beowulf like clusters.
• A collaboration between the Melbourne site of Aus-VO
and AstroGrid aims to develop the existing dvr
software into a grid-based volume rendering service.
• Users will be able to select FITS-format cubes from a
number of "Data Centres",have the data transferred to
a chosen rendering cluster, and then proceed to
visualise the volume of data remotely (See Demo).
C++ VOTable Parser
Motivation:
–
Provide a library for API based access to VOTable
files.
–
APIs can be directly used to develop VOTable
applications without having to do raw VOTable
processing.
C++ VOTable Parser
Salient Features:
•
Implemented as a wrapper over XALAN-C++.
•
XALAN-C++ is a robust implementation of the W3C
recommendations for
XSL Transformations (XSLT) and the
XML Path language (XPath).
•
XPath queries can be used to access the VOTable
data.
C++ VOTable Parser
•
Initial version
- Released on May 31st , 2002.
- Support only for reading of tables.
- Support only for pure-XML TABLEDATA and not for BINARY or
FITS data streams.
- Runs on Windows NT 4.0, Windows 2000 and
RedHat Linux 7.1.
•
Future enhancements
- Can be incorporated quickly and efficiently.
Parser Design
Class Details
•
VTable: In memory representation of a single <TABLE>
from the <RESOURCE> element in VOTable
•
•
•
•
•
•
TableMetaData: Contains MetaData (Fields, Links and Description)
Resource: Represents the <RESOURCE> element in the VOTable.
TableData: Contains Rows
Field: Representation of <FIELD> from VOTable
Row: Representation of <TR> from VOTable
Column: Representation of <TD> from VOTable
Parser Design
API – Typical Operations
•
File Level I/O Routines
– Open VOTable file
– Close VOTable file
•
Table I/O Operations
– Get number of rows
– Get number of columns
– Get column(field) information (column name, column number,
etc.)
– Accessing table data
Parser Implementation
•
•
•
•
•
•
Development on Windows NT 4.0 platform using VC++.
Ported to RedHat Linux 7.1/gcc-2.96 with zero effort.
18 C++ classes representing various elements of the
VOTable format.
8500 lines of C++ code written for V1.1 release
Project start date: April 7th 2002
V1.1 Release: May 31st 2002
Current status: V1.2 design in progress
What is in Release V1.1




Parser to serve as a building block for developing
VOTable based applications.
Can be easily used by users of CFITSIO library.
Supports powerful XPath queries against
VOTable files.
The first version of the VO Table parser can now
be downloaded:
http://vo.iucaa.ernet.in/~voi/html/infopage.html
VOTable Parser Demo

Serves as a tutorial to help understand the basic
APIs provided by the VOTable parser.

Demonstrates how to access the data and
metadata elements of a VOTable file.
Future Work
•
•
•
•
•
Develop APIs for writing data in VOTable
format.
Develop APIs for supporting IMAGE data and
FITS files in VOTable.
Enhance existing API set to allow more
elaborate and flexible operations on VOTable
files.
Support future VOTable versions.
Develop applications for conversion between
FITS and VOTable formats.
References
•
•
•
•
The first version of the C++ parser can now be
downloaded from the VO-India website
http://vo.iucaa.ernet.in/~voi
VOTable Details:
http://vizier.u-strasbg.fr/doc/VOTable/
XALAN
http://xml.apache.org/xalan-c/index.html
XPATH
http://www.w3.org/TR/xpath