Transcript TNG Long Term Archive

The prototype TNG Long-Term Archive
––
Interoperability
issues
andit is worth to
Struggling
to
convince
your
funding
agencies
that
Towards
an Italian Virtual
Joining
the
International
Virtual
withtothe
Italian
GRID
devoteinteractions
manpower and funds
coordinate
efforts
within an
international
VO Observatory?
activity, project
while they
are still not necessarily
Observatory
Alliance
convinced it is useful to develop archives
F.Pasian, R.Smareglia – INAF/O.A.Trieste
L.Benacchio – INAF/O.A.Padova
“TOWARD AN INTERNATIONAL VIRTUAL OBSERVATORY”
Garching, 10-14 June 2002
“Telescopio Nazionale Galileo” (TNG)
•
•
•
•
3.5 m Nasmyth telescope - NTT-based - La Palma
First light in 1998 - AaT since the beginning
Long integration / testing / commissioning phase
First generation instruments completed in 2000:
–
–
–
–
optical imager (OIG)
low-resolution imager/spectrograph (DOLORES)
high resolution spectrograph (SARG)
near-IR camera/spectrograph (NICS)
• ADOPT@TNG, DIMM, Meteo
• Assisted observations
Garching, 10-14 June 2002
“TOWARD AN INTERNATIONAL
VIRTUAL OBSERVATORY”
2
The TNG LTA pilot project
• A pilot project is currently being finalized, aiming at
demonstrating the feasibility of technical solutions
for the construction of the Long-Term Archive (LTA)
for the TNG  building a prototype LTA
• Interfaces to TNG “archiving at the telescope” (AaT)
• End of pilot phase foreseen to be end of June 2002
• Financed by CNAA, results to be submitted to INAF
– INAF board on computing, archives and networks to
produce a report in early July
Garching, 10-14 June 2002
“TOWARD AN INTERNATIONAL
VIRTUAL OBSERVATORY”
3
Scope of the pilot project
• Provide suggestions / plans / prototypes to:
– allow the long-term storage of scientific and technical data
from the TNG
– guarantee accessibility by the TNG Observatory staff and
by the scientific community to original and derived data
– test practically the feasibility of deriving value-added
information from the stored data
– provide tools supporting life cycle of observing proposals
– provide interoperability with a number of existing
international data centers and archives
– guarantee harmonization with other Italian projects related
to archiving of data of astrophysical interest
Garching, 10-14 June 2002
“TOWARD AN INTERNATIONAL
VIRTUAL OBSERVATORY”
4
Product Tree
Interoperability
TNG LoTAr
Prototype
Coordination
Data Interface
Document
name resolver
basic interoperability
recommendations
Archive structure
definition
Data Model
Final
Report
Database
Ingestion tools
Data store
National archives
Data mining
projects
Retrieval tools
Technical data
UIF
Science data
Request servicing
Network
Additional services
visualization
telemetry
science frames
RDBMS
on-line
OODBMS
quasi-on-line
calibration
parallel RDBMS
off-line
consistency checks
quality control
support to TAC
data mining
instr. trend analysis
public outreach
Garching, 10-14 June 2002
“TOWARD AN INTERNATIONAL
VIRTUAL OBSERVATORY”
5
Interoperability
• Direct participation in “Interoperability WG” of
OPTICON
• Prototype LTA uses standards (e.g. VOTable)
defined within “Interoperability WG”  aims at being
compatible with AVO development
• Catalogue of observations available at CDS for
basic-level interoperability with other archives
• More in future LTA
Garching, 10-14 June 2002
“TOWARD AN INTERNATIONAL
VIRTUAL OBSERVATORY”
6
Lessons (being) learned
• Data quality is an issue:
– inconsistencies in data (esp. scientific exposures)
– some information missing (documentation, observation
schedule, etc.)
– improved observing procedures are possible
– AI analysis of technical / engineering data identifies some
possible problems in T&I systems  data quality (see
G.Longo’s talk on Friday)
• An archive must be developed in synch with the
observatory:
– late development of LTA (even if prototype)  late discovery of
inconsistencies
Garching, 10-14 June 2002
“TOWARD AN INTERNATIONAL
VIRTUAL OBSERVATORY”
7
Coordination with Italian research Grid
• The Italian astronomical community will not build its
own AstroGrid
• It will rather coordinate with nation-wide initiatives to
foster a Grid infrastructure:
– eScience Grid
– IG-BIGEST
Garching, 10-14 June 2002
“TOWARD AN INTERNATIONAL
VIRTUAL OBSERVATORY”
8
The eScience Grid in Italy (1/2)
• Proposal submitted in October 2001 to Ministry of
Education University & Research (MIUR)
• Bid for ~15 M€ of available ~30 M€ funds; answer
soon
Title: “Enabling Platforms for High-Performance
Computational Grids Oriented to Scalable Virtual
Organizations”
Strategic Programme on Enabling IC Technologies
for Knowledge Society
Garching, 10-14 June 2002
“TOWARD AN INTERNATIONAL
VIRTUAL OBSERVATORY”
9
The eScience Grid in Italy (2/2)
• Objectives:
– R&D Grid technological development project (CNR&INFN)
– Research on optical networks (CNIT)
– Deployment of an Italian eScience Grid infrastructure
(INFN) for:
•
•
•
•
•
Astrophysics
Biology
Computational Chemistry
Geophysics
Earth Observation
– International integration by INFN (following requirements)
Garching, 10-14 June 2002
“TOWARD AN INTERNATIONAL
VIRTUAL OBSERVATORY”
10
Astrophysics in the Italian eScience Grid
• Four demonstrators:
– distributed computing
– accessing and processing VST data
– accessing archives and databases (inter alia the prototype
TNG LTA)
– accessing observational facilities as “data nodes”
• About 1 M€ (1/15 of the total MIUR request)
Garching, 10-14 June 2002
“TOWARD AN INTERNATIONAL
VIRTUAL OBSERVATORY”
11
The Italian Grid for Business, Industry, Government,
EScience&Technology IG-BIGEST
•
•
•
New initiative leveraging from MIUR proposal and aiming at
the common development of the IG-BIGEST
Italian Participation to the Expression of Interest for an
Integrated Project "ENABLING GRIDS AND E-SCIENCE IN
EUROPE" (EGEE) in the context of 6th EU Framework
All major Italian Scientific National Research Institutions,
Universities and Computing Centres reached general
consensus:
– to take part to the large European Initiative EGEE having CERN as
leading partner and INFN as the organization representing the
Italian Grid eScience comunity
– to have only one large Integrated Project aiming at the
development and deployment of the Grid Infrastructure in Europe
as Geant does for the Networking
Garching, 10-14 June 2002
“TOWARD AN INTERNATIONAL
VIRTUAL OBSERVATORY”
12
The participants and the Italian Grid eScience
infrastructure
•
•
High bandwidth network provided by Garr-G  Geant
High performance computing infrastructures provided by
National Research Institutions, Universities and SuperComputing Centers. Participants :
– INFN, CNR, INAF, INFM, INGV, ENEA……
– Computing Centers: CINECA, CILEA, CASPUR….
– Universities
• Università di Lecce, Università di Napoli, Università di Pisa,
Università di Roma Tor Vergata, Centro Fermi Roma1, Università
di Cosenza, Università di Genova, Università di Padova, Università
di Politecnico Torino, Università di Salerno, Università di Messina
•
Formal participation has been finalized
Garching, 10-14 June 2002
“TOWARD AN INTERNATIONAL
VIRTUAL OBSERVATORY”
13
Conclusions (and roadmap?)
• TNG LTA Pilot Project has coordination duties (of
national archives and with international activities)
• TNG prototype LTA is VO-compliant; so will be future
archives (TNG LTA, VST, LBC, etc.)
• INAF board on computing, archives and networks
will provide recommendations
• The goal is to have the Italian community joining
AVO Phase-B
Garching, 10-14 June 2002
“TOWARD AN INTERNATIONAL
VIRTUAL OBSERVATORY”
14
Message Exchange Network
VERIFY
WSS-std instr.
IWS
TDB
TAG
non-std instr.
Staging
Disk
External
Disk
Data Network
DTIN
DTOUT
A
a
T
on-line
archive
database
User Disk
Data store
user application
U
I
F
DTFLUSH
AS
off-line archives CDs
WR
DTWRITE
commands/messages
telemetry
scientific data
data descriptors
Garching, 10-14 June 2002
AaT process
observer’s CDs
on-line
Storage
storage
External processes
off-line storage
“TOWARD AN INTERNATIONAL
VIRTUAL OBSERVATORY”
15
SA_INS
SA_DET_ID
SA_DET
SA_EXP
DET_ID
INS_MODE
OBS_MODE
OBS_TYPE
EXP_TYPE
FLT_ID
…
…
EXP_ID
GRM_ID
LMP_ID
SLT_ID
…
…
…
…
…
…
DET_ID
BITPIX
PIXSIZE
NAXIS
DETSIZ1
DETSIIZ2
CRDELT1
CRDELT2
DETOFF1
DETOFF2
EXP_ID
INSTRUME
OBSERVER
DATE_OBS
EXPSTART
EXPTIME
RA
DEC
AIRMASS
EXP_ID
SA_FILE
SA_CD
MDBT
EXP_ID
$_filename
DATE_CR
NAXIS1
NAXIS2
$_path_fts
CD_ID
CD_DATE
NCOPIES
CD_ID
wss_sess_id
DATE_CR
TIME_CR
DATE_MD
TIME_MD
$_path
CD_ID
FITS database
Garching, 10-14 June 2002
shared with Technical database
“TOWARD AN INTERNATIONAL
VIRTUAL OBSERVATORY”
16
MDBT
AAA database
SESS_ID
TAG
TIME
DATE_CR
HASHTAB
VALUE
TIME_CR
ACRON
STR
TAG
TAG
DATE_END
CD_TAB
TIME_END
CD_DATE
$path
NCOPIES
CD_ID
CD_ID
…
XYZ database
shared with FITS database
Technical database
TAG
TIME
HASHTAB
VALUE
ACRON
TAG
STR
TAG
…
CUB database
HASHTAB
ACRON
TAG
Garching, 10-14 June 2002
“TOWARD AN INTERNATIONAL
VIRTUAL OBSERVATORY”
TAG
TIME
VALUE
STR
TAG
17
Cyclic operations scheme
from: di Serego et al., 26 Jan 1995
Garching, 10-14 June 2002
“TOWARD AN INTERNATIONAL
VIRTUAL OBSERVATORY”
18