Transcript williams_messina

Some Grid Science
Roy Williams
Paul Messina
California Institute of Technology
Grids and
Virtual Observatory
Grids and
and LIGO
Virtual Sky: Image Federation
http://virtualsky.org/
from
Caltech CACR
Caltech Astronomy
Microsoft Research
Virtual Sky has
140,000,000 tiles
140 Gbyte
Change scale
Change theme
Optical (DPOSS)
Xray (ROSAT) theme
Coma cluster
VO and Grid Computing
Supercomputer center support for
Data Pipelines (not just MPI)
VirtualSky.org Ingestion
HP Superdome
VS1 Science product
HPSS
Raw data
1 TB
HPSS system
12 hr
250 Gb
Today ~ 3 TB
Computation also takes about 12 hrs.
CPU Utilization
Resampling
Copy raw data
and filter (background subtraction)
140 GB
VS2 Web product
Win2000 database
Build VS2
Write VS1
Write VS2
VO and database Join
glon
glat
flux I
flux Q
flux U
RA
Dec
mag J
mag H
mag K
Need same sorting for crossmatch
Who does it?
Or can we impose a standard?
Radio
catalog
SQL select
order by glon?
(indexed by glon)
Infrared
catalog
SQL select
(indexed by RA)
order by RA?
Crossmatch
engine
sort here?
Sorting 10^9 objects is expensive
VO Interoperability
• Directory white/yellow pages
– VO Schema repository
• How to publish, how to define content
• action> Document schema for
– archive content
– web service capability
– table/image/spectrum
• How to publish algorithms
• Plug and play data services
– SOAP, UDDI, WSDL, Jini?
• Semantic web
– “Question and answer in the language of the client”
– Topic maps
VO and Interoperability
What is the standard interaction
with a catalog service?
Client
Service
Client application (e.g., OASIS)
asks for available catalogs
then for attributes
makes a query
then displays the result
Want to be dynamic,
be able to
add catalogs and the
applications still work
VO and International
• Of course there are many
astronomy data collections in
the UK, Europe, Japan, etc.
• We intend to
collaborate/coordinate with
international efforts
LIGO
Laser
Interferometric
Gravitational wave
Observatory
Listening to
Collisions of
Black Holes and
Neutron Stars
LIGO Pictures
Grid LIGO Architecture
Objective: Add security & wide-area data services
LDAS is Ligo Data Analysis System
GriPhyN
Text request
Clients
eg Web,
Script,
Agent
Request
Manager
Replica Catalog
Replica Management
Transformation Catalog
Virtual Data Catalog
LDAS
Gatekeeper
(GRAM)
GridFTP
Local Disk
Science Algorithms
Software Collaboratory
Parallel Computing
other LDAS
Condor jobs
Data
Virtual Data Request
Data Movement
Globus RPC
GridFTP
HPSS
Grid LIGO Logic
• VD Request  Request Manager
– XML, Key-value, script?
– Action> Build document schema for this
• Do we have it already in data caches
– if yes, get from caches, use simple
transformations
– else build LDAS script
• Move relevant data to LDAS local disk
• Queue job to LDAS, get jobID
• Pass updates, then result pointer back to
user
• Perhaps cache result
International GW Grid
GEO
50 kbyte/sec
International GW Grid
• Coincidence in Ligo-Virgo data
– Not astrophysics data yet….
– Try seismic, electromagnetic data for now
Correlate signals
(LDAS)
Coincidence
events!
Currently using rsync
LHO
~12 kbyte/sec each way
Merging
3 streams to 1
Virgo
Electromagnetic
Seismic
Caltech
LLO