Transcript downloading

DC3 Goals and Objectives
Jeff Kantor
DM System Manager
Tim Axelrod
DM System Scientist
DC2 Post-Mortem/DC3 Scoping
February 5 - 6, 2008
DC3 Overall Goals and Objectives
•
•
•
•
•
•
•
•
•
Extend DC2 Application Framework and Middleware to support Data Release
Pipelines
Expand DC2 Nightly Pipelines functionality to address LSST instrument
signature removal and dayMOPS, and to improve quality of data outputs
Provide Image Processing Pipeline for control and commissioning visualization
requirements
Develop first release of Data Release Pipelines (coaddition, detection,
photometric calibration, astrometric calibration)
Develop first release of Science Data Quality Analysis System (SDQAS)
Continue scaled tests of data transfer, data processing, database ingest (to
15% of final LSST requirements)
Conduct first scaled tests of data query (with map reduce/bigtable and DBMS
SQL)
Integrate new team members
Answer key questions for PDR as documented in DM R&D Plan
DC2 Post-Mortem/DC3 Scoping
February 5 - 6, 2008
DC3 Top-level Project Plan
DC2 Post-Mortem/DC3 Scoping
February 5 - 6, 2008
For PDR - Monitor and evaluate computing, storage, and
network resource price/performance trends and
architectures
1.
Produce computing/storage, long-haul network acquisition plans 1/1/08 –
9/30/08 LLNL (mountain/base), NCSA (archive center), SDSC (data access
center), NOAO (long-haul networks)
2.
Produce cyber-security plan 1/1/08 – 9/30/08 NCSA (centers), LLNL
(mountain/base)
3.
Monitor and extrapolate computing, storage, and network architectures and
price/performance trends; produce bi-annual report summarizing findings
9/1/05 – 9/30/10 LLNL (mountain /base), NCSA (centers), SDSC, NOAO
4.
Model and prototype LSST infrastructure architecture and design, including
scalability and reliability features. 6/1/06 – 9/30/10 LLNL (mountain/base),
NCSA (archive center), SDSC (data access center), NOAO (long-haul
networks)
5.
Validate scalability of infrastructure via scaled performance loading tests of
pipeline processing, data ingest, and data transfer (Data Challenge 3, 15% of
LSST final requirements) 1/1/08 – 9/30/08 NCSA, LLNL, SDSC, NOAO
DC2 Post-Mortem/DC3 Scoping
February 5 - 6, 2008
For PDR - Peta-Scale Database Architecture and
Analysis
1.
Test the performance of open source versions of Map/Reduce/Bigtable
(hadoop/hdfs) on ingest and external queries. Characterize performance
for spatial, temporal, and ad hoc meta-data based queries 1/1/08 – 9/30/08
SLAC, JHU, Google
2.
Implement provenance ingest and re-creation of data products from raw
data and provenance 1/1/08 – 9/30/08 SLAC, LLNL
3.
Expand the implementation of persistence to include deep detection
objects 1/1/08 – 9/30/08 SLAC
4.
Evaluate relational database and map-reduce technologies for storage and
query of various types of LSST data, including images, catalogs, and
meta-data. Evaluate open source versus commercial technologies. 1/1/08
– 9/30/08 SLAC, Google
R&D Plan contains 15 items with PDR significance
DC2 Post-Mortem/DC3 Scoping
February 5 - 6, 2008