Transcript glast collab meeting DH 200809

Fermi LAT
Collaboration Meeting, September 2008
Fermi Large Area Telescope
Fermi Gamma-ray
Space Telescope
Tutorial on Science Analysis
Data Access
Tony Johnson
Stanford Linear Accelerator Center
[email protected]
http://glast-ground.slac.stanford.edu/DataPortal
T.Johnson
1/18
Fermi LAT
Collaboration Meeting, September 2008
Data Portal Components
•
•
The Data Portal Comprises:
– Astro Server
– Common History
– Fits Skimmer
– Merit Skimmer
– Data Catalog
• Web Interface
• Linemode Client
– Download Manager
• Web Interface
• Command Line
– WIRED Event Display
This talk will only cover the items in red.
All components available via the web interface at:
http://glast-ground.slac.stanford.edu/DataPortal
T.Johnson
2/18
Fermi LAT
Collaboration Meeting, September 2008
“Astro Server”
T.Johnson
3/18
Fermi LAT
Collaboration Meeting, September 2008
Astro Server
• Allows data to be selected based on any combination of:
– Energy Range
– Position in sky (ra,dec,radius)
• Position can be looked up in SIMBAD catalog
– Event Class (transient, source, diffuse)
– Time Range (MET)
• Can produce FT1 (fits), FT2 (fits) and/or Merit (root)*
• Supports multiple “Event Samples”
– Currently “LEOScience” or “Level1”
• Can support access to multiple versions of data once
we start reprocessing, or if we want to have any kind of
special event selections
• Level1 data currently loaded as soon as it becomes
available from pipeline
• Web and command line interfaces available
* Coming Soon
T.Johnson
4/18
Fermi LAT
Collaboration Meeting, September 2008
Astro Server Web Interface
Page pre-populated with
your most recent selection
You can give job
meaningful name
Leave fields blank
for “no-limit”
Give name of astronomical object,
sirius, M31, MCG+02-60-010
Expert options allow experimental
features to be easily added
T.Johnson
5/18
Fermi LAT
Collaboration Meeting, September 2008
Astro Server Web Interface…
README contains job
parameters
T.Johnson
6/18
Fermi LAT
Collaboration Meeting, September 2008
Common History
•
Data Portal History can be used:
– Show status of all your skimmer/astro server jobs
– To access output directory, pipeline details for all jobs
– Recover parameters of any job
Tutorial Demo
Click to go to output
directory
Click to recover parameters
T.Johnson
Click to go to pipeline
7/18
Fermi LAT
Collaboration Meeting, September 2008
Common History…
• Clicking on job name in history recovers parameters
– Allows rerunning same or similar job
T.Johnson
8/18
Fermi LAT
Collaboration Meeting, September 2008
“Fits Skimmer”
T.Johnson
9/18
Fermi LAT
Collaboration Meeting, September 2008
Fits Skimmer
• Supports merging any FT1/FT2* files in data catalog
• Allows “fcopy rowfilter” style cuts to be applied to FT1 columns
– Example:
• ZenithAngle<105 && CTBClassLevel==3
– http://heasarc.nasa.gov/lheasoft/ftools/headas/rowfilter.html
• Allows selection of data based on meta-data in data catalog
– Examples
• sIntent==“nomSciOps”
• LEOScience=="1835Survey“
• nMootKey==2103 || nMootKey==2104
• Outputs FT1 and FT2* files
* Coming Soon
T.Johnson
10/18
Fermi LAT
Collaboration Meeting, September 2008
Fits Skimmer Web Interface
Page pre-populated with
your most recent selection
You can give job
meaningful name
Row Filter
Data Selection (this is the
selection Richard has been
using for making merged
FT1/FT2 files)
T.Johnson
11/18
Fermi LAT
Collaboration Meeting, September 2008
Fits Skimmer Web Interface…
T.Johnson
12/18
Fermi LAT
Collaboration Meeting, September 2008
Fits Skimmer vs. Astro Server
• Fits Skimmer
– Allows cuts on any FT1
column
– Suitable for merging many
files with minimal cuts
– Can output FT1, FT2*
– Can be used with any
dataset in data catalog
– Fully reads all input files
• Astro Server
– Allows cuts on energy, time,
position, event class
– Suitable for selecting limited
number of events
– Can output FT1, FT2, Merit*
– Can only be used with data
loaded into database
– Uses database for efficient
access to selected events
• In future smarter single web interface may
automatically choose most appropriate backend tool.
* Coming Soon
T.Johnson
13/18
Fermi LAT
Collaboration Meeting, September 2008
“Data Catalog”
T.Johnson
14/18
Fermi LAT
Collaboration Meeting, September 2008
Data Catalog
• Allows access to many datasets including: Level1 data, ASP data,
Reprocessed data, MC data sets
Key locations allow
attention to be drawn
to important datasets
T.Johnson
15/18
Fermi LAT
Collaboration Meeting, September 2008
Data Catalog…
• Data Catalog has been designed to be extensible
• Can science groups make better use of data catalog?
– Use it to define standard datasets?
• As alternative to sending e-mail with location on user disk
• See LEOScience folder as example
– Use meta-data to keep track of common data selections
• E.g. goodData==true instead of complex set of sIntent cuts
T.Johnson
16/18
Fermi LAT
Collaboration Meeting, September 2008
Status
• Working on improving documentation
– Adding FAQ’s at:
• https://confluence.slac.stanford.edu/display/ds/Data+Access+FAQ
– Please contribute your own questions (we will try to provide the answers)
• Data Portal is now mostly complete
– Most of the planned functionality is available
• Documentation is improving
– We need to work with Science Groups to make sure we are
making optimal use of the available tools
– We need input from Science Groups (=you) on what
additional features are needed/desired
• E-mail me, or use [email protected]
T.Johnson
17/18
Fermi LAT
Collaboration Meeting, September 2008
Discussion Items
• Astro server
– What data should be loaded?
• Currently loaded with all FT1 (including LEO) satisfying
“Richard’s” sIntent cuts
– Should we remove early data?
– Is this is the “right” set of data?
• Currently all “transient, source, diffuse” data classes
– Original plan was for less data, may need to beef up server
as we get more data
• Weekly summary DST files?
– What data, when/how to make them?
• Can science groups make better use of data catalog?
– Use it to define standard datasets?
– Use meta-data to keep track of common data selections
T.Johnson
18/18