Transcript Downlink Segment Status - California Institute of Technology

Downlink Segment Test Status (S13.0)
Russ Laher
August 29, 2005
S13.0 Test Readiness Review, August 29, 2005
Russ Laher (1)
Summary of S13 D/L work performed









D/L-group software changes implemented for S13.0
 99 pipeline-related CRs/AIs (105 for S12.0)
 38 database-related CRs/AIs (67 for S12.0)
 22 ARs (31 for S12.0)
D/L segment-testing to check specific CR/AR/AI changes
 Small cluster of 16 pipeline-drones (mini “ops-like” environment)
 SODB (fresh S12 database via EdJackson)
 Product archiver run on pipeline results for end-to-end verification
ISTs doing science verification of pipeline products and testing of SPICE
QA team verification of QA sub-system upgrades
Test plan documents written by CogEs to help I&T
Code walk-throughs and training sessions conducted on a weekly basis over
the last four months to improve software quality and help I&T
Weekly downlink meetings with developers and IST Cog. E.s
Many “Advanced Data Products” development team meetings with developers
and customers
Weekly “Archive Working Group” meetings with developers, IRSA group, SSC
deputy manager (L. Storrie-Lombardi), and archive scientist (M. Lacy)
S13.0 Test Readiness Review, August 29, 2005
Russ Laher (2)
What’s New for S13.0

IRS
New IRS pipeline to produce background-subtracted images and spectra ((plScriptId 2023)

New IRS pipelines to generate mosaics and point-source extractions for peakup images
(plScriptIds 2340 and 2348 for FIF and MOPEX pipelines)

Upgrades to SPICE (various improvements; new spectral rectification
IRAC

New IRAC imask product (SOSDL-SIS-PD-3030)

IRAC delay-time calculation extended across AOR boundary

Superboresight creation, archival and _usage_ in pointing transfer to ensure all IRAC channels get
best refined pointing in future campaign reprocessings (based on optimal reprocessing of channel
1 and 2 data)

New database table (QA_ptg_refine), schema, store procedures to store IRAC pointing-refinement
information on a BCD basis
MIPS:Si

All P-masked pixels will be explicitly NaN'd in final MIPS-24 products

All legitimate 0-valued pixels in the difference plane of MIPS-24 SUR-mode DCEs will be set to
NaN and propagated to the final BCD

For all MIPS channels/modes: use calibrated CSM_PRED versus Y_spot position to correct for
pointing due to unpredictable scan-mirror wobble

All MIPS BCD products will have background model estimate keywords: ZODY_EST, ISM_EST,
CIB_EST

Significant enhancements to MIPS-24 offline pipelines. e.g., user-friendly tutorials and the
inclusion of many utility scripts

For the very first time, a document outlining Frequently Asked Questions on MIPS-24 processing.
To be used by ISTs, I&T, CogEs and the PO. See:
http://spider.ipac.caltech.edu/staff/fmasci/home/sirtfwork/MIPS24_FAQs.pdf



S13.0 Test Readiness Review, August 29, 2005
Russ Laher (3)
What’s New for S13.0 – cont.

MIPS:Ge

Upgrades to GeRT (offline Germanium Reprocessing Tool)

Fixed source bug preventing clean compile on Linux and Solaris with GCC

Modification to treat Infinity as NaN in calibration files

Modification so that NaN's get written to calibration files, not Infinity

Modification to allow namelist to interchange order of Column and Hipass filters

WARMFLAG included in as filtering criterion in caltrans calls

Separate WARMFLAG namelist blocks added

Infrastructure Changes for “Warm” Spitzer Observation Campaigns

New warmFlag field in campaigns, MipsGe Fallback/Metadata database tables and supporting
software

Upgrade to C-API function and database-stored function to read warmFlag from the database

Modification to job.c to write WARMFLAG to fos16.fits FITS header
Post-BCD Software


New offline script to create MIPS SED mosaics
Bandmerge upgrade

Porting of MOPEX and bandmerge modules to Linux and Mac OsX

MOPEX GUI development for Solaris

Improved user documentation for mosaicker

Various post-BCD-pipeline bug fixes (e.g., increased CRVAL precision by one digit)
Generic Software

Robust scheme in pointing-transfer to guard against outliers in input Boresight Pointing History
File (BPHF; all instruments)

New adaptive-smoothing tool


S13.0 Test Readiness Review, August 29, 2005
Russ Laher (4)
What’s New for S13.0 – cont.



On-the-fly reprocessing (automated pipeline running and product archival)

Database schema, stored procedures, triggers, and perl API functions

Webform for SSC-internal job submission

“Lights out” perl script
IRSA Connectivity

Populating new database tables for storing IRSA metadata for BCD and post-BCD
products

Creation of JPEG image previews for all primary BCD and post-BCD products and
compressed/lossy FITS files for mosaics (all imaging instruments), and GIF spectralplot previews for IRS, in order to support the user quick-look interface in Leopard
QA sub-system upgrades

SDAT upgraded to run under proxy apache server

New showQaPtgXfer.cgi CGI script to query QA_PTG_XFER database table and
display results

Populating new QA_IRS_REFFRM and QA_IRS_RHFIND database tables

New fields being populated in QA_PTG_XFER database table

Improvements in SDAT image display

Miscellaneous minor improvements
S13.0 Test Readiness Review, August 29, 2005
Russ Laher (5)
Tests Under S13.0

IRS
All Sci (BCD&BQD), DarkCal, DarkEns, Flatcal and peak-up-imaging pipelines; used fallback cal;
all data run through archiver

Details in IRS S13 Test Plan Document
MIPS-Ge

Sci (scanmap, photometry, sed, tpm), darks, flats, warm AOR, fine photometry; used fallback cal;
all data run through archiver

Details in MIPS-Ge S13 Test Plan Document
MIPS-24

SUR-mode sci scanmap (slow, med, fast cases); SUR- and RAW-mode photometry; SUR- and
RAW-mode TPM; flat-field cal w/ pointing transfer, non-linearity cal w/ pointing transfer; SUR &
RAW dark cal w/ pointing transfer ; used fallback cal; all data run through archiver

Details in MIPS-24 S13 Test Plan Document
IRAC

sub-, full-array science and cal pipelines for all exposure modes, frame times, and combinations
of internal and external repeats; product records and file system contents checked for data run
thru each pipeline; used metadata and fallback cal; all data run through archiver

Details in IRAC S13 Test Plan Document
Post-BCD

FIFcreation; pointing refinement;single_frame_pnt_src_ext;multi-frame pnt-src_ext; mosaic pntsrc_ext; overlap correction; image_interp, outlier; image_reinterp; mosaic_coadd for IRAC,
MIPS24, 70 &160; all data run through archiver

Details in PostBCD S13 Test Plan Document
QA, SDAT

Loader run for all tested pipelines, SDAT tool run, qa pages and statistics pages brought up

Details in QA/Infrastructure S13 Test Plan Document






S13.0 Test Readiness Review, August 29, 2005
Russ Laher (6)
Tests Under S13.0


Testing of the “lights out” script for on-the-fly reprocessing for many reqKeys
Testing of SDM’s new channelized product archiving many reqKeys
S13.0 Test Readiness Review, August 29, 2005
Russ Laher (7)
S13.0 Functions/Test Data Flows
S13.0 Test Readiness Review, August 29, 2005
Russ Laher (8)
S13.0 Capabilities
Sub-system
Capability
Ingest
Supports DCEs, Pntng, ECSV, Rpt files, Mem Dmps.
Sped up, SigEvents, FEIdelete, ReRun, Archiver
naming, HK compression
Pipeline Exec
real-time data reduction; reprocessing
Instrument pipelines
All three instruments science and calibration
Calibration Transfer
Fallback rules (incl NIT) for all three instruments, three
nearest in time rules for all three instruments, blessed
status flag, asymmetric discriminant selection,
interpolation rule
Instrument Performance Monitoring
Create/Populate HK tables in the HK DB; web-based
viewing and data retrieval; updated menus
Quality Assessment
Creation and populating of QA tables; SDAT web-based
QA inspection; analysis and display functions; support
archive release
CDF
Populating of version controlled CDF files on the pipeline
drones; retrieve CDF in pipelines, version tracking in the
SODB
Imaging post-BCD pipeline
Point source extraction, pointing refinement (w/ & w/out
2MASS, USNO, UCAC, FIRST), tiling, overlap
correction, outlier identification and mosaic coadding,
bandmerging
S13.0 Test Readiness Review, August 29, 2005
Russ Laher (9)
S13.0 Success Criteria















Run, re-run requests that failed on OPS
Interact with the SODB via perl & c-APIs
Deposit raw data into the archive file system
Create pipeline manifests & ensembles
Run manifests through APES
Run multiple jobbers on 16 pipeline drones
Use CDF to get configuration files
Use CalTransfer to get calibration data (NIT and/or fallbacks)
Execute post-BCD pipelines including pointing refinemnet
Put QA statistics into QA DB tables
Put processing results in sandbox and set file pointers in DB
Use SDAT to access and view results on segment test
Get “better” BCDs & post-BCD prods
IST sci verification of seg test data & buy off (incremental from IOC)
Pass the running of the (channelized) archiver
S13.0 Test Readiness Review, August 29, 2005















Russ Laher (10)
Applicable SISs
Calibration

SOSDL-SIS-CL-3000 : darkcal output FITS file, dark image

SOSDL-SIS-CL-3001 : lincal output FITS file, linearity model cube

SOSDL-SIS-CL-3002 : lincal output table file, pixel cal data

SOSDL-SIS-CL-3003 : lincal output cmask image

SOSDL-SIS-CL-3004 : darkcal output cmask image

SOSDL-SIS-CL-3005 : pmask definitions

SOSDL-SIS-CL-3006 : Electronic Nonlinearity Calibration File

SOSDL-SIS-CL-3007 : MIPS-Ge stim_latent

SOSDL-SIS-CL-3008 : function loadcalkeywords

SOSDL-SIS-CL-3009 : IRAC-loadcalkeywords version-1 SODB Table

SOSDL-SIS-CL-3010 : darkest cmask

SOSDL-SIS-CL-3011 : CalTrans Rule List

SOSDL-SIS-CL-3012 : IRS Umask

SOSDL-SIS-CL-3013 : IRS Lmask

SOSDL-SIS-CL-3014 : IRAC flatcal data cube

SOSDL-SIS-CL-3015 : IRAC flatcal cmask

SOSDL-SIS-CL-3016 : IRS flatcal cmask

SOSDL-SIS-CL-3017 : IRS Omask

SOSDL-SIS-CL-3018 : IRS Flux Conversion Table

SOSDL-SIS-CL-3019 : IRS FOV PSF Table

SOSDL-SIS-CL-3020 : IRS COADD2D AMASK

SOSDL-SIS-CL-3021 : IRS COADD2D FMASK
Updated

SOSDL-SIS-CL-3022 : IRS WAVSAMP TABLE

SOSDL-SIS-CL-3023 : MIPS-Ge cmask
Data Management

SOSDL-SIS-DM-3000 :
Updated
Data Tools

SOSDL-SIS-DT-3000 :

SOSDL-SIS-DT-3004 : profile.tbl (IRS BQD)

SOSDL-SIS-DT-3005 : ridge.tbl (IRS BQD)

SOSDL-SIS-DT-3006 : extract.tbl (IRS BQD)

SOSDL-SIS-DT-3007 : irs_tune.tbl (IRS BQD)
New
Executive

SOSDL-SIS-EX-3000 : plinexyz.cdf

SOSDL-SIS-EX-3001 : run time environment variables
Ingest

SOSDL-SIS-IN-3000 :
Instrument Performance Monitoring

SOSDL-SIS-IP-3000 :
S13.0 Test Readiness Review, August 29, 2005
New
Products

SOSDL-SIS-PD-3000 : cvti2r4 output FITS file, 32-bit real data

SOSDL-SIS-PD-3001 : cvti2r4 output FITS file, dmask

SOSDL-SIS-PD-3002 : muxbleedcorr output DCE FITS file

SOSDL-SIS-PD-3003 : imfliprot output FITS file

SOSDL-SIS-PD-3004 : MIPS Ge bmask

SOSDL-SIS-PD-3005 : imagest output bmask

SOSDL-SIS-PD-3006 : tranhead keyword-translate-only list

SOSDL-SIS-PD-3007 : Tile/BCD association files

SOSDL-SIS-PD-3008 : Mosaic-geometry interface

SOSDL-SIS-PD-3009 : BCD delta file

SOSDL-SIS-PD-3010 : PREDICTSAT s-mask

SOSDL-SIS-PD-3011 : dldeltafile library

SOSDL-SIS-PD-3012 : PRF/PSF map

SOSDL-SIS-PD-3013 : Point source extraction table

SOSDL-SIS-PD-3014 : Detection/Tile/BCD Association File

SOSDL-SIS-PD-3015 : imagest output slope image

SOSDL-SIS-PD-3016 : latimflag l-mask

SOSDL-SIS-PD-3017 : fiducial frame definition parameters

SOSDL-SIS-PD-3018 : mosaic_interpolation table

SOSDL-SIS-PD-3019 : matched-point-source refined position table

SOSDL-SIS-PD-3020 : fiducial frame input parameters

SOSDL-SIS-PD-3021 : bandmerged point source output file

SOSDL-SIS-PD-3022 : bandmerger pointer dump file

SOSDL-SIS-PD-3023 : band-pair registration uncertainties

SOSDL-SIS-PD-3024 : bandmerge statistics

SOSDL-SIS-PD-3025 : MIPS24 bmask

SOSDL-SIS-PD-3026 : FPGen Command Line Input

SOSDL-SIS-PD-3027 : FPGen SODB Entries file (input)

SOSDL-SIS-PD-3028 : FPGen Template Input file

SOSDL-SIS-PD-3029 : Bandmerge (QA) spectral combination counts

SOSDL-SIS-PD-3030 : IRAC imask
Pointing Transfer

SOSDL-SIS-PT-3000 : pixel-scale table file

SOSDL-SIS-PT-3001 : Ptg Refinement Absolute Source List Input Fmt

SOSDL-SIS-PT-3002 : MIPS Scan Mirror Parameters Table

SOSDL-SIS-PT-3003 : Instrument FOV Table

SOSDL-SIS-PT-3004 : Super Boresight Pointing History File

SOSDL-SIS-PT-3005 : Refined DCE Position Table File

SOSDL-SIS-PT-3006 : Aperture-ID-to-name translation CDF table
Quality Assessment

SOSDL-SIS-QA-3000 : QA file (*.qa) format
Russ Laher (11)
Pipeline-Related CRs/AIs (99)
6576
6578
6845
6844
6545
6595
6565
6608
6254
6246
6604
6697
6403
6625
6655
6533
6698
6529
6715
6839
6416
6675
6594
6848
6674
6822
6693
6692
6805
downlink
downlink
downlink
downlink
downlink
downlink
downlink
downlink-analysis
downlink-irac
downlink-irac
downlink-irac
downlink-irs
downlink-irs
downlink-irs
downlink-irs
downlink-irs
downlink-irs
downlink-irs
downlink-mips-24
downlink-mips-24
downlink-mips-24
downlink-mips-ge
downlink-mips-ge
downlink-mips-ge
downlink-mips-ge
downlink-pl-cal
downlink-pl-cal
downlink-pl-cal
downlink-pl-cal
Hesselroth_Ted
Hesselroth_Ted
Laher_Russ
Laher_Russ
Hesselroth_Ted
Fowler_John
Hesselroth_Ted
Pesenson_Meyer
Lee_Wen-Piao
Lee_Wen-Piao
Khan_Iffat
Narron_Bob
Makovoz_David
Narron_Bob
Narron_Bob
Narron_Bob
Narron_Bob
Narron_Bob
Masci_Frank
Makovoz_David
Masci_Frank
Henderson_David
Henderson_David
Henderson_David
Laher_Russ
Wheelock_Sherry
Henderson_David
Lee_Wen-Piao
Lee_Wen-Piao
In libtblio build, skip soft link step...
In libmathc90 build, skip soft link step...
ParchiverForDnl bug fix
Update to D/L segment-test build script
Tblio null filepointer bug
Have cvti2r4 return an exit code of "1" for MIPS Ge stims
Move shared library suffix definitions to platform-dependent makefiles
Release adaptive smoothing tool
Creation of imask, final product mask file
Skydark Delaytimes within the Header
Change needed in CRVAL precision
Update BCD extraction wrapper: w_irs_bqd_process2.pl
New IRS Post-BCD Peakup Imaging pipeline
Coadd2d to pass bits from BMASK to AMASK
Do not perform a simple average of RA_SLT, DEC_SLT and PA_SLT
IRS background subtracted images and spectra
Update w_irs_bqd_process.pl to include call to "LoadIrsaPtgMetaData".
Add header card for cal_set ID
Ensure that all p-masked pixels are NaN'd in final mips24 BCD products.
Do source extraction for MIPS-24 on mosaic, not individual BCDs
Make 0-valued pixels in raw difference frames NaN's for bcd pipeline
Add warmflag keyword to caltrans calls
Allow changing the order of MIPS-70 filtering
Slopecal hipass filter does not always produce reliable results
Write new WARMFLAG FITS-keyword to fos16.fits
Upgrade MIPS calibration-observation database
Add "WARMFLAG" to calkeywords input-keyword-list for MIPS:Ge
Mod. to calkeywords library for new "warmFlag" field
Caltrans.h modification
S13.0 Test Readiness Review, August 29, 2005
Russ Laher (12)
Pipeline-Related CRs/AIs (99) – cont.
6571
6572
6765
6642
6789
6680
6682
6723
6725
6726
6727
6756
6764
6599
6741
6670
6777
6640
2597
1891
6621
6525
6632
6549
6550
6553
6630
6555
downlink-pl-cal
downlink-pl-cal
downlink-pl-exec
downlink-pl-exec
downlink-pl-exec
downlink-pl-exec
downlink-pl-exec
downlink-pl-exec
downlink-pl-exec
downlink-pl-exec
downlink-pl-exec
downlink-pl-exec
downlink-pl-exec
downlink-pl-ing
downlink-pl-prod
downlink-pl-prod
downlink-pl-prod
downlink-pl-prod
downlink-pl-prod
downlink-pl-prod
downlink-pl-prod
downlink-pl-prod
downlink-pl-prod
downlink-pl-prod
downlink-pl-prod
downlink-pl-prod
downlink-pl-prod
downlink-pl-prod
Fang_Fan
Fang_Fan
Laher_Russ
Makovoz_David
Laher_Russ
Laher_Russ
Makovoz_David
Makovoz_David
Makovoz_David
Makovoz_David
Makovoz_David
Laher_Russ
Laher_Russ
Masci_Frank
Masci_Frank
Laher_Russ
Masci_Frank
Fowler_John
Henderson_David
Lee_Wen-Piao
Makovoz_David
Khan_Iffat
Masci_Frank
Hesselroth_Ted
Khan_Iffat
Hesselroth_Ted
Masci_Frank
Masci_Frank
Wavsamp module output signed offset files
Wavsamp module pixel sizes need to be changeable
APES static-file changes for new IRS "reffr" and "mopex" threads
Rename fif.tbl to mosaic_fif.tbl in mosaic.pl, mosaic_mask.pl, ...
APES static-table changes for new IRS plScriptId=2023
ReqMode=111 not recognized by createEnsembles.pl
New offline script to create MIPS SED mosaics
Update w_bqd_run_overlap.pl not to create empty offsets tables
Modify w_bqd_run_fiducial_frame.pl to accomodate IRS peak-up imaging
Modify createFifTable.pl to accomodate IRS peak-up imaging
Modify getFifTableFilenameAndStatus in WrapperUtils.pl to include IRS
"Lights out" script for on-the-fly reprocessing
APES static-file changes for new Onthefly job-manifests
Add "refinedPointing" fileType to subroutine "getFileTypeAbbreviation"
Code walk-through fixes/updates to "LoadIrsaPtgMetaData" perl subroutine.
New productTypes table entries for preview images for IRSA connectivity
Update w_bqd_run_mopex.pl for IRSA previews and metadata-generation
Improve imagest data-reduction logic and chi-square computation
Add photometric nonlinearity module to MIPS:Ge pls
Module needed to flag "banding" effect in IRAC channels 3 & 4
Add call to new subroutine "addQA_ptg_refine" in w_bqd_pointrefine.pl
New module to replace data in a FITS file by a given value
Add subroutines "LoadIrsaPtgMetaDataTable()", ...
String literal treated as character array in nl_and_params.c
New module to combine coverate weighted images
Multiply defined global variables in irs modules
New libraries/modules to create new downlink products to support IRSA
Register some pointing-refinement (IRAC) keywords into the DB...
S13.0 Test Readiness Review, August 29, 2005
Russ Laher (13)
Pipeline-Related CRs/AIs (99) – cont.
6620
6584
6840
6593
6681
6384
6679
6427
6694
6728
6612
6438
6733
6468
6734
6735
6737
6740
6818
6607
6437
6554
6710
6600
6100
6517
6752
6781
5706
downlink-pl-prod
downlink-pl-prod
downlink-pl-prod
downlink-pl-prod
downlink-pl-prod
downlink-pl-prod
downlink-pl-prod
downlink-pl-prod
downlink-pl-prod
downlink-pl-prod
downlink-pl-prod
downlink-pl-prod
downlink-pl-prod
downlink-pl-prod
downlink-pl-prod
downlink-pl-prod
downlink-pl-prod
downlink-pl-prod
downlink-pl-ptg
downlink-pl-ptg
downlink-pl-ptg
downlink-pl-ptg
downlink-pl-ptg
downlink-pl-ptg
downlink-pl-ptg
downlink-pl-qa
downlink-pl-qa
downlink-pl-qa
downlink-pl-qa
Laher_Russ
Narron_Bob
Khan_Iffat
Masci_Frank
Khan_Iffat
Lee_Wen-Piao
Masci_Frank
Hesselroth_Ted
Makovoz_David
Hesselroth_Ted
Khan_Iffat
Masci_Frank
Masci_Frank
Fowler_John
Masci_Frank
Masci_Frank
Fowler_John
Khan_Iffat
Masci_Frank
Masci_Frank
Masci_Frank
Masci_Frank
Masci_Frank
Masci_Frank
Masci_Frank
Hesselroth_Ted
Hesselroth_Ted
Jacobson_Jeff
Laher_Russ
Please add new subroutine "addQA_ptg_refine" to WrapperUtils.pl
Libim should optionally not to convert all non-finite numbers to NaNs
New mode in mosaic_outlier to create minimum value mosaic
Write background model estimate keywords: ZODY_EST, ISM_EST, CIB_EST
New module to compute FIF for MIPS70 SED mosaic
Remaining code walk-through action items for latimflag module...
Update w_pointinghistory.pl to add calls to "LoadIrsaPtgMetaData"
Tblio/dlimage duplicate symbol errors
Update w_bqd_run_mopex.pl to include call to LoadIrsaPtgMetaData.
Port of mopex to OS X
New module to compute the covariance matrix in a set of images
Use calibrated CSM_PRED to correct for MIPS pointing
Update cvtR4i2 module to report MIN, MAX pixel values to stdout.
Upgrade/bug fixes to bandmerge
New library and modules to create of lossy FITS images for U/L.
New module to robustly compute image noise...
Linux and OS X ports of bandmerge
Bug in tblio related to delete_rows processing
Update pointing transfer thread to write ORIGINAL pointing keywords
Update getPH_online for "refinedPointing" fileType (SBPHF) queries.
Come up with scheme to archive and register superboresight PH files..
Implement robust scheme in pointing transfer to guard against outliers
Update GenSBPHFsForCampaign.pl following comments from code walk-thru..
Update GenSBPHFsForCampaign.pl to optionally ingest SBPH files
Update pointing transfer to optionally use new SBPH files
Allow "Frame" window to be able to close in SDAT
Improvements for SDAT image display
Add qa_irs_rhfind table for qaloader
Write SDAT.log to $SOS_LOGS/cgiout directory
S13.0 Test Readiness Review, August 29, 2005
Russ Laher (14)
Pipeline-Related CRs/AIs (99) – cont.
6564
6774
6769
6799
6751
6783
6808
6736
6479
6775
6129
1577
1732
downlink-pl-qa
downlink-pl-qa
downlink-pl-qa
downlink-pl-qa
downlink-pl-qa
downlink-pl-qa
downlink-pl-qa
downlink-pl-qa
downlink-pl-qa
downlink-pl-qa
sos
sys-engring-i+t
sys-engring-i+t
Laher_Russ
Narron_Bob
Jacobson_Jeff
Masci_Frank
Wheelock_Sherry
Narron_Bob
Laher_Russ
Laher_Russ
Wheelock_Sherry
Jacobson_Jeff
Laher_Russ
Bennett_Lee
Yu_Felix
Dqa_deploy.csh: check to see if SOS_LOGS is defined
Qaloader change for the new QA_irs_reffrm database table
New qa_irs_reffrm database table
Make association of refined ptg tables with BPHFs more robust
Allow SDAT to be displayed in the proxy server environemnt
Qaloader change for the new QA_irs_rhfind database table
New SODB_Utils.pl function SODB_Get_QaPtgXfer
QA scripts to dynamically read PERL_PATH from environment
Display some fields in table qa_ptg_xfer
New database stored procedure addQA_irs_reffrm
Change QA primary proxy URL
Super-Boresight Test Plan and operational instructions for S13.0.
MIPS-24 S13.0 Test Plan..
S13.0 Test Readiness Review, August 29, 2005
Russ Laher (15)
Database-Related CRs/AIs (38)
6660
6701
6339
6763
6696
6677
6782
6784
6785
6787
6795
6798
6402
6800
6434
6802
6807
6809
6813
6816
6678
6631
6743
6615
6712
6747
6749
6753
archive-database
sdm
sdm-archive
sdm-database
sdm-database
sdm-database
sdm-database
sdm-database
sdm-database
sdm-database
sdm-database
sdm-database
sdm-database
sdm-database
sdm-database
sdm-database
sdm-database
sdm-database
sdm-database
sdm-database
sdm-database
sdm-database
sdm-database
sdm-database
sdm-database
sdm-database
sdm-database
sdm-database
Jacobson_Jeff
Jacobson_Jeff
Jacobson_Jeff
Jacobson_Jeff
Jacobson_Jeff
Jacobson_Jeff
Jacobson_Jeff
Jacobson_Jeff
Jacobson_Jeff
Jacobson_Jeff
Jacobson_Jeff
Jacobson_Jeff
Laher_Russ
Jacobson_Jeff
Laher_Russ
Jacobson_Jeff
Jacobson_Jeff
Laher_Russ
Jacobson_Jeff
Jacobson_Jeff
Jacobson_Jeff
Jacobson_Jeff
Laher_Russ
Jacobson_Jeff
Jacobson_Jeff
Jacobson_Jeff
Jacobson_Jeff
Jacobson_Jeff
Update SPs to support the archive by channel
Add softwareVersion column in processingVersion table
Be able to archive on an channel basis
New plScripts DB records for IRS "reffr" and "mopex" threads
Add warmFlag to C-API-function sdm_getDceRecForDP (sdm_sodb.ec)
New "warmFlag" boolean field in DB campaigns table
New database stored procedure addQA_irs_rhfind
Mod. to database procedure deleteNonArchivedProducts
New IRS background-subtraction pipeline (plScriptId=2023)
Database SP to compute new cross-AOR delaytimes from r_sclk
Perl API function for createPvIdsForOnTheFlyReprocessing
New database stored function getQaInfoForRequest
New DB stored function for calculating delay time
Perl API for computeIracDelayTimeBetweenSuccessiveRequests
New perl API-function computeIracDelayTime
New and modified stored procedures to support OTF request archiving
New DB stored function getQaPtgXferDataForRequest()
New perl API function getQaPtgXferDataForRequest()
Need schema and perl API update for new columns in QA_ptg_xfer table.
Need stored function to query QA_ptg_xfer table for input dceId.
New "warmFlag" boolean field in MIPS-Ge cal. DB tables
Update campaign stored procedures for new "warm" MIPS campaign.
Perl APIs for on-the-fly reprocessing
New "QA_ptg_refine" table requested by IRAC IST..
New records to be deleted in DB SP deleteNonBestProducts
New database SP createPvIdsForOnTheFlyReprocessing()
New database tables for on-the-fly reprocessing
New database fields for on-the-fly reprocessing
S13.0 Test Readiness Review, August 29, 2005
Russ Laher (16)
Database-Related CRs/AIs (38) – cont.
6754
6695
6627
1719
1720
1695
1716
1667
1664
1727
sdm-database
sdm-database
sdm-database
downlink
downlink
sdm
sdm
sdm
sdm
sos
Jacobson_Jeff
Jacobson_Jeff
Jacobson_Jeff
Laher_Russ
Bennett_Lee
Jackson_Ed
Jackson_Ed
Bennett_Lee
Bennett_Lee
Laher_Russ
New database stored procedures for on-the-fly reprocessing
Return warmFlag in DB stored function getDceProfile
Schemas, sql loading stored procedures and perl-API for IRSA metadata
New productTypes entries for RSA-related products
Change exposure type for several MIPS fine scale flat observations
New ensemble-creation rules for new IRS "reffr" and "mopex" threads
New IRS ensemble-creation ruleId=8 for plScriptId=2023
Data-content change in ProductTypes database table
New data content for ProductTypes database table
Update release level in product types table for 7157
S13.0 Test Readiness Review, August 29, 2005
Russ Laher (17)
ARs (22)
6575
6583
6577
6773
6748
6833
6713
6758
6122
6431
6705
6503
6704
6501
6820
6827
6582
6563
6838
6750
6770
6755
downlink
downlink
downlink
downlink-irac
downlink-irac
downlink-irs
downlink-irs
downlink-mips-ge
downlink-mips-ge
downlink-mips-ge
downlink-mips-ge
downlink-mips-ge
downlink-mips-ge
downlink-mips-ge
downlink-mips-ge
downlink-mips-ge
downlink-mips-ge
downlink-pl-prod
downlink-pl-prod
downlink-pl-qa
sdm-database
sdm-pr-archiver
Hesselroth_Ted
Fowler_John
Hesselroth_Ted
Lee_Wen-Piao
Lee_Wen-Piao
Narron_Bob
Narron_Bob
Henderson_David
Henderson_David
Henderson_David
Henderson_David
Henderson_David
Pesenson_Meyer
Henderson_David
Henderson_David
Henderson_David
Henderson_David
Fowler_John
Makovoz_David
Hesselroth_Ted
Jacobson_Jeff
Jacobson_Jeff
In libim build, skip soft link step ...
Pick does not display Infinity values ...
In libintegrate build, skip soft link step (conditional)
Darkdrift module change
INSBPOSDOM writes no processing history to header
RIDGE not reading FOV file
EXTRACT module: improper dependency of table output on input header
Slopecal mistreats calibration files with NaN values
Slopecal module generates fitsio errors under certain conditions
Ge_STIMEXTRAP bmask bit
Typo in exptype to cdf block variant lookup for exptype="dfl"
Hardcode interp module to use geb_STIMEXTRAP
Mipsflat needs command line option for coverage map file name
Mips_ge_mask.h
MIPS-160 exptype=FINE doesn't exist
Downlink/include/nanvalue.h does not produce linkable code ...
NaN's in a data file turn into "Infinity" when read in by fitsio
The pick utility has a display error for unsigned integer...
Apex_1frame.pl can not handle images > 10000 pixels in one dim
SDAT can not display MIPS channel 3 DCE
More robust getCalProducts... database stored procedures
GetCalProductsForRequest need to return int8 type for file size
S13.0 Test Readiness Review, August 29, 2005
Russ Laher (18)
Major issues to be tackled in S14













Recent bug found in post-BCD position-refinement pipeline: the
QA_IRAC_REFINE database table is not being populated!
Bandmerge pipeline wrapper upgrade so it can be executed in routine
operations
Upgrades to MOPEX GUI
Porting MOPEX GUI to Linux and Mac OsX
Bandmerge GUI development
Porting bandmerge GUI to Linux and Mac OsX
More user-friendly e-mail notification from on-the-fly reprocessing
Improved user documentation for point-source extractor
More work needs to be done to make before/after pointing-refinement
performance data more accessible (need to store it in the database)
Least-square stim interpolation for MIPS:Ge
TPM MIPS:Ge pipeline that uses self-generated darks
Multi-threaded GeRT
GeRT port to Mac OsX
S13.0 Test Readiness Review, August 29, 2005
Russ Laher (19)
Sample of new on-the-fly reprocessing webform
S13.0 Test Readiness Review, August 29, 2005
Russ Laher (20)
Sample output from new
on-the-fly reprocessing script
sscpipe14% $SIRTF_SCRIPTS/otfReprocessing.pl -o 20050812T1825.out
otfReprocessing.pl, Version 1.0
By Russ Laher ([email protected]),
Ron Beck ([email protected]), and
Jeff Jacobson ([email protected]).
Copyright (C) 2005
Spitzer-Space-Telescope Science Center (SSC),
California Institute of Technology, Pasadena, CA
Script for "lights out" on-the-fly reprocessing.
Database, role = sodb_dnl5, pipelineswrole
Date/time = Fri Aug 12 18:22:31 2005
Open database connection...
List of unique reqKeys:
Checking (reqKey, chanNum) pairs for current status...
CampaignId
662
708
708
708
Instrument
MIPS
IRSX
IRSX
IRSX
ReqKey
7120384
4980736
4980736
4980736
ChanNum
1
1
2
3
ReprocessFlag
1
0
0
1
List of pending onTheFlyReprocessing records:
ProcId
UserId
54
54
54
55
6
6
6
6
E-mail
[email protected]
[email protected]
[email protected]
[email protected]
S13.0 Test Readiness Review, August 29, 2005
Instrument
IRSX
IRSX
IRSX
MIPS
ReqKey
4980736
4980736
4980736
7120384
ChanNum
1
2
3
1
Russ Laher (21)
Sample output from new
on-the-fly reprocessing script – cont.
Deleting records from WIP tables...
Loading WIP tables...
Making new pvIds...
Aug 12 18:22:32: Querying WIP tables...
CampaignId
662
708
708
708
ProcId
Instrument
MIPS
IRSX
IRSX
IRSX
UserId
54
54
54
55
6
6
6
6
ReqKey
7120384
4980736
4980736
4980736
ChanNum
E-mail
[email protected]
[email protected]
[email protected]
[email protected]
1
1
2
3
ReprocessFlag
1
0
0
1
pvId
1055
0
0
1056
Instrument
IRSX
IRSX
IRSX
MIPS
ReqKey
4980736
4980736
4980736
7120384
ChanNum
1
2
3
1
Aug 12 18:22:32: setup manifests ...
Aug 12 18:22:32: create campaign_reproc thread entries ...
dnlAccess -> setPlHopper 54
dnlAccess -> setPlHopper 55
Aug 12 18:22:34: monitor pipeline processing ...
checking progress of reqkeys ...
Onthefly-IRSX-3 4980736 has not started yet ...
Onthefly-MIPS-1 7120384 has not started yet ...
Aug 12 18:22:34: sleeping 300 ...
checking progress of reqkeys ...
start: Aug 12 18:27:13 Onthefly-IRSX-3 4980736 2017
stop: Aug 12 18:27:14 Onthefly-MIPS-1 7120384 1030
Aug 12 18:27:35: sleeping 300 ...
dnlAccess -> setPlDone 54
dnlAccess -> setPlDone 55
Aug 12 18:32:35: End pipeline processing...
S13.0 Test Readiness Review, August 29, 2005
Russ Laher (22)
Sample output from new
on-the-fly reprocessing script – cont.
Aug 12 18:32:35: Begin product archiving...
Setting archiveCampaign flag for pvId = 1055
Setting archiveCampaign flag for pvId = 1056
Aug 12 18:32:35: Polling to see if product archiving has completed...
Certified flag for pvId = 1055 is equal to 0
Certified flag for pvId = 1056 is equal to 0
Certified flag for pvId = 1055 is equal to 0
Certified flag for pvId = 1056 is equal to 0
Certified flag for pvId = 1055 is equal to 0
Certified flag for pvId = 1056 is equal to 0
Certified flag for pvId = 1055 is equal to 1
Certified flag for pvId = 1056 is equal to 1
Aug 12 18:35:35: Terminating OTF-reprocessing episode...
Aug 12 18:35:35: The following e-mail notifications were sent:
[email protected] -> New Spitzer products are available for reqKeys: 4980736 7120384
Aug 12 18:35:35: Close database connection...
Date: Fri, 12 Aug 2005 18:35:35 -0700 (PDT)
From: Rick Ebert <[email protected]>
To: [email protected]
Subject: SIG EVENT #305740, 050812181451F1 INFORMATION
Received 2005-08-12T18:35:35 from terminateOnTheFlyReprocessingEpisode( PID N/A) on host DB Procedure,
message # 1, New Spitzer products are available for reqKeys: 4980736, 7120384
S13.0 Test Readiness Review, August 29, 2005
Russ Laher (23)
Sample of new ptg xfer Q/A statistics
S13.0 Test Readiness Review, August 29, 2005
Russ Laher (24)
Sample of new ptg xfer Q/A statistics – cont.
S13.0 Test Readiness Review, August 29, 2005
Russ Laher (25)
Sample of new ptg xfer Q/A statistics – cont.
S13.0 Test Readiness Review, August 29, 2005
Russ Laher (26)