Transcript MPC Data Ingest

MPC Data Ingest Notes
Orbit calculation and creation of
MOPS objects
• Objects include Detections, Orbits (original
MPC orbits), Orbits (JPL-calculated)
• Stored to disk
• ~19 h running time with one CPU
Additional object creation and DB
insertion via MOPS routines
• Object creation includes Tracklets,
DerivedObjects
– 159366 Tracklets
– 159357 DerivedObjects
• DB insertion includes Detections,
Tracklets, DerivedObjects
– 31778048 Detections
– 32096771 or ~32.09*10^6 objects
DB transaction rates
• Additional object creation and DB insertion
via MOPS routines
– 32.09*10^6 objects + 159357 records in table
derivedobject_attrib = ~32.25*10^6 records
– 36.5 h running time or ~245 transactions/sec
DB transaction rates
• Batch DB insert using optimized SQL
directly on the database server
– 32.25*10^6 records
– 8.73 h running time
– ~1025 transactions/sec
• Batch DB insert using optimized SQL over
the network
– 9.75 h running time
– ~918 transactions/sec
MOPS overhead
• Retrieving MOPS objects from disk and
calling the insert method for each object
– Includes various data integrity checks
– Prepares SQL statements
– Executes SQL statements
– Processes database return values
• 26.75 h additional processing time versus
straight SQL insert
Improving the situation
• Reducing number of detections
– Date-limiting observations
• Optimizing the detections table definition
– Removing indexes
DB transaction rates
• Batch DB insert using optimized SQL with
the removal of six indexes in detections
table
– 32.25*10^6 records
– 55 min running time
– ~9772 transactions/sec
• Undetermined how indexes affect
performance using MOPS routines
Conclusions
• 245 transactions/sec is a realistic value for
object insertion (mostly Detections) via MOPS
• Scale of MPC data and/or my implementation
may be problematic
– Inserted Spacewatch fields and detections along with
NEOS model SSM to MPC data instance
– Normal runtime for MOPS simulation using
Spacewatch data and NEOS model is reported to be
half a night
– Started DTCTL on Friday, still running…
– Received two interruptions with error “Too many links”
in attributeOrbits