Transcript ICT-CPM2-AsdmRelational

ALMA Integrated Computing Team
Coordination & Planning Meeting #2
Santiago, 28-29 January 2014
ASDM relational database
Rafael Hiriart / Jorge Avarias
Original Goals
 Our main motivation to create this database was to improve
DataCapturer and QuickLook.
 DataCapture was running out of memory, observations were reaching a limit of ~40
minutes with 32 antennas.
 The original DataCapture “incremental writes” architecture – based on incremental
writes at the level of the ASDM and XML Store – was dropped because of lack of
resources in HLA/ASDM and Archive. It was suggested at this time to explore a
relational DB solution.
 We implemented a workaround for the Pointing table, the biggest ASDM table. By
saving this table incrementally to a file and streaming it at the end of the observation
to the binary store, we now support long observations (several hours).
 We continued working on DC relational database aiming to improve QuickLook. This
subsystem, which currently relies in a complicated collaboration between its own
components, DataCapture an Quicklook, can be refactored to be a simple application
that queries the database, performs a little processing and plot results.
ICT-L1 10-12 December 2013
Status
 First version of the relational database has been
delivered (ICT-150) as part of release 10.4. It is
working in the OSF.
 A new version of QuickLook is being delivered with
10.6.
 This has become quite urgent because of frequent
“freezes” of QuickLook running in the OMC.
 Besides, QuickLook plots don't scale well with the
number of antennas. A more interactive application is
required, able to support filtering on antennas, scans,
baselines, etc.
ICT-L1 10-12 December 2013
Demo
ICT-L1 10-12 December 2013
Implementation
 Based on Hibernate
 The Hibernate mapping file is generated using Java reflection to
know what are the columns and the keys of every table.
 ASDM Java row classes cannot be used directly as mapping classes
for Hibernate.
 New wrapper classes were created to circunvent problems:
• Define the no-param constructor.
• Setters need to be public.
• Some getters need to be fixed (they throw runtime exceptions).
 Database is auto-generated by Hibernate, currently deployed in
Scheduling database.
 Custom Hibernate types were implemented to handle simple
types like angle, temperature, speed, etc.; and complex types like
timeInterval, arrays, IDL enumerations.
ICT-L1 10-12 December 2013
Problems/Improvements
 Wrapper classes were manually (and painfully)
produced. Mappings are created statically.
 The system is fragile over changes in the ASDM.
 The ideal way to create these artefacts would be to
generate them as part of the HLA/ASDM build.
 Relationships between tables (FK) are not
supported.
 Arrays are stored in the database as strings
following the ASDM array serialization (<num-dim>
<dim1> <dim2> … <value-1> <value-2> ...).
 Only Java is supported.
ICT-L1 10-12 December 2013
Our plan
 We stop here!
 The mappings, as they are, fullfill our requirements to refactor
QuickLook.
 We don't have plans (or resources) to continue dedicating to this
task.
• Jorge is in transition to 100% Scheduling.
• Other resources in Control are fully allocated.
 We are well aware of the interest to use this database in a
broader context, but
• The database could need significant work/modifications to be useful
for other applications (e.g., QA).
• DB is temporary. If a permanent DB is needed, APO needs to be
involved. Maintainance, migrations, data duplication, etc., will need
to be addressed.
• These developments are well outside Control group's scope and
expertise.
ICT-L1 10-12 December 2013
Decision from Last Leads Meeting
ICT-2048, action for Alisdair Manning:
Provide a list of options to ICT management about possible delays to
"fund" making the ASDM relational database permanent, and integrated
into the code generation system.
To be discussed in Alisdair's presentation.
ICT-L1 10-12 December 2013