Transcript Slides - Indico

SPYC project : Command Line Interface
and Scripting solution on top of Tango
S. PIERRE-JOSEPH ZEPHIR
(Computing Division - ICA – Synchrotron SOLEIL)
SPYC Project : CLI and scripting solution on top of Tango
30th Tango collaboration meeting, 21-23 June 2016
1
SPYC project : Command Line Interface
and Scripting solution on top of Tango
S. PIERRE-JOSEPH ZEPHIR
(Computing Division - ICA – Synchrotron SOLEIL)







History
Working tools
Main Guidelines
Architecture
Status
Human Resources
Conclusion
SPYC Project : CLI and scripting solution on top of Tango
30th Tango collaboration meeting, 21-23 June 2016
2
HISTORY
SPYC Project : CLI and scripting solution on top of Tango
30th Tango collaboration meeting, 21-23 June 2016
3
Historic
 Before 2011 : for various reason, there is one CLI per beamline who need it
 End 2011 : A Scripting Working Group created by J. Daillant (Soleil General
Director)
 Goal of the working group and of the Scripting project :
 Provide a Soleil CLI fully integrated to our control system
 With well known and common commands for all beamlines
 These commands should be scripted
 7 beamlines implemented their own Python Command Line
SPYC Project : CLI and scripting solution on top of Tango
30th Tango collaboration meeting, 21-23 June 2016
4
Historic
 Existing Solutions studied
 SARDANA at ALBA
 SPEC at ESRF
• Has inspired the required set of commands
 Both have their own integrated Control System
 Soleil Control System is built with Tango devices
The command line has to manage these devices
 Then the SoleilPYthonCommand or Spyc is born
SPYC Project : CLI and scripting solution on top of Tango
30th Tango collaboration meeting, 21-23 June 2016
5
Historic
 SPYC is based on Python/IPython/C++ development languages
 Begin 2013 :
 11 beamlines had their own Python Command Line
 For 3 new beamlines the CLI usage was not so clear
 November 2013 : first release deployed on a Beamline
SPYC Project : CLI and scripting solution on top of Tango
30th Tango collaboration meeting, 21-23 June 2016
6
WORKING TOOLS
SPYC Project : CLI and scripting solution on top of Tango
30th Tango collaboration meeting, 21-23 June 2016
7
Working Tools : 1 - Working group
 It is composed of :



General Direction : Soleil General director
Experiment Division :
 5 beamline scientists
 1 Transversal Engineer
Computing Division
 Service Information Infrastructure – for calculation cluster and its python
installation
 3 persons of Computing Control and Acquisition team (or ICA)
 Meetings


At the beginning, each 1,5 month working group meeting : decides of the project
orientation. Now more spaced.
Regularly depending on the needs technical meetings with Scientists and Spyc
developers :
o To prioritize the requirements
o To define the content the commands
 Importance of constructive exchange between the customer and the provider
SPYC Project : CLI and scripting solution on top of Tango
30th Tango collaboration meeting, 21-23 June 2016
8
Working Tools : 2 - Documentation
 “Scripting Requirements”




Written by the working group scientists
Help to start the discussion on the Scripting subject
Describes what is required by this project
No more updated : users demands arrived through our Requests Tracker – Jira.
 “Detailed Specification”




Written by ICA
Detailed explanation about the commands content (algorithm, syntax, errors …)
Allowed to discuss with beamline people at migration step
Updated whenever a new command is added in Spyc
 “Technical Description”



Redacted by ICA
Explained technical points :
o how Spyc works,
o which are the interactions with other control system tools
Updated often : at each new Spyc release
SPYC Project : CLI and scripting solution on top of Tango
30th Tango collaboration meeting, 21-23 June 2016
9
MAIN GUIDELINES
1 - HOMOGENEOUS CONTROL SYSTEM
2 - IMPROVE CURRENT SUPPLY
SPYC Project : CLI and scripting solution on top of Tango
30th Tango collaboration meeting, 21-23 June 2016
10
Main Guidelines: Homogeneous Control System
 SPYC has to be fully integrated to the existing control system to keep it
Homogeneous
 Existing notions are used : i.e : scan with Sensors/Actuators/Timebase notions
already used by scanning tools, Salsa GUI and ScanServer device
 Tango Database is used for some configuration aspects – Spyc config (start with a
set of equipments), Tango attribute alias usage
 Spyc requirements can impact the existing software applications: for instance
Salsa/ScanServer or Tango devices, attribute alias impact
SPYC Project : CLI and scripting solution on top of Tango
30th Tango collaboration meeting, 21-23 June 2016
11
Main Guidelines: Homogeneous Control System
 Tango attribute alias usage impacts our Control System
Spyc
DataBrowser
Salsa
At the beggining :
Full
attribute
name usage
Attribute
alias usage
Now: Attribute alias usage too
TANGO Software bus = Beamline Control System
Device
Device
I06-C-C07/EX/FZP-X/position = full attribute name
fastshx = attribute alias
Attribute alias is also used by « Experimental Data file management » Service – also called NeXus
SPYC Project : CLI and scripting solution on top of Tango
30th Tango collaboration meeting, 21-23 June 2016
12
Main Guidelines : improve current supply
 Take advantage of working process description to improve the current supply
 During migration step we discover some control systems issues which were solved
through abnormal workarounds in existing Beamline Python code
o
We tried to solve them asap
o
Or at least described them in the bugs tracker for future solutions
 These
two guidelines help all beamlines ( even those not involved in
Spyc !)
SPYC Project : CLI and scripting solution on top of Tango
30th Tango collaboration meeting, 21-23 June 2016
13
ARCHITECTURE
SPYC Project : CLI and scripting solution on top of Tango
30th Tango collaboration meeting, 21-23 June 2016
14
Legend :
Architecture
 Description
Python
Java
C++
Beamline code = PyBeamLineRoot
Beamline responsability
ICA responsability
DeviceRoot
PyCommandRoot
shopen …
SPYC
PyNeXusRoot
ShutterControl
calcafterscan
h5py
DataFitter
PyAbstractDevice
ascan, dmesh…
tables
PyScientificRoot
PyTangoRoot
PyTANGO
matplotlib
SimpleScan
(Salsa API)
Python
ScanServer
scipy
numpy

PyTango : allow access to Tango Devices access

PyNexus : Experimental data files access (H5py, pytables)

PyScientific : Python scientific modules (Numpy, scipy, matplotlib ..)
IPython
amove, ct, limit …
ScanUtils (Plugins)
 Responsibilities are shared between ICA and Beamlines
SPYC Project : CLI and scripting solution on top of Tango
30th Tango collaboration meeting, 21-23 June 2016
15
Architecture
 Spyc
 Main commands are provided
 Specifics Beamline commands can be easily added
 Some behaviors can be customized – add code before/after scan operations
 PyBeamline : toward a beamline standardization development
 Contains beamline specific code
 Gives a certain autonomy – flexible system
 Same code organization is recommended for all the beamlines
 Beamline code started to be organized into a configuration source code tool
(true for 2 beamlines)
 To help beamline persons who are not familiar with this kind of tool, a
procedure description is still a work in progress
SPYC Project : CLI and scripting solution on top of Tango
30th Tango collaboration meeting, 21-23 June 2016
16
STATUS
SPYC Project : CLI and scripting solution on top of Tango
30th Tango collaboration meeting, 21-23 June 2016
17
Status: PYCommand Implementation
 Solution implementation rules:
 “Inspired” from the existing Beamline code
• To facilitate the merge with it
 Reuse existing tools of our Control System
 Keep a technical consistency with the current ICA supplies by putting all the
intelligence in C++ libraries
• Can be reused by other Tango devices
• Consistent with the ICA team people competences
• Easily integrated in our continuous build mechanism (Maven/Jenkins)
 Last implementations :
 Better management of the exception inspired of PyTango with python_error and
tango_error functions
 Specification redaction is on going for commands dedicated to new services on
our control system
SPYC Project : CLI and scripting solution on top of Tango
30th Tango collaboration meeting, 21-23 June 2016
18
Spyc Status : Contents
 Available commands
SPYC Project : CLI and scripting solution on top of Tango
30th Tango collaboration meeting, 21-23 June 2016
19
Spyc Status : Deployment
 Deployment Status on Beamlines (can be useful for at least 14
beamlines)
 Spyc is used in production : 7 beamlines
 Migration starting point is not defined by time missing : 2 beamlines
 Not interested for the moment : 5 beamlines
SPYC Project : CLI and scripting solution on top of Tango
30th Tango collaboration meeting, 21-23 June 2016
20
Spyc Status : Migration process and difficulties
 Migration of a beamline consists in transferring the existing Python code to Spyc
solution
 Migration is done in collaboration between the beamline contact (one of the main
requirements is to have designated a unique entry point) and ICA
 When the Spyc philosophy is fully understood by the beamline contact, then ICA is
no more needed
 But when the command line implementation of the beamline is very complex, then it
is needed that ICA help deeper the migration process
 Accommodate “testing time” is not always easy.
SPYC Project : CLI and scripting solution on top of Tango
30th Tango collaboration meeting, 21-23 June 2016
21
HUMAN RESOURCES
SPYC Project : CLI and scripting solution on top of Tango
30th Tango collaboration meeting, 21-23 June 2016
22
Human Resources
 Direct ICA human resources
 A temporary contractor, who works full time on the subject
 Myself, not full time
 Indirect human resources

Punctually, all ICA people, if some requirements lead to improvements in other existing
software components
 More Indirect human resources

ICA and Beamlines developers work together to merge the solutions and test them.
SPYC Project : CLI and scripting solution on top of Tango
30th Tango collaboration meeting, 21-23 June 2016
23
CONCLUSION
SPYC Project : CLI and scripting solution on top of Tango
30th Tango collaboration meeting, 21-23 June 2016
24
Conclusion
 Migration step will require a lot of time
 A beautiful project based on good exchanges between working group people
and Spyc contact on beamlines
SPYC Project : CLI and scripting solution on top of Tango
30th Tango collaboration meeting, 21-23 June 2016
25
Questions ?
SPYC Project : CLI and scripting solution on top of Tango
30th Tango collaboration meeting, 21-23 June 2016
26