Transcript 179

LDAQ – the New Lujan Center
Data Acquisition Application
Frans Trouw, Gary Cooper and Paul Lewis
Lujan Center, Los Alamos National Laboratory
Operated by Los Alamos National Security, LLC for NNSA
UNCLASSIFIED
Outline
•
•
•
•
•
•
•
•
Brief Overview of Lujan Center
Description of hardware
Existing data acquisition application (LOS)
Issues with LOS (reminder, see Paul’s talk)
Guiding principles for new application (LDAQ)
Overview of existing new LDAQ application
Experiences with LDAQ (good!)
Summary
Operated by Los Alamos National Security, LLC for NNSA
UNCLASSIFIED
Data Acquisition at the Lujan
Center - hardware
• Detectors are all 3He gas tubes or area
detectors
• Pre-amps at detectors generate a
differential signal
• Custom “Time-of-Flight” boards take
differential input and convert into events in
a buffer
• VME (VXI) crate with single board
computer collects events from TOF units
Operated by Los Alamos National Security, LLC for NNSA
UNCLASSIFIED
Overview of Hardware
Configuration
T0 from accelerator
VME Crate
Processors
VxWorks
TOF
TOF
TOF
TOF
TOF
TOF
TOF
TOF
TOF
TOF
ROC
CPU
Remote access
Chopper
EPICS IOC
Private DAQ
Network
100 Mbit/s
TOF
TOF
TOF
TOF
TOF
TOF
TOF
TOF
TOF
TOF
ROC
CPU
LANSCE Facility
EPICS Gateway
"Public" LANL
Network
100 Mbit/s
DAQ Host PC
Windows XP
TOF
TOF
TOF
TOF
TOF
TOF
TOF
TOF
TOF
TOF
ROC
CPU
Data Archive
PC
Controls PC
Operated by Los Alamos National Security, LLC for NNSA
VXI Crates
Sample
Environments
UNCLASSIFIED
Detector & Preamp
Panels
Crate contains SBC, ROC module,
and 1-11 TOF units
ROC module
TOF unit
SBC
To Detectors
Operated by Los Alamos National Security, LLC for NNSA
UNCLASSIFIED
LOS Development Methodology
Domain:
Domain:
class diagram,
class diagram,
Domain X:
stateclass
charts,
diagram,
state charts,
state charts,
action
language
action
language
action
language
“Model compiler”:
templates
generation scripts (rules)
Source code
generator
Source file
Source file
Build script
Source file
Source file
Source file
Source file
Source file
Imported
software
“Model compiler”:
run-time library
Build
Executable
Executable
Executable
Operated by Los Alamos National Security, LLC for NNSA
UNCLASSIFIED
xtUML is in principle attractive
• High level description generates
consistent code and less errors
• Extending or changing model generates
new code that takes into account
interdependencies
• Less hand-coding => less manpower
• Quality is consistent and assured
Operated by Los Alamos National Security, LLC for NNSA
UNCLASSIFIED
xtUML has Significant Drawbacks
• Requires the desire to think at a very high
level of abstraction – requires expert
• Not all of the code can be generated
• Highly dependent on message passing
• The community using xtUML is very small
(< 100 at SMUG) - “early adopters”
• Only available for Windows
• Debugging next to impossible!
Operated by Los Alamos National Security, LLC for NNSA
UNCLASSIFIED
xtUML – communication risky?
• Message passing (MP) is critical to xtUML
• MP is done using Windows message
queues or sockets
• A single communication failure kills the
whole system
• VxWorks 5 network stack obsolete
• Windows network stack documentation
limited (being generous here)
Operated by Los Alamos National Security, LLC for NNSA
UNCLASSIFIED
LOS move to Linux – MP problems!
• As Windows is so opaque and a single
user environment, xtUML system libraries
were ported to Linux
• Communication was implemented using a
typical client server model (TCP/IP
sockets)
• Linux and VxWorks 5 network stacks are
incompatible
Operated by Los Alamos National Security, LLC for NNSA
UNCLASSIFIED
LDAQ “design principles”
• VXI crate LDAQ processes should be
independent entities – fault tolerant
• No “host” computer controlling task
• Minimal code base
• Homogenous and state of the art
operating system environment
• Open source preferred – transparency,
quality & cost
Operated by Los Alamos National Security, LLC for NNSA
UNCLASSIFIED
What is LDAQ?
• Threw out all of LOS except hand coded
pieces that do VME communication
• Moved VXI CPU’s from VxWorks to Linux
• “Host” computer is Linux
• Wrote a minimal application that integrates
the VME calls – about 1 month of effort
• LDAQ runs independently as a user
application on each VXI crate
Operated by Los Alamos National Security, LLC for NNSA
UNCLASSIFIED
LDAQ visual description
LANL
MySQL
(on host)
Local Network
Computer
20 Hz
Start
Stop
LDAQ
ROC module
Ready
Data
Trigger
TOF units
Operated by Los Alamos National Security, LLC for NNSA
UNCLASSIFIED
LDAQ benefits versus LOS
• Code can be changed and recompiled in
seconds using Eclipse (> 1/2h for LOS)
• LDAQ failure on any VXI crate does not
affect others – can be restarted
• Remote cross-debugging using gdb
successful
• Linux is a true multi-user system (security)
• Open source – no licensing costs
Operated by Los Alamos National Security, LLC for NNSA
UNCLASSIFIED
Single Event and Histogram
• LDAQ does single event “out of the box” –
temporary NFS limitations
• Single event -> large amount of data, use
only when useful
• LOS histograms implemented in LDAQ
• “Live data” transfer call implemented
• Defined “time-focusing” histogram – e.g.
live GSAS coming very soon!
Operated by Los Alamos National Security, LLC for NNSA
UNCLASSIFIED
Experience with LDAQ
• Pharos has run LDAQ for most of this year –
only failures were power cuts!
• HIPD and more recently HIPPO are successfully
using LDAQ with legacy scripting
• LDAQ is a user application; changing SBC’s is
straightforward (e.g. PPC -> Intel) – cost, effort
• Restart takes seconds
• Infrequent/development VME bus “hangs” now
limiting reliability
Operated by Los Alamos National Security, LLC for NNSA
UNCLASSIFIED
Why is LDAQ successful?
• Each VXI crate is an independent agent –
does not know about others
• No controlling task
• Minimal features, implement bells and
whistles externally (e.g. run timing)
• Linux is robust and transparent
• Keep it simple!
Operated by Los Alamos National Security, LLC for NNSA
UNCLASSIFIED
LDAQ development - observations
• Exclusively using Linux - economical (up
front and manpower) – cultural challenges
• Does require significant Linux expertise
• MySQL straightforward, robust & free
• Agile development, develop features as
needed – requires IS collaboration
• Close collaboration between physical and
computer scientists
Operated by Los Alamos National Security, LLC for NNSA
UNCLASSIFIED
LDAQ - future
•
•
•
•
LDAQ development is essentially done!
Documentation
Needs external setup & control scripts
Scripting – leave to IS, provide, or port
existing control scripts?
• Integration with slow controls – EPICS
• Visualization toolkit
• Real time analysis
Operated by Los Alamos National Security, LLC for NNSA
UNCLASSIFIED