Transcript ADS_CS_reliabilityx

Control System
Considerations for ADS
EuCARD-2/MAX Accelerators for Accelerator Driven Systems
Workshop, CERN, March 20-21, 2014
Klemen Žagar <[email protected]>
Robert Modic <[email protected]>
Mark Pleško <[email protected]>
2
High Availability
 Fault tolerance and redundancy of the accelerator
use of components far from their limits,
 parallel and serial redundancy of components,
 ability to repair failing section.

 Control strategies for high availability
Reliable components in the first place
 Redundant elements
 Protection systems without false positives
 Predicting faults before they occur
 Working around faulty equipment

3
Standard CS Architecture
Central Services
Gateway
Model
Central Room
Operator’s workstation
Archive
Other networks
IP over Ethernet
Power Supplies Sector A
Control Box
RF Sector B
Control Box
Equipment Interface – Control Boxes
4
Planning: work breakdown
5
Hardware platform
 Considerations:
Maturity
 Performance
 Use in other facilities
 Obsolescence management

 Today’s choices:
VME [mature, nearing obsolescence]
 cPCI [suboptimal performance; cPCIe immature]
 PXI, PXIe [limited choice of vendors]
 mTCA/ATCA, mTCA.4 for physics
[not much support from industry – yet]

6
Software Framework
 We recommend EPICS as the control system
infrastructure.
Widely used in ACC community.
 Good community and commercial support.
 Significant reuse of existing components possible.
 Mature and proven technology.
 Hooks allow implementation of a redundancy scheme.

7
About EPICS
Channel Access Client
Channel Access Server
(IOC)
Channel Access Client
Sub-system
Process Variables:
CWS-PHTS-DLHT:VC1FCVZ
Computer
Interface
CWS-PHTS-DLHT:VC1-FCVY1
CWS-PHTS-DLHT:VC1-FCVY2
Computer
Interface
CWS-PHTS-DLHT:MT2-TT
Computer
Interface
Flow
Control
Valve
Thermometer
8
EPICS Data Flow
 The Channel Access network communication protocol.
UDP for discovery.
 TCP for data exchange.

“connection request”
or “search request”
“get” or
“put” or
“caGet”
“caPut”
Who has a PV named
“CWS-PHTS-DLHT:TTSPTARGET”?
Change its
value to 30.5
What is its
value?
Channel Access Client
“set a
monitor”
Notify me
when the
value
changes
CA Client
CA Server
Channel Access Server
Process Variables:
CWS-PHTS-DLHT:VC1-FCVZ
CWS-PHTS-DLHT:VC1-FCVY1
CWS-PHTS-DLHT:VC1-FCVY2
CWS-PHTS-DLHT:TTSPTARGET
I do.
25.5
degC
OK, it
is now
30.5
“put complete”
It is now
20.5 degC
It is now
10.3 degC
It is now
9.2 degC
“post an event”
or
“post a monitor”
9
EPICS and redundancy
 One of the IOCs is a primary, and one is a backup.
 Primary IOC sends all state changes (e.g., changes of
values) to the backup to keep it in sync.
 if heartbeat fails, backup node takes over, in the same
state where the primary left off.
Equipment interfaces
Valve
Pump
RPM
Pump
Pump
power
DO
DI
AI
Field
bus
EPICS IOC
Valve
open/close
Valve
state
EPICS Channel Access
 How to integrate equipment:
Local controller
AO
Equipment
Equipment
IOC 1
IOC 2
Enable / OK signal (IOC®equipment)
Actuator signals (IOC®equipment)
Sensor signals (equipment®IOC)
EPICS Channel Access
 Redundancy?
Myrrha
Demux
Responsibility of equipment supplier
or
10
11
Use And Integration Of PLCs
 Logic neither complex nor very fast (>10ms)  robust.
 Used in off-the-shelf industrial systems

Cryo plant, vacuum, building automation/HVAC, …
 Used for personnel protection (interlocks).
HMI
Alarms
Archives
Supervision
Communication
Processing
I/O
monitor
PLC
put
get
PLC
IOC
Channel
Access
12
Industrial Redundant Systems
 PLCs

implement redundancy in the CPU and with redundant hot
swappable IO modules.
 Network switches
Predefining routing tables on nodes and switches
 This way communication can resume more quickly after
switchover

13
Machine protection system
 Multiple levels of protection:
Hardwired protection system.
Required for nuclear safety.
 Personnel protection system.
 Machine/investment protection.
Quick reaction to faults. Graceful shutdown.

 The first two are outside the scope of control system.
But can be integrated with it (e.g., via 4-20mA signal
interface).
 MPS issues a mitigation action when a problem is detected.

 Topology:
Source
Source
LEBT
LEBT
RFQ
RFQ
NC DTL
S C DTL
S C DTL
NC DTL
Spokes, β=0.35
Elliptical, Medium β=0.47
Elliptical, High β=0.65
Spare part
Dump
Reactor
14
Machine Protection is Redundant
to Control System
Control System Services
Control Room
Control System Network
(Ethernet)
Timing System
EPICS IOC
Device Network
(Ethernet)
Trigger(s)
Machine Protection System
Ion Source controller
15
Machine protection system
CONTROL SYSTEM
(Configuration &Supervision)
RPS
(Run Permit
System)
FDS
(Fault-Diagnostic
System)
MID
(MPS Input
Devices)
MOD
BIS
(Beam Interlock System)
TIMING
SYSTEM
Post
Mortem
(MPS Output
Devices)
16
Predictive diagnostics
 Statistical analysis of archived data (e.g., trends) to
identify components nearing a fault.
 Model and detailed monitoring of subsystems.

E.g., monitoring of vibrations in mechanical subsystems.
 Uses:
Preventive maintenance planning.
 Preventively taking a component off-line.

17
Virtual accelerator
 Simulator of the machine.
 Uses real-time configuration data of beamline elements
to simulate beam characteristics.
 Useful to analyze failure scenarios.
 An R&D topic: automatic reconfiguration in case of a
subsystem failure.
18
Key recommendations
1.
Initiate collaboration on control system with similar projects.
2.
Introduce a naming convention early in the project.
3.
Standardize and define control system interfaces for all delivered
components and devices at the time of procurement.
5.
Equip RFQ@UCL with fully functional and stable control system for its
operation.
4.
Foresee time and resources for reliability and availability investigation
on RFQ@UCL.
5.
Define the scope of the control system well – if subsystems don’t have
a control system, foresee that it needs to be developed.
QUESTIONS
20
Alarms
 Supervision of alarm state.
 Guides operator in reacting to alarms.
 E.g., BEAST.

Part of the Control System Studio suite.
21
Archiving
 Storing values of process variables (PVs) through time.
 Usage:
Monitoring (and analysis) of (mid-/long-)term trends.
 Predictive diagnostics.
 Comparison of performance at various times.

 E.g., BEAUTY.

Part of Control System Studio.
 Not a high-performance
scientific archiving tool!
22
Timing system
RF Clock
Timing Generator
RF Clock Generator
Timing
sequences
TS transport layer core
Clock + Data
Switch / Fan-out
Crate
Crate
Crate
TS TL core
Client
device
Timing
Receiver
Response
generation
Client device
TS TL core
Client
device
Timing
Receiver
Client
device
Response
generation
Client device
Equipment interfaces
Other
subsystems,
central
services,
control room,
etc.
 The Control Box
Ethernet Switch
Ethernet/IP
Responsibility of the ESS integrators
Responsibility of the subsystem
developers/integrators
Input Output Controller (IOC)
Channel Access
EPICS IOC
Device
Support
Device
Support
Device
Support
analog inputs
23
A/D module
Ethernet Switch
PLC
PLC
PLC
analog/digital I/O
PROFINET
TCP/IP, UDP/IP
Ethernet Switch
Intelligent Controller
Intelligent Controller
Intelligent Controller
analog/digital I/O
24
ITER CODAC
 Packaging of control system software.
Operating system.
 EPICS.
 User interface tools.

 In addition, ITER-specific tools

E.g., Self Description Data toolkit for providing meta-data
and development of “plant system instrumentation &
control”.
 Can be used elsewhere as a baseline

E.g., ESS.