Transcript General-Purpose Framework for Real Time Control in Nuclear

RFX-mod Programme Workshop 2011 – Padova, February 7th 2011
RFX-mod
Programme Workshop 2011
RFX-mod
Feedback Control System
Upgrade
G. Manduchi, A. Barbalace
Consorzio RFX – Padova, February 7th 2011
Current Control System Architecture
MHD Mode Control
Plasma Position Control
Toroidal Field Control
Consorzio RFX – Padova, February 7th 2011
Main Features of the Current System
Distributed
• To account for different location of sensors
and actuators;
• to distribute computing load.
• Advantages: computing load can be spread
among computers. Sensor and actuators
can be dislocated around the experiment.
• Drawbacks: network connections introduce
delays in system response. The system must
be pipelined to achieves higher throughput.
Modular
• The same basic software used on all
nodes;
• only control algorithms are changed.
• Advantages: modularity proved to be a
successful approach.
• Drawbacks: a more comprehensive
software Framework is required to handle
complex systems
Configurable
• Algorithm updates and IO upgrades can be
fulfilled by editing a C header file
(recompilation is required).
Consorzio RFX – Padova, February 7th 2011
• Advantages: easy configuration proved to
be a key factor in reducing development time
• Drawbacks: reconfiguration need
recompilation.
Critical Factors
Latency
• Current overall latency is around 1.5 ms
(between 3 ÷ 4 sampling periods).
• This represents a critical factor in quality of
control sometimes leading to instabilities.
Sampling frequency
• Current sampling frequency is 2.5 kHz.
• A higher sampling rate improves the quality
of integration/derivation.
Computing Power
• Currently only most significant modes are
considered.
• Operations such as sideband correction and
sensor radius extrapolation are highly
computing-intensive and require more power
Simulation
• Currently it is not possible to inject a
dummy signal to test the control system.
Consorzio RFX – Padova, February 7th 2011
• The possibility of dry runs of the system,
automatically performed before the shot,
allows the detection of possible malfunctions
which may affect plasma performance and
machine integrity.
New Control System Design
processor
switch
switch
from Distributed processing..
• To account for different location of sensors
and actuators.
..to Centralized processing
• Sensors and actuators are still displaced
around the system;
• PCIe Bus Extenders (optical fiber or copper
cable) will provide direct connection to the
central processing (no Reflective Memory).
There is no latency due to
data distribution over a network
• To distribute computing load.
Consorzio RFX – Padova, February 7th 2011
• Multicore/processor technology provides all
the required computing power;
• Modularity is retained: the work carried out by
the former embedded computer it is now
executed by a dedicated processing core.
New Control System Architecture
Consorzio RFX – Padova, February 7th 2011
Evaluated Hardware Details
Data Acquisition
• We have evaluated National Instruments PXI6255; a non simultaneous sampling
multiplexed 16bit ADC with 80 analog inputs.
• Each acquisition node is made up with a PXI
rack hosting 3 PXI-6255 boards. Every board
is setup to scan 64 channels periodically.
• A scan completes in 90us and data is ready to
the central processing after few microseconds.
• Data transfer is mastered by the devices via
Direct Memory Access (DMA).
Consorzio RFX – Padova, February 7th 2011
Waveform Generation
• We have evaluated National Instruments PXI6723; a 13bit multi Digital to Analog converter
with 32 analog outputs.
• Each reference generation node is made up
with a PXI rack hosting 3 PXI-6723 boards.
Every output is used.
• Data generation completes after 60us (Intel
QPI bus) or after 120us (Intel FSB bus).
• Data transfer is governed by the processing
core via Programmed I/O.
Non Simultaneous Sampling
Is it possible to realign the samples?
Yes, because channels are sampled at known instants.
In Real-Time
After the Pulse (offline)
• The reconstruction will lead to a quasi
exact low-frequency signal and a less
precise high frequency signal.
• The acquisition downgrade to 14bit.
• Data is filtered via a Fractional Delay FIR
implementing Lagrangian interpolation
• The reconstruction will lead to a
theoretically exact signal.
..but multiplexing introduces also
time issues
• Allsettling
16bit could be restored.
• Data is convolved with a sinc function.
due to switching between different channels
Frequency spectrum of a sample DFLU signal
Consorzio RFX – Padova, February 7th 2011
Simultaneous Sampling
Solutions alternative to evaluated hardware
National Instruments
• Data bus PXI Express (PCIe) in PXI rack;
• PXIe-6358 16 input channel boards, 16bit;
• Each 192-channel acquisition node may be
implemented with a PXIe-1075 chassis with
12 (18 available) PXIe-6348 boards.
IST-IPFN
• Data bus PCI Express in ATCA rack;
• Custom devel 32 input channelboards, 24bit;
• Each 192-channel acquisition node may be
housed in an ATCA shelf with 6 custom
developed ATCA-MIMO-ISOL boards.
Both focus on high data throughput
IPFN solution is designed for high availability
(five nines)
Consorzio RFX – Padova, February 7th 2011
New Software Framework
What about the previous Framework?
• The software framework developed and currently used in RFX did a good job.
• However, it has been developed with a well-defined architecture in mind, so it
is not so easily portable to new architectures and platforms.
Joint efforts in fusion community gave rise to..
• General frameworks for real-time control, notably PCS and MARTe.
• PCS has been developed at DIII-D and is used in a few machines (e.g. DIII-D
, MAST, EAST).
• MARTe has been developed at JET (by the PPCC group in collaboration also
with RFX) and is also used in few machines (e.g. JET, FTU, ISTTOK,
COMPASS).
MARTe has been chosen!
Consorzio RFX – Padova, February 7th 2011
New Operating System
The current system uses Wind
River VxWorks
• Rugged real-time operating system;
• Some important bugs in TCP/IP
communication;
• Very expensive.
http://www.windriver.com
http://www.rtai.org
We have chosen Linux with its
Linux is becoming an interesting
officially
released
RT patches
candidate
for the new
system
http://www.xenomai.org
• Soft real-time system;
• Real-time extensions of Linux make
it a hard real-time low-latency
operating system;
• Linux’s official RT patches are good
enough to make it hard real-time OS.
Consorzio RFX – Padova, February 7th 2011
http://www.kernel.org
http://rt.wiki.kernel.org
Pipelined Multicore Execution
Real time computation for MHD control is carried out in a pipelined
organization with three stages: Data Acquisition, Control
Computation, and Reference Waveform Generation.
New cores can be added to provide additional computation in the case
it can be performed in parallel.
acq Clock 0
acq Clock 1
acq Clock 2
acq Clock 3
Software
PIPELINE
CPU
core 0
CPU
core 1
ADC
stage
CPU
core 2
CPU
core 3
...
CPU
core 4
...
CPU
core 5
...
Consorzio RFX – Padova, February 7th 2011
data out 0
data out 1
time
algorithmic
stage
DAC
stage
Realtime Performance
8kHz data acquisition Clock Frequency
Prototype running:
• A simple two stage pipelined I/O cycle
developed in MARTe;
• RedHat Enterprise Linux 5.5 with RT
patches from CERN (Scientific Linux
Distribution) kernel 2.6.24.7-149.el5rt;
• Multicore Intel Xeon processor E5500 @
2GHz (4 HT cores x86 64bit) QPI bus;
Prototype setup:
• A PXI input rack hosting two NI PXI-6255
acquiring 64*2 input channels;
• A PXI output rack hosting two NI PXI-6723
generating 32*2 output references.
Consorzio RFX – Padova, February 7th 2011
Measured
Overall latency
150us (average)
200us (WCET)
Costs Comparison
racks
Qty Unit Price Total
network Qty Unit Price Total
IO
Qty Unit Price Total
cabling
Qty Unit Price Total
Computing Qty Unit Price Total
PXI-1033
4
1110
4440 PXI-PCI8336 1
3079
3079 PXI-6255
9
2271
20439 SHC68-68-EPM
9
130
1170 server
1
4336
4336
PXI-1031
1
989
989
0 PXI-6723
6
1362
8172 SHC68-68
9
90
810
0
rack mount 5
161
805
0
0 SH68-c68-s 12
100
1200
0
0
0
0
0
0
6234
3079
28611
3180
4336
racks
network
IO
cabling
computing
total
6234
3079
28611
3180
4336
computing
10%
racks
14%
cabling
7%
netw ork
7%
45.440
NI Prototype Cost
IO
62%
racks
Qty Unit Price Total
VME
5
racks
network
IO
cabling
computing
RTOS
total
4500
22500
5000
127300
3180
20000
20000
network Qty Unit Price Total
IO
0 budgetary
1
5000
5000 VGX
0
0 MPV914
22500
0 FPDP
0
0 MPV922
22500
5000
Qty Unit Price Total
9
8700
78300
6
4000
24000
5
4000
20000
5
1000
5000
127300
RTOS
10% racks
11%
computing
10%
cabling
Qty Unit Price Total
Computing Qty Unit Price Total
SHC68-68-EPM
9
130
1170 CPU
5
4000
20000
SHC68-68
9
90
810
0
SH68-c68-s 12
100
1200
0
0
0
3180
20000
netw ork
3%
cabling
2%
197.980
IO
64%
Consorzio RFX – Padova, February 7th 2011
Old System Cost
Commissioning Plan (1 of 2)
Mailstone 1 - Deadline 30 March 2011 Test of NI Prototype in RFX Experimental Campaign
Objectives
• This installation will assess the usability of the evaluated hardware for data
acquisition (checking cross-talk and cross-channel switching problems).
• Investigate the possibility of numerical integration (both integrated and non
integrated signals will be acquired).
Procedure
• A first prototype system is being installed for acquiring poloidal field values
Requirements
• Front-end duplication
Costs
• All the computing and acquisition components are available in house.
– The support computer is recycled (there are no real-time computing to
implement on)
– The IO boards were used in the prototype
• Front end boards (i.e. signal splitters) must be designed and made.
Consorzio RFX – Padova, February 7th 2011
Commissioning Plan (2 of 2)
Mailstone 2 - Deadline TO BE DEFINED MHD Feedback System Commissioning
Objectives
• Run the new MHD Control System in parallel with the current System.
• The new system will not interfere with RFX operations.
Procedure
• The new system will be installed in parallel to the current one using signal
splitters.
Requirements
• Multicore server;
• IO boards, cabling and racks;
• Development of few software components.
Costs
• Multicore server;
• IO boards, cabling and racks;
• Front end boards (i.e. signal splitters).
Consorzio RFX – Padova, February 7th 2011
Conclusions
switch
• The new Feedback Control System of RFX proposes an
innovative hardware architecture in Fusion Devices;
distributed nodes communicating via a real-time network
(ATM, Ethernet, Reflective Memory) moved to
multiprocessor cores and communication is achieved on
the same RAM via shared memory.
processor
switch
• COTS solutions for computing (multicore servers) ADC
and DAC (PXI-based solutions with bus extenders)
proved cost-effective and reliable.
• Linux is becoming a very good candidate for real-time
feedback control systems in fusion devices.
• MARTe proved to be an extremely flexible environment
to develop Hard Real-Time Control Systems.
Consorzio RFX – Padova, February 7th 2011