Transcript Padova, CHEP 2000

The Data Flow System of the ATLAS
DAQ/EF "-1" Prototype Project
G. Ambrosini 3,9, E. Arik 2, H.P. Beck 1 , S. Cetin 2, T. Conka 2, A. Fernandes 3, D. Francis3,
Y. Hasegawa 4, M. Joos 3, G. Lehmann 1,3 , J. Lopez 3,10, A. Mailov 2, L. Mapelli 3,
G. Mornacchi 3, Y. Nagasaka7, M. Niculescu 3,5, K. Nurdan 2, J. Petersen 3,
D. Prigent 3, J. Rochez 3, L. Tremblet3, G. Unel3 , S. Veneziano 3,6, Y. Yasu8
1. Laboratory for High Energy Physics, University of Bern, Switzerland
2. Department of Physics, Bogazici University, Istanbul, Turkey
3. CERN, Geneva, Switzerland
4. ICEPP, University of Tokio, Tokio, Japan
5. Institute of Atomic Physics,Bucharest, Romania
6. I.N.F.N. Sezione di Roma, Roma, Italy
7. Nagasaki Institute for Applied Science, Nagasaki, Japan
8. High Energy Accelerator Research Organization (KEK), Japan
9. Now at Lightning Instrumentation S.A., Lausanne, Switzerland
10. Now at EDF, Grenoble, France
2
The DAQ/EF “-1” Project
Study of a vertical slice of the ATLAS DAQ system to:
– define requirements on different sub-systems,
– design the elements of the DAQ with their boundaries
and their interaction with other components,
– implement a prototype to evaluate technological
solutions with the requested performance.
The project has been organized in 4 main activities:
–
–
–
–
Detector interface
Data Flow
Event Filter
Back-End (see talk by I. Soloviev on Thursday)
Padova, CHEP 2000
G. Lehmann, LHEP Bern / CERN
3
View of the Data Flow System
Detector Electronics
Front End DAQ
consisting of
read-out crates
Input Rate
L
T
D
AR
QG
L
T
D
AR
QG
E
B Readout
I Buffers
F
From trigger
systems (L2A, L2R, ROI)
Event Builder
Farm DAQ
consisting of
sub-farm crates
75 - 100 kHz
E
B Readout
I Buffers
F
To level 2
DFM
+LDAQ
L
DS
AF
QO
Switching Network
S EF Sub
F Farm
I
L
DS
AF
QO
Switch
Supervision
See talk by S.
Veneziano
Bandwidth
1- 2 kHz
4-5 GB/s
S EF Sub
F Farm
I
~100 MB/s
Mass Storage
Padova, CHEP 2000
G. Lehmann, LHEP Bern / CERN
4
Prototype
Implementation
of the
Data Flow
Padova, CHEP 2000
G. Lehmann, LHEP Bern / CERN
5
Global Performance Measurements
Event Building Rate = f (L2R)
•ROB fragment size variable, with
mean ~1.5 kBytes
2
1.5
1
•The EB dictates the performance
for a L2 rejection ratio below 95%
0.5
0
89
90
91
92
93
94
95
96
97
98
99
100
L2 reject factor
Event Rate = f (ROB fragment size)
•Measurements with no L2
rejection
•The performance is limited by
the EB interface which collects
fragments over VME and sends
them out on the ATM network.
3.5
Event Builder rate (kHz)
Event Building rate (kHz)
2.5
3
2x2 system
2.5
2
1.5
1
0.5
0
0
500
1000
1500
2000
2500
ROB fragment size (bytes)
Padova, CHEP 2000
G. Lehmann, LHEP Bern / CERN
3000
6
The Event Builder (EB)
• The EB is responsible for merging data fragments
to complete, formatted events
• The EB design is based on a two layer approach
which separates the technology specific aspects
from the functionality of the EB elements and their
interaction protocol
Src
• The EB has been
studied through
prototyping and
simulation
GetId
Bsy/NotBsy
DFM
EoT
EoE
Padova, CHEP 2000
Transfer
Dst
G. Lehmann, LHEP Bern / CERN
7
EB Performance Measurements:
Gigabit Ethernet
Processor:
Intel Pentium PC @
450 MHz
OS: Linux
Protocol: TCP/IP
Padova, CHEP 2000
G. Lehmann, LHEP Bern / CERN
8
EB Performance Measurements: ATM
ATM bandwidth
Data
Processor:
RIOII 8062 SBC @
200 MHz
OS: LynxOS 2.5.1
Protocol: AAL5
Padova, CHEP 2000
G. Lehmann, LHEP Bern / CERN
9
Modelling of the EB
• Main purpose: study of the scaling of the EB
system performance to ATLAS sizes
• Model design: 2 layer approach as in the
prototype in order to be capable of studying
different technologies with the same model
• Simulation Program: implementation with the
discrete event simulation domain of the PTOLEMY
(http://ptolemy.eecs.berkeley.edu) simulation tool
Padova, CHEP 2000
G. Lehmann, LHEP Bern / CERN
10
Network Node Scheme
Buffer
Check
a p p li c a t o n
User
CPUrunning the EB
Send buffer queue
NIC
Check queue
Check queue
Receive buffer queue
NIC
Flow of data
Flow control: checking of queues and requesting data
•Every EB element is a network node. Only the application
specific part distinguishes between DFM, EBIF and SFI.
•The Network is modelled as an ideal router introducing a
constant delay between input and output.
Padova, CHEP 2000
G. Lehmann, LHEP Bern / CERN
11
EB Scalability Studies
ATM
The time to build an event increases fairly linearly with the
number of EBIFs; nevertheless the data cannot exclude a
weak quadratic dependency yet.
Padova, CHEP 2000
G. Lehmann, LHEP Bern / CERN
12
Scaling of the EB
Results of the simulation calibrated with the processing times
and the ATM technology parameters measured in the
prototype.
Link speed
EB performance
ATLAS region
The performance is strongly dependent on the evolution of the
processing time as a function of the number of nodes.
Padova, CHEP 2000
G. Lehmann, LHEP Bern / CERN
13
Scaling: Possible Improvements
Reduce the number
of nodes in the
system by
introducing higher
bandwidth links
Increase the
processing power
of the nodes
Padova, CHEP 2000
G. Lehmann, LHEP Bern / CERN
14
Conclusions
• An Event Builder prototype has been implemented
for Gigabit Ethernet and ATM; the latter has been
used to calibrate a computer model of the EB.
• The model shows that the EB design is scalable
and that the required performance is in reach.
• A small but complete Data Flow prototype has
been designed and implemented; a LVL2 accept
rate of 2.3 kHz could be sustained for ROB
fragment sizes of ~1.5 kBytes.
Padova, CHEP 2000
G. Lehmann, LHEP Bern / CERN