Transcript Introduction to DAQ Architecture - Indico

Introduction to DAQ Architecture
Niko Neufeld
CERN / IPHE Lausanne
Features
•
Two types of data flows on the same network
•
•
•
One shared farm
•
•
•
•
Level 1: 5.5 kB @ 1 MHz
HLT: 40 kB @ 40 kHz
subdivided in ~ 90 sub-farms
contains ~ 2000 CPU nodes (single CPU)
Everything connected by Gigabit Ethernet (1000
BaseT) – use IP as transport protocol
Pure push protocol, with back-pressure
• local: Ethernet flow-control
• global: throttle signal via TFC
LHCb DAQ Architecture
Level-1
Traffic
126-224
Links
44 kHz
5.5-11.0 GB/s
Front-end Electronics
FE FE FE FE FE FE FE FE FE FE FE FE TRM
62-87 Switches Switch
Switch
Switch
Switch
Switch
Storage
System
L1-Decision
SFC
SFC
94-175
SFCs
Switch Switch Switch
Mixed Traffic
Sorter
TFC
System
94-175 Links
5.5-10 GB/s
SFC
HLT Traffic
29 Switches
32 Links
Readout Network
Level-1 Traffic
323
Links
40 kHz
1.6 GB/s
Multiplexing
Layer
64-137 Links
88 kHz
Gb Ethernet
HLT
Traffic
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
SFC
SFC
Switch
Switch
CPU
~2000
CPUs
CPU
CPU
CPU
CPU
CPU
CPU
Farm
Front-end board
•
•
•
•
•
•
Most FE boards are of
the same type: TELL1
connect to the detector
receive L0 and L1 trigger
from TTC
format data for L1 and
HLT and sends to DAQ
send throttle signal to
TFC if buffers threaten to
overflow
Some (e.g. RICH) do not
participate in L1
FE
FE
A-RxCard
A-RxCard
PP-FPGA
PP-FPGA
L1B
L1B
FE
FE
O-RxCard
PP-FPGA
PP-FPGA
L1B
L1B
Sync Li nk-FPGA
ECS
ECS
TTCrx
TTC
FEM
RO-Tx
L1T
HLT
L0 andL 1
Thro ttl e
Properties relevant for simulation
•
•
•
•
•
Latencies: zero
suppression, …
Adjustable water marks
(for throttle signals)
Separate data-paths
for L1 and HLT
Buffer sizes (L1-buffer,
possibly HLT formatting
buffer)
should be adaptable to
final specifications /
functionality of TELL1
FE
FE
A-RxCard
A-RxCard
PP-FPGA
PP-FPGA
L1B
L1B
FE
FE
O-RxCard
PP-FPGA
PP-FPGA
L1B
L1B
Sync Link-FPGA
ECS
ECS
TTCrx
TTC
FEM
RO-Tx
L1T
HLT
L0 andL 1
Throttl e
L0
Decision Uni/t
Generator
TFC (simplified)
•
Readout
Supervisor
TFC switch
L0 throttle switch
L1 Decision
Sorter
L1 throttle switch
•
TTCtx
TTCtx
TTCoc
TTCoc
TTCoc
TTCrx
TTCrx
TTCrx
TTCrx
L1
FE
VELO
L1 FE
L0 throttle OR
TTCrx
TTCrx
L1 throttle OR
L1
FE
VELO
L1 FE
TTCoc
L0
FE
VELO
VELO
VELO
L0
FE
L0
L0FE
FE
L0 Throttle OR
L0
FE
VELO
VELO
VELO
L0
FE
L0
L0FE
FE
TTCrx
TTCrx
TTCtx
TTCrx
TTCrx
TTCrx
TTCrx
L1 Throttle OR
TTCtx
Only blue
parts need to
be simulated
Decision
sorting and
RS only
complex
devices with
several
delays,
buffers, etc…
L1 Decision Sorter
•
•
•
L1 decisions must be sorted when they arrive
from the farm
L1 Decision Sorter delivers sorted decisions to
Readout Supervisor
L1DS checks for time-outs of decisions
Switching Network
•
•
•
•
•
•
•
•
Parameterised switch model
Nice to have parameters, which can be
measured (in our test-bed)
Mostly buffers and latency
Possibly back-pressure (flow-control) on
Ethernet links
Support for various frame sizes (MTUs)
Routing / switching on the basis of Ethernet or IP
Much more detail in Jean-Pierre’s presentation
Aggregation switches should be included in
simulation  they add a “hop”, possibly “smear”
arrival times at the main switch
Subfarm Controller
•
•
Performs event-building, checks for corruption,
time-outs, etc.
Level-1 events:
• if free worker CPU  forward, set timer for maximal
•
•
•
•
processing time
if decision received in time forward to Level 1 decision
sorter, else send default decision (programmable)
if no free worker CPU, buffer until either
• time-out (discard)
• CPU free – send onwards with time-penalty  events age in
the SFC buffer
HLT events:
•
send to free CPU  nodes buffer several events
Can issue throttle signal (via ECS  slow!)
Farm node
•
•
•
•
•
•
•
•
Runs algorithms for L1 and HLT
Generates time-out (when algorithm does not
terminate)
Buffers several HLT events
Has at maximum one L1 event at any given time
For HLT forwards events to SFC
For L1 forwards decision to SFC
Can fail, loop, time-out...
Communicates with SFC with a token-scheme
Summary
•
•
•
•
•
The DAQ system is designed to consist of
simple components of well-defined functionality
Most components will have a one-to-one
correspondence in the simulation
Details of TFC system irrelevant
Details of latencies and buffers in FE-boards and
farm very important
The system is scalable. Upgrades will add data
to the L1 stream (can double the total required
band-width)