Transcript need to encode 8b/10b or HDLC data

FELIX Design
Upgrades of detector readout meeting
9 June 2014
Lorne Levinson, for the FELIX group
L. Levinson
Upgrades of detector readout meeting, 9 July 2014
1
Guiding principles
• Goal: a GBT to LAN “switch” with as little awareness of detector specifics
as possible
– No user code in FELIX (so far): configuration params for options
• Separate data movement from data processing
• Configurable data paths from E-link to network endpoint
• Scalable at Front end, Back end, and switch levels
• Do as much as possible in software
• COTS FPGA PCIe board
– But… see talk on TTC clock recovery by Jinlong Zhang
• Readout, so latency is not important,
but provide low latency GBT-to-GBT links
• Future proof by following COTS market, separately upgradeable
components:
GBT, FPGA PCIe board, serve PC, network cards
• Routing is for E-links; they are the logical connections
L. Levinson
Upgrades of detector readout meeting, 9 July 2014
2
Virtex-7 PCIe board
From Global HiTech, PCIe gen 3, 8 lanes
Virtex X690T
2x12 bidir CXP connectors, FMC
L. Levinson
Upgrades of detector readout meeting, 9 July
2014
3
Off- detector architecture with FELIX
• Functions can now be
separated, physically,
or just logically.
• “ROD” = Event fragment
builder/processor for ROS
L. Levinson
Upgrades of detector readout meeting, 9 July 2014
“ROD”
4
L. Levinson
Upgrades of detector readout meeting, 9 July 2014
5
Routing
• Configured via control port on main or separate network
• Simple/fixed:
FELIX port # / E-link-range → IPaddress / port
or
→ Infiniband local identifier / queue pair
• E-link packet protocol:
– Allows several logical streams to use the same E-link
 header with: stream-id + packet len (if not fixed length packets)
FELIX port # / stream-id → IPaddress / port…
• Cloning, broadcasts / multicasts possible, in either direction
• Reliable transport from FELIX to end-point via TCPIP
• Configurable Quality of Service per logical connection
– Less-than-best-effort for event data for monitoring
L. Levinson
Upgrades of detector readout meeting, 9 July 2014
6
Use cases – 1
1.
Receive data from an E-link or bonded group of E-links (a channel) and
route it to one or more network endpoints via TCP or UDP. Bits must be
arranged into correct order.
– Optional 8b/10b or HDLC checking and/or decoding
– Optionally verify checksum
– Packet boundary identification either by SOP/EOP or length, optionally
there can be no packet boundary, just a byte stream.
• If a maximum packet length is configured, data arriving after the
maximum is discarded and this error is marked in the packet status
word.
– If streams are defined for this channel packets are routed according to
the stream.
The stream ID may contain an extension filled by the detector with
part of the Level-1 ID or the Trigger Type for advanced event routing.
– Sampling can be requested
L. Levinson
Upgrades of detector readout meeting, 9 July 2014
7
L. Levinson
Upgrades of detector readout meeting, 9 July
2014
8
Use cases – 2
2.
Receive BUSY from an E-link command, internally from FELIX itself, or
from the network. Then route it to a network endpoint or to a standard
ATLAS RODBUSY signal (open collector Lemo). In Phase II it may be sent
to the TTC system.
3.
Receive TTC information (BC, L1A, ECR, BCR, user bits) and broadcast
them to the requested E-links
– The BC clock is supplied to the GBT because the FELIX's GBT link clock
is the BC clock supplied by the TTC system to FELIX.
– Three options: 2-bit, 4-bit, 8-bit as described in the datasheet.
Receive data from a TCP port and route it to one or more E-links
– Optional 8b/10b or HDLC encoding
– A specific multicast channel can broadcast packets to all out-going Elinks with the "MC" attribute.
4.
L. Levinson
Upgrades of detector readout meeting, 9 July 2014
9
Use case – 3
8.
Scheduled data transfer lists: setting and activating (list is per GBT, but Elinks are enabled by a bit map.)
Open Issue: Or, several lists per FELIX and each E-link configures to use a
specific list. Each item in the list is qualified by a delay to wait from the
previous item in the list.
– Activation by TTC user bit
– Activation by command received via TCP
9.
Receive data from an E-link or bonded group of E-links (a channel) and
route it to a Direct Output Low Latency link (probably a GBT). Typically
only a particular stream in the channel would be so routed.
L. Levinson
Upgrades of detector readout meeting, 9 July 2014
10
Low latency path is
between predefined
GBT pairs.
Uses “cut-through”
Routing.
L. Levinson
Upgrades of detector readout meeting, 9 July
2014
11
FELIX: FW/SW interface
Data interface
• PCIe low level packet format (fixed length)
• PCIe high level packets, spans low level packets
• Push/pull? Hints from RobinNP architecture
Configuration parameter interface
• Need description of parameters that can generate:
XML for OKS, VHDL for FPGA
• See (but needs to be updated):
https://twiki.cern.ch/twiki/pub/Atlas/GBT2LAN/MappingGBT_DataToNetworkEndpoints.pdf
Monitoring and statistics interface
FPGA configuration interface
• IP core from HitechGlobal ~4k$
L. Levinson
Upgrades of detector readout meeting, 9 July 2014
12
L. Levinson
Upgrades of detector readout meeting, 9 July
2014
13
L. Levinson
Upgrades of detector readout meeting, 9 July
2014
14
Backup
L. Levinson
Upgrades of detector readout meeting, 9 July
2014
15
Functionality of E-lane handlers
From GBT: need to decode 8b/10 or HDLC data
• Collect 2, 4, 8, 16 bits per 40MHz clock
• Align using comma symbols
• Decode 8b/10b, discard comma, Start-of-Packet, End-of-Packet
• Add 1KB block headers and “chunk” trailers
• Optionally verify checksum
To GBT: need to encode 8b/10b or HDLC data
• Read from FIFO
• Encode (8b/10b(?) or HDLC) 2 bytes at a time from FIFO
• Chop into 2, 4, 8, 16 bits per 40MHz clock
Probably run at 80MHz to get extra clocks for headers/trailers
L. Levinson
Upgrades of detector readout meeting, 9 July 2014
16
Clock recovery
• The 2x40MHz clock comes from the TTC link, i.e. a recovered clock.
• Probably OK for GTH’s in the same GTH block of 4 transceivers
• ???
– Can it be distributed to the other blocks?
– Does the recovered clock have sufficiently low jitter for the other
GTH’s?
• We should probably consider a TTC receiver and jitter remover on an FMC
card.
– rework the GLIB version so that the FMC is on top of the PCIe mother
board instead of extending beyond the PCIe board boundary
L. Levinson
Upgrades of detector readout meeting, 9 July 2014
17
Error handling
• Just beginning the considerable thought that needs to be done here.
• Two issues:
– Discovery of corrupted data
– Surviving errors without out loosing event boundaries
• This requires some effort on the front ends
• Corruption within a GBT 80-bit word is detected by the GBT’s FEC
• Hard to believe E-links can be mixed into each other,
so expect errors within E-link channels: repeated clocks, lost clocks
• If E-links use 8b/10b with packet framing codes, fixed length, or variable
length with counts, then errors (not all) can be detected.
• FELIX can flag these errors in the header it sends with a stream’s data
L. Levinson
Upgrades of detector readout meeting, 9 July 2014
18