Transcript fCIM

Department of Particle & Particle Astrophysics
Modular Data Acquisition
Introduction and applicability to LCLS DAQ
Michael Huffer, [email protected]
Stanford Linear Accelerator Center
December 14, 2006
Representing:
Ryan Herbst
Chris O’Grady
Amedeo Perazzo
Leonid Sapozhnikov
Eric Siskind
Dave Tarkington
Matt Weaver
Department of Particle & Particle Astrophysics
Outline
•
•
•
•
Introduction
– Concepts
– Architecture
– Implementation
Examples…
– Petabyte scale, low access latency storage for SLAC Computer Center
– LSST camera data acquisition system
Application design
Discuss applicability for LCLS Data Acquisition?
2
Department of Particle & Particle Astrophysics
The Module
•
•
Is the basic building block of the architecture
Specified as:
– A hardware design (schematics, BOM & layout guidelines)
– A series of base services implemented as:
•
•
•
•
VHDL (interfaced through core IP libraries)
Software (OO interface - provided through header files and shared libraries)
– documentation
Module neither specifies or constrains application’s physical partitioning model
Architecture specifies three different types of modules
– CEM (Cluster Element Module)
•
•
•
–
fCIM (Fast Cluster Interconnect Module)
•
–
Provides a processor + RTOS (the Cluster Element)
Provides many channels of generic, high speed, serial I/O
Provides commodity network interface (10 GE & 100-Base-T Ethernet)
Provides 10 GE connectivity for up to 64 Cluster Elements
sCIM (Slow Cluster Interconnect Module)
•
Provides 100 Base-T & 1 GE connectivity for up to 64 Cluster Elements
3
Department of Particle & Particle Astrophysics
Cluster Element Module (CEM)
Each lane
operates
up to 10 Gb/sec
•
•
•
•
PHYs (4-20)
PHYs (0-16)
May mix and
match lanes
to each CE
Two variants…
– One channel CE
– Two channel C2
A Cluster Element
(CE) is a
processor
Two variants…
footprint:
– ~ 50 cm2
power:
– ~ 7 watts total +
– ~ 3/4 Watt/port
CE
CE
CE
Common to
both elements
To sCIM
To fCIM
reset
JTAG reset
options
10 GE 100B-T
CEM (1 channel)
reset
JTAG reset
options
10 GE
10 GE 100B-T
100B-T
CEM (2 channel)
4
Department of Particle & Particle Astrophysics
Fast Cluster Interconnect Module (fCIM)
To CE
10 GE (0 – 8)
•
•
Is a collection
of managed
switches
footprint:
– ~ 144 cm2
Power:
– ~ 1 ½ Watt/port
– 64 elements ~ 110 watts
fCIM
10 GE (0 – 8)
Supports a variety of
electromechanical
standards
X2 & XENPACK MSA
CX4
Long haul & short haul
fibers
1 GE
To
management
network
5
Department of Particle & Particle Astrophysics
Slow Cluster Interconnect Module (sCIM)
To CE
100B-T (2 – 64)
•
•
footprint:
– TBD (less then fCIM)
Power:
– TBD (much less then fCIM)
Is a collection
of unmanaged
switches
sCIM
To
management
or control
network
1 GE
Supports a variety of
electromechanical
standards
6
Department of Particle & Particle Astrophysics
32 Element Cluster
1 GE
fCIM is
managed
control
network
To
management
or control
network
sCIM
CE CE CE CE CE CE CE CE CE CE CE CE CE CE CE CE CE CE CE CE CE CE CE CE CE CE CE CE CE CE CE CE
fCIM
To data network
10 GE
7
Department of Particle & Particle Astrophysics
CEM block diagram
Fabric clock
Right side
PPC-405
(450 MHZ)
Right side
MGT clock
FPGA (SOC)
200 DSPs
Lots of gates
Xilinx XC4VFX60
Right side
Memory
(512 Mbytes)
Micron RLDRAM II
Right side
Configuration
memory
128 Mbytes)
Samsung K9F5608
Right side
Multi-Gigabit Transceivers
(MGT)
8 lanes
Left side
MFD
Reset options
JTAG
Left side
100-baseT
Reset
8
Department of Particle & Particle Astrophysics
Base services provided by CEM
•
“Fat” Memory Subsystem
– 512 Mbytes of RAM
– Sustains 8 Gbytes/sec
– “Plug-In” DMA interface (PIC)
• Designed as a set of IP cores
• Designed to work in conjunction with MGT and protocol cores
•
•
•
•
•
•
•
Bootstrap loader (with up to 16 boot options and images)
Interface to configuration memory
Open Source R/T kernel (RTEMS)
10 GE Ethernet interface
100 base-T Ethernet interface
Full network stack
Utility software to manage I/O
9
Department of Particle & Particle Astrophysics
Extended services provided by CEM
•
Pretty Good Protocol (PGP)
– Physical interface is serial with 2 LVDS pairs/lane)
– Point-to-Point connectivity
– Allows clock recovery
– Full duplex
• Symmetric capabilities in either direction from either end
– Provides reliable frame (packet) transmission and reception
– Deterministic (and small) latency
• Lightweight “on the wire” overhead
• Specifies 4 VCs in order to provide QOS
– Implemented as an IP core
• Small footprint
• Interface hides user from protocol details and implementation
• Implemented on CE (through the conical model described above)
– Asynchronous
• Extensible in both bit-rate and # of lanes
•
Flash Memory Module (FSM)
– Provides as much as 256 Bytes/CE of persistent storage
– Low latency/high bandwidth access(1 Gbyte/sec)
– Interfaced using PGP
10
Department of Particle & Particle Astrophysics
Cluster Element as used in petacache
65 Gbyte flash memory
(Flash Storage Module)
FSM
FSM
FSM
FSM
Application specific
PGP
1 lane @
250
Mbytes/se
c
PGP core &
interface
CE
10 GE
Called a SAM
(Storage Access Module)
To/From fCIM
To client
nodes on
client network
100B-T
To/From sCIM
From
management
network
11
Department of Particle & Particle Astrophysics
Cluster Element as used in LSST DAQ
In cryostat
Raft Readout
System
Services 9 CCD mosaic
288 Mbytes/sec
(Replicated 25 times)
Application specific
PGP (fiber-300 M)
1 lane @ 300
Mbytes/sec
PGP core &
interface
CE
10 GE
Called a RNA
(Raft Network Adapter)
To/From fCIM
To client
nodes on DAQ
network
100B-T
To/From sCIM
From Camera Control System
on CCS network
12
Department of Particle & Particle Astrophysics
The “Chassis”
1U
Air-Outlet
1U
Fan-Tray
High-Speed Network Card
(8U)
Passive
Backplane
8U
X2
(XENPACK MSA)
Accepts
DC power
1U
Air-Inlet
Daughter board Card (4U)
13
Department of Particle & Particle Astrophysics
Chassis Physical Interfaces (19”)
client network card
10 GE
CCS network
card
1 GE
to odd raft
to even raft
Guider
Array (2)
Science Array (12)
WFS
Array (2)
bank
8U
Number
is
TBD
to DAQ network
to CCS
Daughter cards replicated twice for:
Redundancy &
simulation
14
Department of Particle & Particle Astrophysics
A prescription for application design
•
•
•
Partition problem into three domains:
– Device/sensor specific Read-Out (RO)
The later two are within
the realm of the CE
– Device/sensor monitoring and configuration
– Data transport and processing
Define a consistent and regular interface between RO & CE systems
– independent of device/sensor
Define CE customization
– How many lanes of I/O necessary between RO and CE?
– What are the protocols on these lanes?
– Specify data processing
• How should this processing be partitioned between software and hardware?
•
CE number
– What is the underlying, inherent, parallelism of the data (if any)?
– How many CPU cycles and gates should be dedicated per data byte?
• processing effort/byte
•
Define physical partitioning of design
– How many boards?
– What type and number of modules on a board?
– Incorporate with custom logic?
15
Department of Particle & Particle Astrophysics
Typical usage patterns
RO
RO
RO
RO
•
•
•
Many different types of devices
Physically separated
Processing/byte/device is low
•
•
•
Many different types of devices
Physically separated
Processing/byte/device is high
•
•
•
Homogeneous devices
Perhaps physically separated
Processing/byte is high
CE
RO
RO
RO
RO
CE
CE
CE
CE
RO
CE
CE
CE
CE
16