Transcript CSC_Status - CMS-EMU SliceTest

CSC Status Report
Status reports;
Issues with System reliability and Maintainability (LV, HV,
Electronics modules and crates, internal interfaces)
External Interface reliability (Cooling, Power, TTC, DAQ)
Prepared for CSC by
Fred Borcherding FNAL
6-Nov-08
ESSC Nov 2008 > CSC Status Report
by Fred B.
1
Overview
•
Status of Electronics > Installed / Total at CERN / Good Spares
– Status of Crates and boards, how many are needed, how many are at CERN and
how many are good spares
•
Issues with System reliability and Maintainability
– LV > Wiener Maraton system (air cooled)
• Problems with OPFC modules
• CANBus communication problems with Maraton units
– HV > Resistors
– Electronics modules and crates
• On-chamber Electronics
• Other Electronics
– internal interfaces
•
External Interface reliability
–
–
–
–
6-Nov-08
Cooling
Power
TTC
DAQ
ESSC Nov 2008 > CSC Status Report
by Fred B.
2
Status of Electronics > Installed
• Status of Electronics > Installed / Total at
CERN / Good Spares
– Status of Crates and boards, how many are
needed, how many are at CERN and how many
are good spares
– The following few slides are included for
completeness
6-Nov-08
ESSC Nov 2008 > CSC Status Report
by Fred B.
3
On-Chamber Boards
•
CSC On-chamber Electronics Boards
installed /total /spares
•
•
•
•
•
•
•
•
•
CFE
ALC
ALC
ALC
ALC
ALM
LVD
LVM
AFE
2268 /?? /19
72 /?? / ??
72 /?? /10
214/?? /11
108 /?? /4
468 /?? /??
468 /?? /??
468 /?? /??
?? /?? /??
•
NOTE: Have ~18 spare chambers, each with full complement of on-chamber boards – these are not in
the sums above
6-Nov-08
30520306050001xxxxx
30520112030001xxxxx
30520112030288xxxxx
30520112030384xxxxx
30520112030672xxxxx
30520112130001xxxxx
30521222040001xxxxx
30521222130001xxxxx
30520106050001xxxxx
CFEB one type, 5(4) per chamber
ALCT ME1_1
ALCT288 ME1_3
ALCT384 ME1_2, ME234_2
ALCT672 ME2_1, ME3_1, ME4_1
ALCT_MEZ one type, 1 per chamber
LVDB one type, 1 per chamber
LVMB one type, 1 per chamber
AFEB one type, many per chamber
ESSC Nov 2008 > CSC Status Report
by Fred B.
4
LV & PCRATE
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
CSC LV Power Supplies > Wiener Maraton System, Air Cooled
PFCB
LVPS 72500xxxx
OPFC Bin (Crate)
PFCM
LVPS 71960xxxx
OPFC Module
MARS
LVPS 72920xxxx
Maraton Supply
MARC
LVPS 72960xxxx
Maraton Crate
Peripheral Crate, pcrate, Components
EMUPC
PC 305216030320050xxxx pcrate crate
CBP
PC 305203021620050xxxx pcrate back plane
CRB
PC 305203180220050xxxx pcrate regulator brd
PCM
PC 305216031320060xxxx PCMB
ELM
PC 305205121320050xxxx ELMB on PCMB
Pcrate boards
VCC
PC 305222030320050xxxx Crate Controller
CCB
PC 305203030220050xxxx Clock and Control
DMB
PC 305204130220040xxxx DAQ Mother Brd.
MPC Trigger
PC 305213160320050xxxx Muon Port Card
TMB Trigger
PC 305220130220050xxxx Trig Mother Brd.
RAT Trigger
PC 305218012020050xxxx RPC ALCT Trns.
6-Nov-08
ESSC Nov 2008 > CSC Status Report
by Fred B.
installed /total /spares
6 /8 /2
36 /40 /4
36 /40 /4
36 /38 /2
installed /total /spares
75 /60 /15
70 /60 /10
88 /60 /28
79 /60 /19
79 /60 /19
installed /total /spares
73 /60 /15
84 /60 /24
550 /468 /82
70 /60 /10
570 /468 /102
690 /468 /222
5
CSC HV
•
•
•
HV ME1/1
Cc
HV1
Cc
HV1
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
HV
HV
HDS
HDL
HDC
HDP
HMB
HMC
HPS
HCC
HLV
HMP
HLP
HCP
HPC
6-Nov-08
> CAEN System
3052##
Crate
3052##
Board
> UF Custom System
UF Custom System
installed /total /spares
30520804192005xxxxx Distribution-36
30520804122005xxxxx Distribution-30
30520804032005xxxxx Distrib Crate
30520804162005xxxxx Distrib Patch
30520813022005xxxxx Master board
30520813032005xxxxx Master Crate
30520816192005xxxxx Primary HV src
30520803032005xxxxx Control comp
30520812222005xxxxx LV power supply
30520813162005xxxxx Master Patch
30520812162005xxxxx LV patch panel
30520803162005xxxxx Control patch
30520816032005xxxxx Primary HVPS ctr
ESSC Nov 2008 > CSC Status Report
by Fred B.
installed /total /spares
2 /2 /0
16 /18 /2
installed /total /spares
126 /159 /33
144 /173 /29
30 /34 /4
8 /9 /1
40 /48 /8
8 /10 /2
8 /10 /2
2 /3 /1
2 /3 /1
2 /1 /3
2 /4 /2
2 /4 /2
2 /3 /1 (spare parts)
6
CSC Track Finder
•
•
•
•
•
•
•
•
•
CSC TF
CCC Trigger
0
Trigger
0
Trigger
DDU Trigger
DDE Trigger
TSP Trigger
CCB Trigger
0
Trigger
6-Nov-08
installed /total /spares
TF 3052000000200500000 9U CAEN VME
1 /3 /1
TF 3052200603200500000 wiener crate ( trigger)
1 /2 /1
TF 3052200602200500000 TF backplane
1 /3 /2
TF 3052 200500000 DDU
1 /2 /1
TF 3052040405200500000 DDU extender
1 /3 /1
TF 3052201916200500016 SP
12/17 /2
TF 3052030302200500025 CCB
1 /3 /1
TF 3052000000200500005 muon sorter
1 /5 /3
ESSC Nov 2008 > CSC Status Report
by Fred B.
7
CSC FED
•
•
FED Crates
> Wiener 9U VME
FED
FED 3052060504200510000 FED Crate
installed /total /spares
4 /4 /0 (1 sp PS)
•
•
FED Boards
> OSU Custom Brds
CVC
FED 3052032203200510000 CAEN 6U CC
installed /total /spares
4 /5 /1
–
•
DCC
–
One sent to CAEN for repair, good spare is loan from pool
FED 3052040303200510006 Clock and Control
•
DDU
•
•
Pcrate Network > GBit
GNS
GIG 3052071419200510001 GBit Ethernet switch
6-Nov-08
4 /6 /2
2 more to be shipped to CERN
FED 3052040421200510032 Data Board
36 /?? /??
installed /total /spares
8 /10 /2 (1 sp installed)
ESSC Nov 2008 > CSC Status Report
by Fred B.
8
Issues with System Reliability and
Maintainability
•
LV > Wiener Maraton system (air cooled)
– Problems with OPFC modules
• The front panel switch has been a problem – the quick connect to the switch becomes
unreliable
–
Plan is to have the connection soldered after agreement with Wiener
• The Soft Start can fail and damage module
–
–
–
Turning the power OFF and then back ON too quickly can cause the failure
This is a feature of the slow ramp-down inside the module
For now the fix is administrative
» Only experts cycle switches > with proper time interval observed
» Shifters instructed (& procedures) to switch ON only if they are found OFF > never cycle
– CANBus communication problems with Maraton units
• Recently observed > not yet diagnosed
• CANBus PC crashes – could be symptom or cause
–
–
•
•
•
•
•
6-Nov-08
Note LV stays ON and hardware protections stay in place
But DCS monitoring and software protections are lost
Reboot computer
Cannot communicate with Maratons
Cycle power to Maraton(s) – see problem above
Then communication can be restored
Works BUT have to restart and reinitialize electronics for multiple crates plus 9 chamber
per crate. Also cannot be done on the fly but requires time between runs.
ESSC Nov 2008 > CSC Status Report
by Fred B.
9
Issues with System Reliability and
Maintainability
•
HV > problems with resistors in UF custom system
–
–
A resistor is used for each channel to measure the exact voltage applied to that channel >> over
10,000 channels
The resistor used has failed to retain its factory specifications
•
•
–
We will replace these resistors during the coming shutdown
•
•
•
•
•
•
The resistance moves up or down significantly over relatively short time periods
This change can be calibrated out but would require re-calibration before each run
Modules will be removed from tower racks in cavern
Resistors will be swapped in the ISR clean room area
QA, tests and calibrations will be carried out at ISR
Modules will be re-installed
The process will be carried out sequentially so that most of the HV for CSC is operational at all times.
Electronics modules and crates
–
On-chamber Electronics
•
•
–
Old problem with fuses on ALCT boards is corrected
Very limited access makes repair of ‘normal’ failures very difficult > thus we have an inventory of
deferred interventions
Other Electronics
•
PROM issues for pcrate boards
–
–
•
•
•
Program for onboard FPGA’s is stored in local PROM and reloaded at each hard reset
These PROM’s loose their program with the result that the connected chamber is lost to the readout
A reprogram of the PROM brings it back without problem
But need to find a solution > being worked on
internal interfaces > no issues
6-Nov-08
ESSC Nov 2008 > CSC Status Report
by Fred B.
10
External Interface reliability
•
Cooling
–
–
•
Power
–
•
•
During CRAFT running the cooling on detector and in counting rooms has been reliable
We are working to add additional DCS linked temperature sensors in CSC cavern racks to
detect problems in a finer grained way and to monitor trends
Power upstream of our LVPS has been reliable.
TTC > no particular problems
DAQ
–
–
6-Nov-08
Occasionally the CSC FEDS have sent FMM warnings and errors generated by problem frontend boards
• Since the Central Trigger stops a run on an FMM warning or error we have disabled FMM
reporting by our front-end boards
• This will result in a negligible loss of data
• We will review this decision when we are allowed access to fix the small number of
boards
CSC along with other systems have problems associated with start of run
• the source has been identified in the software and fixes will be implemented
ESSC Nov 2008 > CSC Status Report
by Fred B.
11