Transcript From the Commissioning Run

CDF
ESPERIENZA di
Commissioning a CDF
Anna Maria Zanetti
INFN Trieste
Bologna - 23 Novembre 2006
CDF
Run I : first data in 1985 (prehistoric organization & detector)  not too
much to learn. (L~ 2 x 1031 cm-2 s-1 )
Run II: Official Start march 2001: almost New Detector! (10-20xLrunI)
From CDF I: solenoid, central calorimeter, part of muon system
All the rest is NEW!
 Endplug Calorimeter
 Tracking
-Silicon:SVXII,ISL,Layer00
-Central Outer Tracker
 Front End Electronics
 Trigger
 DAQ System
 Muon systems
 TOF
 Offline/Online Software
2
La
Sfida didiLHC
LHC
La SFIDA
Energy: 14 TeV
Length: 27 km
Magnetic Field: 8.3 T
Beam Energy: 350 MJ
Bunch Collisions: 40 MHz
Instantaneous Luminosity
# of Collisions in an event
# of Detector Channels: 100 M
# of Scientists (~2500/expt)
LHC: proton-proton
= 7 x Tevatron
= 4 x Tevatron
= 2 x Tevatron
= 250 x Tevatron
= 20 x Tevatron
= 60 x Tevatron
= 10 x Tevatron
= 100 x Tevatron
= 3 x Tevatron
Tevatron:
proton-antiproton
7 accelerators
LHC
3
Commissioning CDFII
Tevatron
LHC
Begin
Cosmic Ray Run
Commissioning Run
with Partial Detectors
Late 1999
2000
Oct. 2000
2006
Detector “Completion”
Jan. 2001
Summer 2007
Commissioning Period
Mar. 2001 - Feb. 2002
Nov 2007 (1 TeV)
Start Spring 2008 (14TeV)
Beginning of Physics Run
Feb. 2002
?
…but 2002 still struggling with fully commissioning some
detectors/electronics/software and with problems in
detectors and Beam.
4
First Phase: Late 1999-2000
CDF non yet completed
 Integration of components into DAQ
– Daily running – pedestals, calibration runs
– November 1999: Three system readout test (DAQ w/ multiple
readout systems: Calorimeter/TDC/Si DAQ
– January 2000: L1 calorimeter trigger established. Sum Et, Single
tower, Missing Et triggers
 Cosmic Ray Running
– Once L1 trigger established, begin Timing-in of Electronics
• Across all detector subsystems, and across trigger subsystems
– Basic Level 3 filtering established
– Development of detector monitoring
– Calorimeter thresholds/noise rates
 A lot of work accomplished in debug and commission all the trigger
systems and the Electronics
– Essential to be able to inject data/ read your system, test it
5
indipendently by the others and in final environment
Commission Trigger & Electronics
A lot accomplished with standalone test (no beam or cosmics):
Take Silicon Vertex Tracker (SVT) as example (~100 custom VME boards
and a complex task) - but applies to all complex trigger systems.
Note: SVT was well thought on testing capability and monitoring
the data flow on each board. Probably the best in CDF:
 Independency from CDF DAQ (data driven device)
 Common data communication protocol
– Boards as building bricks that can be combined at will (~lego)
– Can adapt SVT configuration to various test needs
 Ability to inject/read data from every board
 Can test most board functions with no additional hardware
 Software with board “objects” (ram, regs…) in common framework
Still it was not enough !!!
System missing all these-> struggle, building on the fly
boards for testing purposes, suffered delays, etc
6
..so what was missing?
 Plan for lack of input/output (done, but not enough):
– Must be able to test SVT in place, with proper timing and
data flow, even without SVX/COT/L2/beam
– Not only hw test, also operating/monitoring software
 More functions and flexibility for board/system testing
added on the road
– Plan for long, demanding, integration/commissioning
 Should have invested much more in software much sooner
– More features could be implemented
– More people easily trained (less expert demand)
be creative…
 When come to integrating electronics: be creative.
Any way to bypass/emulate other system/boards must
be pursue and strongly looked for.
7
First Phase: Organization
 CDF has early established shifts/DAQ always
running  once IN, a system must work and be
correctly monitored and checked
– Sometimes testing activities not so easy.
– The payoff is a system kept working, running
and steady growing
Very important: Fight hard the Entropy
 Train “Shift Professionals”: ACEs. Stay in shift
for 3 monthes- overlap 1 monthes. CDF still works
this way.
8
Begin Commissioning with Beam
 Oct 2000 Commissioning Run
– Si “Barrel 4” only
– Many other systems partial
– COT recently on-line (seen 1st
cosmics few days before roll-in)
Commiss.Run had some of everything:
enough to shake down much of systems
 Nov. 2000-March 2001
– Complete the detector
– Continued integration work
– Daily cosmic running
 March 2001-February 2002
– Commission for physics data
9
The Commissioning Run
Crucial event of CDF commissioning
Date
5/9
Week
-2
Period
Lum.
Bunches
Ottobre 2000
18/9
-1
Roll-in
0
1
2
A
3
B
4
5
C
10^29
proton
1 x 8
6
10^30
1 x 8
36 x 8
36 x 36
 Period A : Proton only beam (1.5 wks)
 Period B : Observe first collision (1 wk)
 Period C : Subsystem commissioning (3.5 wks)
Y.K. Kim/Sep.2000
10
Commissioning Run Plan
 Period C (1x8, 36x8, 36x36 bunches)
– Understand operation of COT with colliding beam
•
•
•
•
–
–
–
–
Stability of the chamber with a large amount of ionization
Determine hit occupancies / efficiencies per superlayer
Begin to understand tracking issues / t0, drift velocity
Synchronous noise from Silicon readout ?
Understand operation of Si Barrel-4, new endplugs.
Commission calorimetry and muon systems.
Commission DAQ system (Hardware Event Builder, L3, Data Logger …)
Establish operation of L1 Trigger system functionality
• Calorimeter & muon stubs triggers
• Tracking slice COT – XFT – XTRP to Muon / Calorimeter
– Capture data in L2 processors, simple tagging/prescaling
• Read-in L1 and XFT info, Cluster and ISO cluster operation
• SVT for instrumented region
– Take a few hundred k good events for the COT for the post-run
Y.K. Kim/Sep.2000
11
The First Collisions!!
L1
Beam
profile
L2
Non dimentichiamo!!
A volte pochi giorni di collisioni
producono risultati straordinari
Good
Tracks
Impulso alla
collaborazione
12
From the Commissioning Run
 some data
K short peak
Cambiare plots
SET=500 GeV di-jets
 a lot of work accomplished
…and a better understood list of the
work to be done
13
Still 4 monthes to go: Begin Run II March 01
Downtime logger
 Detailed accounting of the reason CDF is not
taking data (loosing luminosity)
 Very powerful tool to immediately identify what
systems are causing inefficiency (not always so
obvious)
 Used by operation/commissioning manager to
prioritize and decide work schedules
 Identify weakness/limitations of systems
14
Silicon Commissioning
 Only prototype Si installed for commissioning run
– Allowed Si DAQ commissioning.
– Si readout did not cause noise problems elsewhere.
– BUT most of Silicon commissioning still to be done!
 Si was installed in Jan 2001 with just 2 months to
start of Run II (722K channels)
- shifts 24 hours a day, 7 days a week
 But Installation completed
in May 2001 (beam in Mar 01)
– Access to collision hall restricted before
connection complete:schedule complicated
15
E’ iniziato il run II..
Commissioning with Data
 Photon conversions used to
understand the radial material
distribution
August 2001
1pb-1
 Early J/y data (few pb-1)
– basic momentum scale
for tracking
– measure muon
chamber efficiencies
– SVX vertex resolution
16
Tracking Chamber Alignment
 Cosmic ray based alignment: Cell tilts/shifts
– Includes corrections for electrostatics and gravity
Impact parameter vs. phi
17
First unexpected problems
 Early TeV beam had high losses
– Silicon frequently off for protection
– Muon chamber currents very high (installed shielding)
 Power supply failures with beam
– Transistor deaths due to “single event burnout”
• Reduced bias/more resistant transistors/shielding
 ISL cooling lines blocked
– Initially could not operate detector
– Blockage due to epoxy in 90o bends
– Cleared using Yag LASER + prism
Recovered June 02
18
Beginning of Physics Run
 February 2002 is the START OF PHYSICS date
 Still 2002 was a painful year: still a lot to learn and
improve
– Unexpected problem in detectors
– Beam incidents
D,Ds
 Still in 2003:
The first run II paper published
M = 99.410.380.21 MeV
PDG: 99.20.5 MeV
19
Silicon Jumper failure
Aging COT
A small but steadily growing number of
CDF silicon detector modules were dying.
Breakage of a wirebond
CDF central tracking
chamber:
Aging  resolved
• Some broke during a trigger test at
~20 kHz
• Oriented orthogonal to 1.4 T B field
• Fundamental frequency for 2 mm
Al bond ~20 kHz
Resonant oscillation from Lorentz
forces during special trigger
conditions!
•Reduced current
through jumper
•Eliminated guilty
trigger test mode
•Lost some sensors
(z-side mainly)
Resolved!
20
Beam Incidents & CDF Safety
 Based on Run I experience:
– Procedures for store fill and scrape, and store end
– hardware and procedures for minimizing radiation dose to
silicon detector – intended to lengthen life of detector
– Measure losses from p and pbar bunches
NOT ENOUGH !!
Not well protected against
beam incidents. A run II news
LHC beam power = 250 x Tevatron!
21
Beam related Problems
 Very Serious
– Fast beam loss (risk was known, but..) – Damage Silicon
– Damage to silicon from low doses (100’s of rads) at high rate (100
nsec) [particular failure mode not reproduced in tests]
 Serious
– Damage to various electronics in collision hall due to SEB (single
event burnout) or similar single events  abnormally high losses
• One bad example: beampipe misaligned during access  proton
halo scraped  Lost ~12 crate power supplies over about an hour
Actions:
– Added shielding around low-b quads
– Reduced bias voltage in VME power supplies / modify power supplies
 Annoying
– Example: Beam induced background in missing ET trigger  halo
scraping upstream of CDF
22
Abort Kickers
 Kickers are very fast  Danger of fast beam loss:
 Kicker prefire
 Actions:
– Reduce prefire rate (kicker
conditioning)
– Add collimator for almost
perfect shadowing  needed
full lattice+MARS simulation
D0
ant iprotons
IR
E0
C0
ta rget
A11 collimator
Already in place
a “task force” in AD
colli m ator
A0 proton abort
kickers
~ end 2004
IR
B0
Add .5 m Collimator
at A48 to shield
against prefires
F0
proton s
A0
23
The Abort Gap
 Kickers fire correctly, but beam in the abort gap
– Discovered beam in the abort gap when quenched and
suffered silicon damage on abort!
 Monitor the gap
– CDF added monitoring of local losses in abort gap  useful
diagnostic for accelerator – adopted jointly, in TevMon
– Accelerator added better instrumentation– adopted jointly
 Failure of specific Accelerator systems can spill beam
into the abort gap
– Early incident: RF problem drove significant beam into abort
gap 1% of silicon detector lost (unable to talk to chips)
– Added beam abort interlock, monitored in TevMon
– “Tevatron Electron Lens” used to clean the abort gap,
monitored in TevMon
24
25
Important Lesson
 monitor state of
potentially dangerous
systems in the
accelerator - RF system,
electron lens etc
– … Learned by analyzing
each serious machine
accident
 monitor the accelerator
as if it were a detector
system
26
Important Lesson
 Experiment must worry about its own safety – and work
closely with Accelerator Division to ensure it
 CDF enjoyed good communications with AD Operations
Manager and Tevatron experts – this is important
 Joint CDF+AD instrumentation for monitoring
 Determine the cause of every serious beam incident and
take corrective action (bullet may not miss you next time)
 Corrective actions may require significant work from the
Accelerator Division ’’
(quoting J. Spalding)
LHC
LHC, ATLAS, CMS failure modes will not be the same.
But potentially all loss issues will be more severe
Importance: monitoring, diagnostic tools, collimater,
shielding, communication between machine and exp. teams
27
A Physics Heavena Very Complex Trigger!
~ 200 trigger path
185 a L3 - 131 a L2 – 56 a L1
 Defining a working and suitable trigger table  one
of the more complex task of commissioning
 CDF began planning long in advance of run II:
- a group devoted to organize, define, estimate
trigger rate based on data on run I and detailed
simulation
- for every trigger proposed: a long list of works
to be done before trigger approval
this is true now as well -> plan for ~1 year work
Time
 Despite all this work  in CDF: a continuous
fight
28
 never ending upgrades/commissioning
Trigger & L
 Principle: physics process trigger cross section, σ = B (const)
Reality: a trigger cross section, σ ~ A/L + B + CL + DL2 + …
 CDF has worked a lot on trigger
rates -> still failed to correctly
predict how they grow with L.
It is a difficult task!
MET25+2JET
A good trigger system allows one to easily adopt
(CDF trigger has lot of features: L enable, Dyn.Presc. etc)
Still “hard choices” could be needed (drop some physics)
Still one of the top CDF headaches today
29
Trigger Summary
 A flexible trigger table handling is essential to
cope with the continuous changes and increasing
performance demand
 Work to insert relevant physics channels in the
trigger table since the beginning. Late insertion can
turned to be painful
 A very good trigger simulation is an essential tool:
be sure all you need is in since early days
30
…and now: Where can I run my jobs?
 While in the commiss. period (2001) it became clear the
Computing model for data analysis was not good anymore.
Needed CPU x10 + painful tape access -> Old system trashed
 2002 a new model (CAF). In ~ 6 monthes a small CAF was
working -> In 1 year users enjoyed our beloved CAF
First impact with data (and users) could destroy all your
planning -> Don’t panic: there is some time -> A complete
revolution is possible and sometimes desirable
Are CPU and human energy waisted? Yes - CDF did not provide:
 simple tools to manage the analysis of large datasets
 strong set of easily available debugging and code analysis tools
 Motivation/organizations to more centralized processing
When conference pressure-> too late. Users find their way…
31
…Non manca molto al passaggio del testimone
 It is time to begin running together!
Still CDF has his best years of physics
production ahead. Should get 4 X Lint.
 CDF work for LHC in many areas (backgrounds, MC
tuning, QCD, W&top mass etc): don’t overlook the
possibility to learn something today of your
favorite physics channel.
Put your request on the table now: but this does
not mean you will get it (manpower)…… The best
attitude: do it yourself!
Why not begin to commission LHC using CDF ??
Join CDF as a Visitor: you could play
with your favorite CDF data with no duties 32
..c’e’ il rischio (o la speranza!) che qualcuno esageri un po’
TEVATRON
LHC
Don’t tell me
you discovered Higgs!!
Cartoon: courtesy of Young Kee Kim
Many thanks to: Y.K.Kim, T. Liss, T.Liu,
J. Spalding and many others
33
BACKUP
34
Beam Loss Monitor snapshot for a
messy abort
low b quads
D0
CDF
note: CDF
shields D0
abort dump
35
Run II
36
Delivered Luminosity pb-1
Tevatron Run-II
• ICHEP-04 Results : 200 pb-1
• ICHEP-06 Results : 1000 pb-1
(per experiment)
2002
2003
2004
2005
2006
• Data set has doubled every year
37
Triggering in Run 2
45 kHz
300Hz
60 Hz
~20MB/s
Central Tracker (Pt,)
EM + HAD/EM + Track
EM+HAD (Jet)
Muon + Track
Missing ET, S ET
Si Secondary vertex
EM shower max, ISO
Jet Clustering
Multi object triggers
Farm of ~200 PC’s
running fast versions
of Offline Code 
more sophisticated
selections
38
Commissioning with Data
 Additional J/y data used to understand material
Additional 0.455 g/cm2
M(J/y) vs. Pt :
Corrected for nominal
material in simulation
No corrections
 And alignment
Residuals in 5 SVXII
layers
39
Abort Gap
Tevatron has 3x12 bunch trains and 3 abort gaps (2 ms long)
40
Average Luminosity
0.9 fb-1
1E32cm-2s-1
By fiscal year
41