Transcript MP3 report - CERN Indico

MP3 report : W27 to W33
(22 slides, 10’ ?)
- Technical issues
- Organisation update
J.Ph Tock for the MP3 team:
TE-MPE : N. Catalan, R. Denz, M. Koratzinos, A. Verweij, M. Zerlauth
TE-MSC : B. Auchmann, S. Le Naour, M. Modena, A. Musso, J.Ph. Tock
+ K Dahlerup, E Nowak, A Rijlart, Z Charifouline
JPhT – MP3 18th August 2010
1
Switches from board A to board B
• EICs have the right to and are doing the swaps / Also sending logging
MP3 has to be informed of QPS issues (written in MP3 logbook or email
or …)
• Will run on board B before next technical stop to identify problematic
channels ; only communication is affected and not the protection
• A new firmware is under test in the lab so this will become transparent
Implementation is foreseen during the Christmas break
• Circuits concerned (from MP3 logbook) on > 5 weeks:
RCBXV2.R2 (10), RCBXH2.R2(2), RCO.A56B1&2 (5), RCBXV&H1.R2 (3) , RQS.A12B2 (5), RQTL11.L5B2
(5), RQTF.A67B2 (2), RQTF.A81B1 (5), RQTF.A56B2 (2), RQTD.A56B2(1) + some more from OP logbook
Out of 2400 boards
JPhT – MP3 18th August 2010
2
Discharge of Quench Heater Power Supply
•
•
•
•
•
•
•
15/7 : RB.A45 : Quench heater power supply of B17.R4 discharges
Replacement postponed to the day after with monitoring of the 3 other power supplies
17/7 : RB.A56 : Quench heater power supply of C34R5 discharges
20/7 : RB.A12 : Quench heater power supply of C34L2 discharges
20/7 : RB.A56 : Quench heater power supply of B11L6 discharges
23/7 : RB.A45 : Quench heater power supply of C18.R4 discharges
28/7 : RB.A45 : Quench heater power supply of B8.L5 discharges
17/8 : RB.A12 : Quench heater power supply of B24.L2 discharges
7 cases in > 5 weeks – MTBF is 106 hours - > 6000 QHPS in the LHC
Fault identified ; some replacement will take place. The availability should be improved
by a factor of at least 10
The procedures were reviewed and agreed.
We should not start a fill without redundancy but we may allow to loose redundancy during the fill.
So, if an HDS is lost when there is beam in the machine, nothing will happen. However the injection
permit is lost and no new beam will be injected … except if QPS_OK masked
•
•
Was not applied yesterday … QPS_OK was masked (no mention in logbook) for injection so relying
on the operator to check if no more QHPS discharges.
Software will check this automatically and initiate (after EIC validation?) a slow power abort if 2
dipoles or 1 quadrupole QHPS is discharging. Implementation planned for October TS
JPhT – MP3 18th August 2010
3
Global BB detector : mains
Spurious trigger signal not related to any real event which is self curing
(FILLING BUFFER --> FILLING BUFFER NOT OK --> FLT TIMEOUT -->
LOGGING --> QPS-OK back)
The origin of the temporary problem is related to the Global Busbar
Detector, not filling the buffer, is related to a time-out and will last for 2
to 5 minutes. It has an automatic reset (6/8, 9/8,..)
 In case of loss of QPS_OK, waiting 5 to 10 minutes can solve the problem
• 15/8: Checking QPS signal in a stable beams phase, it was noticed that
QPS_OK was lost for a communication problem. Protection is still
ensured. Intervention for a reset of the local main BB detector in the
tunnel was planned that solved the issue
 Personnal computer to be rebooted locally (access) ; will be solved during
Chritmas break
 Check allowed to plan the intervention and minimize its impact
• There is a plan to remove it during the Christmas break as nQPS is taking
care of this. The information on the total voltage across the BB being a
useful information, this will be kept but only in monitoring and not in the
interlock
•
JPhT – MP3 18th August 2010
4
Trips due to too fast request
•
Request functions have been smoothened
–
–
–
–
–
–
–
•
•
8 RQTD.B2 tripped ; feedback functions (13/7)
Various RQTF/D; real time trims (15/7 & 5/8)
Various RQS; new ramping-up parameters (23/7)
Various RQTF/D due to tune feedback re-enabled by mistake (1/8)
RCBV16.R4B2 (3/8)
RSF2.A23B2 and RSF.A67B2; squeeze functions(12/8)
RCBXH1.R2: Trips due to mismatch (30 mA) of the current before the start of squeeze
Partial watch dog is implemented (OK for preloaded functions)
Limits increased:
–
RQX.L/R2 & 8 : During setting for the 10A/s ramp, problem encountered with these 4
inner triplet circuits (slightly exceeded the initial commissioning value of 6A/s - needing
something like 6.05 A/s).
There is no limitation for a slight increase from the converter point of view so the limit
was increased to 6.2A/s. Values have been changed in LSA and synchronised into the FGC.
Approved by MP3
JPhT – MP3 18th August 2010
5
Undulators
• Status regularly checked to plan for a reset,
minimizing operation disturbance
• No perturbation in the last 6 weeks
• Autonomy > 10 days
Triplets circuits
• Many trips due to PC when ramping down
• Triggers MP3 verifications of the heaters firing
• EPC issue under investigation to assess if
solution is possible
JPhT – MP3 18th August 2010
6
Communication problems with access
•
•
•
•
•
B18R5 old QPS; Fip com lost; access required; incorreclty assembled
connector (8/7)
Segments 8R4 & 15 R4 of the mains: WorldFip experts accessed tunnel
(10/7)
RB.A12 : No communication with current leads; Access to tunnel to restart
main BB detector (12/7) ; will not be used any more next year
S56 global BB detector (26/7): reboot from tunnel ; will not be used any
more next year
DQAMGS.B11L5 : needed MPE piquet access in tunnel (10/8)
JPhT – MP3 18th August 2010
7
Communication problems without access
•
•
•
•
•
•
•
•
•
•
•
•
•
Communication lost on DQAMC.21L7 S67: cured by resetting the gateway(8/7)
RCBV10.R1B1 : PC communication (8/7)
RQT13.L6B1, RQT13.L6B2, RCO.A56B1 and RCO.A56B2 : (Fast) Power cycled cured it
(8/7 & 15/7 & 17/7)
RQTF/D in 78 : Power cycle cured it (10/7)
RCO/D.A56B1 : communication lost on crate; Power cycle cured it (12/7)
RQRF.A67B2 LD1 com pb (13/7); Power cycle cured it
Network in CCC; lost QPS_OK of 6kA&600A in34 not visible; masked as at injection
level ; came back < 15 minutes later (15/7)
S67 communication lost with gateway; cured by rebooting gateway (15/7)
RQD/F.A12 : Lost communication with MQ.16L2 (DQAMC.16L2) while ramping up the
current. By miracle, communication came back after 25 minutes (23/7)
RQT13.L6B1&2, RCO.A56B1&2: Loss of com. with the crate ; cured by power cycle
(4/8)
S67 mains lost due to communication problem(8/8)
Loss of QPS signals in S12 ; server had to be rebooted (12/8)
Loss of Loss of communication with main BB detector of RQF/D.A67 cured by
sending logging (EIC) (15/8)
JPhT – MP3 18th August 2010
8
External? electrical perturbation (1/2)
•
10/7 : S67&78 lost: origin RQ10(9).R6 & L8
– No PM of RQ9, as on same acquisition system as RQ10, will be solved after
Christmas break
– Long cables and specific settings (see further)
•
13/7 : RQ5.L2
– Trip du to noise on U_RES ; QH discharge is OK ; During a ramp ;
Noisy board that was replaced
JPhT – MP3 18th August 2010
9
External? electrical perturbation (2/2)
• 23/7 : RQ9.R2
– Trip ; heaters fired ; U_Res over threshold but origin of perturbation
not identified
• 27/7 : RQ6.R1
– Trip ; heaters fired ; U_Res over threshold but origin of perturbation
not identified ;
– typical analysis with heater discharge and recharge checked
JPhT – MP3 18th August 2010
10
MP3 is 1st line for analysis
Some examples:
•
•
•
•
•
•
•
•
•
16/7 : RQT12.R5B2 : Trip due to fan problem on PC
18/7 : RQ6.R5 : PC trip (EPC issue)
25/7 : RCS.A45B1 : Tripped 3 times; EPC changed a power module
1/8 : RB.A56 : beam dumped by RB trip due to water fault issue in the PC. EPC
intervened
7/8 : RQ8.R6 : Trip first thought to be due to noisy QPS board but finally was due to a
PC component that has been changed
8/8 : RCO/D.A56B1 : Trip (<1A) during the night ; indication that possible earth
current fault ; EPC intervened
16/8 : RB.A56 : Cannot be rearmed due to water fault so intervention by CV
17/8 : RCBH30.L2B2 : Circuit tripped due to a VIN DC UNDER VOLTAGE voltage
17/8 : RQT13.L2B2 : Tripped while precycling due to the PC with an internal voltage
reference fault
JPhT – MP3 18th August 2010
11
8th of July
• RSS.A34B1
Self opening of the switch (#3) ; first event of this kind in the LHC
The faulty switch Z of the EE system has been replaced on 19/07
Circuit not used in between
• RB.A81
Trip du to nQPS board B21R8 (was changed- Broken resistor on the
board)
JPhT – MP3 18th August 2010
12
22nd of July
•
RCBCV7.L2B2 and RCBCH7.L2B1
–
–
–
Test following intervention on the PC side
Resistance measured during PNO.A1 is slightly larger than the tolerance. Already observed in
the hardware commissioning but very stable.
To be followed by MP3
JPhT – MP3 18th August 2010
13
Not really MP3 but safety issue : 23rd of July : RB.A78
 Trip of RB.A78
•
•
•
Sector trip when switched ON at 08h18m18s. To get access in L8, the EiC has put the
RB on OFF without first requesting STANDBY. The result was that the current in the RB
was SLOWLY decaying towards 0 (see timber plot). Access was then given with current
in the RB (about 500 A) although the EiC thought there was 0 A.
After access was finished, the EiC put the RB on standby. This created a huge voltage
in the circuit because there was still a current of about 200 A in the circuit at the
standby command.
Then the main busbar detector tripped because U_RES went above the threshold and
the switches were opened.
Sequence was not followed
This is managed in another body
Sector trip with 200 A
Access with 500 A in the RB
JPhT – MP3 18th August 2010
14
Monday 3th August
• Beam dump triggered by QPS
• The trip was caused by the spontaneous openig of a circuit breaker
powering the QPS detector in 18L3. Intervention was required
• 2nd occurrence this year
JPhT – MP3 18th August 2010
15
Tuesday 27 July
• RCBXH1.R2 U_res signal with a significant
offset tripped during pre-cycle
• Required an access at 02:00 to be reset from
the tunnel ; will be investigated at next TS
JPhT – MP3 18th August 2010
16
Q10-Q9 sensitivity (to perturbations)
• Special configuration applied to these magnets
to measure higher range voltage
– Poor resolution on the low voltage range
– High sensitivity to noise of the triggering
signal
• Restoring the nominal settings will imply
removing two resistors for each card.
– MP3 recommended to test in the lab and
implement 1-2 examples during the next
technical stop.
– Activity on-going
JPhT – MP3 18th August 2010
17
IPQs bus-bar resistance measurements
• Measurement of bus-bar resistance for the 6kA
circuits
– Reference values for time evolution and
nQPS upgrade
• Already 14 magnets measured in the dispersion
suppressor
– All values around 5 nOhms for 4 to 5 splices.
– Measurements require special equipment
displaced and connected during the
technical stops.
JPhT – MP3 18th August 2010
18
Parallel activities
• Hardware procedures and test plan for next
shutdown/break under revision
• New tools on preparation
– New interlocks that may lock the circuits after
automatic analysis of the data
– 1st version to be released for September 2010
• BLM threshold adjustment tests
– A dedicated MP3 meeting will be organized for this
subject. Location was already identified but needs to
be confirmed.
JPhT – MP3 18th August 2010
19
MPE interventions at next Tech Stop
• IPQs: New system for Q10R2 (Maybe Q9 & 8 too)
– Will require a full re-commissioning (Thursday night?)
• Change of switch on QHPS :
– Beginning of a campaign ; will improve availability
– Recharging will be monitored / no discharge test
planned
– Then campaign to be completed during Christmas break
• Snubber capacitors (S12 or 78)
–
–
–
–
Will be installed but not connected
Connection during the Christmas break
Intervention on critical safety devices
Dedicated tests have to be foreseen, with powering
JPhT – MP3 18th August 2010
20
New supervision mode from MP3
(Since 28th of July 2010)
• Two persons on call for LHC operation
1. 7:00 to 19:00 on weekdays
2. 19:00 to 7:00 on weekdays and 24/24 on
weekends
• Attendance to the 8:30 and 9:00 meetings
• Close follow-up of the LHC operation
• Shifts will be organized after technical
stops
• Same commitment and workload for the
MP3 members but resource optimization
JPhT – MP3 18th August 2010
x162266
21
MP3 Actions
Average: 2.1/day, stable (2.0 over the last 30 days)
Technical Stops – Recovery from TS
Switch to supervision mode
JPhT – MP3 18th August 2010
22