Transcript Re-planned PEWs

crm
BT Wholesale Planned Engineering Works (PEW) 90 day overview
– March to June 2011.
The information contained in this presentation is confidential information as per your terms
and conditions with BT. Please do not forward, republish or permit unauthorised access.
The content is accurate at the time of writing and is subject to change.
customer relationship management
1
BTW 2011 Network Changes – 3 month rolling plan
•
Whilst BTW has continually worked to enhance its network, we are currently
undergoing an exceptional level of change driven by four main drivers:–
–
–
–
•
•
•
•
Stability – to improve the reliability of the network, BTW is targeting certain network elements with
hardware and software improvements.
Capacity – the need for additional capacity is driven by an increase in the number of broadband users on
the 21C network (through migration from 20C and growth) and also by the ever-increasing bandwidth
demands of those users.
Resilience –to improve the resilience of the network , BTW is actively removing single points of failure from
the network
New Functionality – to meet the needs of the market BTW needs to enhance its network to deliver the
services its customers require. There is a current focus on efficient content delivery that will be offered
through the Wholesale Content Connect product.
This presentation outlines the key upgrade programmes, their drivers, customer impact
and the number of upgrades per programme per week.
BTW is keen to deliver all the required improvements as quickly as possible whilst
actively managing the risk to customers. For this reason a range of run-rates against
each programme has been outlined. The actual run-rates achieved will be determined
by this ongoing live analysis of risk.
Customer outage times for the PEWs start between 00:01 and 02:00 with the aim of
completing customer impacting work by 04:00. This allows 2 hours for roll-back should
the PEW be unsuccessful.
Business as usual capacity upgrades and upgrades to resilient structures have not been
listed here. All the activities outlined on slides 2, 3 and 4 involve an end user outage.
2
20C Upgrades
Upgrade
Key Drivers
Key:Outage is defined as full loss of service
Planned month for upgrades
Contingency month for upgrades – 20% likelihood
Contingency month for upgrades – 50% likelihood
#
1-3
IPSC
Av. End
user
outage
(Mins)
120
Upgrades
per week
Month
June
Av 10500
Min 3000
Max 18000
Affected
Products
May
Pt1 - 194/194
Pt2 - 81/194
Upgrades
per PEW
April
EUs per
upgrade
March
Functionality
20C BRAS
/ Stability
Code &
Card
1
Upgrades
Complete /
Total
18– 48
20C BRAS code & card – this
upgrade is needed to get the
Juniper ERX BRAS ready for HQOS
which is required for WCC
(content). Rollout (Pt1) of Junos
code only is proceeding as this
does fix some minor stability
issues on the BRAS. Pt 2 will add
new cards and HQOS and is
currently being tested with a view
to intercepting the Pt1 rollout
plan.
See diagram on slide 5 for network location
3
21C Upgrades
Upgrade
Key Drivers
Upgrades
Complete /
Total
EUs per
upgrade
Upgrades
per PEW
Affected
Products
Av. End user
Upgrades per
outage (Mins) week
Month
Av 17000
Min 0
Max 32000
1–3
WBC
100
24-48
Provider Edge
switch
Separation 5
Capacity
15 / 15
Av 20000
1
WBMC (host
link)
180
0-1
SVLAN moves
Capacity,
Stability
1100 / 1100
BAU Ongoing*
Av 800
Min 0
Max 1000
1–5
WBC
35
32-40
Av 50000
Min 1000
Max
132000
1
WBMC (host
link)
60
1-2
6
FER MCLAG &
Virtual
Switch
7
Resilience,
Capacity
7 / 16
* Not including BAU capacity mgmt moves which will be ongoing at estimated rate between 32 and 96 per week
June
3/151
(2nd upgrade)
0/151
(3rd upgrade)
May
3
Stability,
Capacity,
Functionality
April
March
21C BRAS code
& card
upgrade
21C BRA
upgrade
upgrade
processo
memory
Forward
Provide
these u
capacit
Shared
SVL
off
Bro
to a
alle
(co
FE
se
im
W
4
7750
serv
diffe
This
the c
(allo
used
conn
21C Upgrades
Upgrade
Key Drivers
Av. End user
outage
(Mins)
Upgrades
per week
BB EU’s
Av 20000
Max 120000
Ethernet:
Av 60 Ccts
Max 900 Ccts
1
WBC
MEAS
ETHERNET
5
60 - 100
Resilience
213/ 580
Av 2500
Min 0
Max 8000
1
WBC
1
4 - 12
Resilience
3365
/3365
Av 600
Min 0
Max 992
>120
WBC
3
300 – 500
MPLS Core Code update
Router
Code
11
Upgrades
3/120
Av 20000
Min 0
Max 100000
1
EFM Code Functionality
Upgrade 12
692/ 730
Ethernet
Av 7 Ccts
Max 60 Ccts
1
Access
MCLAG
Copper
MSAN
9
10
* Not including BAU capacity mgmt moves
Month
WBMC
ETHERNET
(Copper
Access
Only)
180
(240 Max)
15
(30 max)
June
8
Affected
Products
May
878/1231
Upgrades
per PEW
April
Capacity
BB EUs
and/or
Ethernet
Circuits per
upgrade
March
7750 Code
Upgrade
[EES, EEA
& IEA ]
Upgrades
Complete
/ Total
Ac
po
(m
ag
an
ov
Co
im
fu
MP
req
soo
1 (Mar to
mid-April)
4 (mid-April
onwards)
FEF
to u
gre
fun
80-100
5
Re-planned PEWs
•
•
•
The level of network change that BTW is driving through inevitably leads to a need
to re-plan a number of PEWs.
The reasons for these re-plans include:- unforeseen technical issues specific to (or
indirectly impacting) that programme, re-prioritisation of work, insufficient time to
complete all work on the night and operational decisions.
BTW have made some improvements to the management of re-planned PEWs and
where possible will seek to avoid using Emergency PEWs for this re-planned work.
In some cases re-planned work will need to be carried out with less notice.
6
Re-planned PEWs
Specific Programmes with high proportion of replans:Programme
Reasons
20C BRAS Code & Card
upgrades
1.
2.
Unforeseen technical issue impacting operational mgmt of some BRAS
Delayed fix meant re-planned PEWs had to be moved again
FER MCLAG upgrades
1.
Prioritisation (before Xmas) decision to re-allocate resource to BRAS
MCLAG programme
Process issues have led to abort decisions on the night before service
impacted. This is under review & some shorter-notice PEWs (~ 1week)
may be required to avoid capacity issues
2.
7750 Code upgrades
1.
2.
Productivity issues (now resolved) meant high proportion not upgraded
on the night
Additional checks were requested from the vendor to provide better
control on the night for the most heavily loaded 7750s. This meant that
some of the PEWs were replanned. These are now being carried out.
7
Broadband Services on 21CN
2
1
11
9
10
8
8
3 4
8
6
8
5
11
7
8
Ethernet Service on 21CN
8
EEA/EES
EEA/EES
F
8
12
EFM
21CN Core
EEA/EES
F
Key
ETHC Ethernet Circuit
ETHA Ethernet Access
(Fibre/Copper)
EEA/EES
EFM
7750 Edge Router
EFM Router
Customer / CP Site
8
EEA/EES
8
Descriptions
•
•
•
•
•
•
•
•
•
•
•
•
•
•
Colossus refers to BT's 20C UK core Internet backbone network mesh
which runs on IP technology
DSLAM - A Digital Subscriber Line Access Multiplexer (DSLAM, often
pronounced dee-slam) allows telephone lines to make faster
connections to the Internet. It is a network device, located in the
telephony exchanges of the service providers, that connects multiple
customer Digital Subscriber Lines (DSLs) to a high-speed Internet
backbone line using multiplexing techniques.
BRAS - A broadband remote access server (BRAS or BBRAS) routes
traffic to and from the digital subscriber line access multiplexers
(DSLAM) on an internet service provider's (ISP) network. The BRAS
manages the logical path from the consumers modem through to the
IP core network.
Fibre MSAN – Fibre MSAN provides both direct access for Fibre fed
customer services and aggregation/resilience protection to the
Network. MSAN is essential a next generation DSLAM. This is
essentially the edge of 21C transmission backhaul.
Copper MSAN -Copper Multi Service Access Node equipment provides
all Access for Copper/Metallic fed services, PSTN Voice DSL,
Broadband, Slow Speed Ethernet and converts the end user service to
IP.
EEA – Ethernet Edge Aggregator (a 7750)
EFM – Ethernet in the First Mile. Ethernet over copper access
IEA – Internet Edge Aggregator (a 7750)
EES – Ethernet Edge Switch (a 7750)
MSIL - Multi-service Interface Link is the Ethernet pipe connectivity
between BT’s and CPs’ 21CN networks. MSIL provides for the needs of
WBC, WBC (Converged) and NGN Call Conveyance.
BEA - Broadband Edge Aggregator.
FER – Front End Router
MSPE – Multi-service Provider Edge Takes 20C BRAS capability closer
to the end user. Less latency.
MCLAG – Multi-chassis Link Aggregation Group –
•
•
•
•
•
•
•
•
•
•
•
•
NGA – Next generation architecture. Fibre based internet access such
as FTTC/FTTP (Fibre to the cabinet/premises)
GEA – Generic Ethernet access, allows local loop equipment (e.g.
Openreach) to be connected to fibre.
EU – End User. The consumer or business that uses the WBC
connection
Access Node – A local exchange containing one or more MSANs
Metro Node – The backhaul network from the Access Nodes
terminates on the metro nodes.
Core Node – One of 20 sites where WBC traffic is aggregated and
handed over to CPs
LPA - Logical PoP Aggregator (a router that aggregates the traffic from
BRASs at a 20C broadband point of presence)
MPLS Core- Multi-Protocol Label Switching Core network. A high speed
IP network, where packets are given a predefined route and pass
straight through, Unlike a normal IP network where each packet is
inspected and routed by each node in the network.
AP - Aggregation Point. This is where multiple end users are
aggregated into a single path for connection to the CP. Allow CPs
access to broadband end users
EP - Extension Path. This is the connection from the AP at the WBC
Interconnect Node to the CP.
Multiplexing – Many of these network elements perform the same
functions of multiplexing many transmission pathways onto a single
pathway, such that a point is reached where a single physical
connection can route connections into an ISPs network.
Alcatel 7750 Infrastructure Ethernet Aggregation Switch (IP
Multiplexor)
10
Management of risk
• In addition to the standard risk mitigation that would be in place for any
network upgrade project, BT is applying additional rigour as outlined
below:– Co-ordination of PEW activities to avoid co-incident clashes with other
network and systems changes.
– Geographical rollout to focus Field Engineering resource and spares for 72
hour post-PEW support following higher-risk PEWs
– Daily calls to sanction all PEW activity for following night based on latest
available information
– Separate Go / No go calls for all key upgrades
– On-the-night efficiencies to reduce risk of failures and overruns.
– Combinations of co-incident BRAS upgrades scheduled to minimise impact on
RADIUS servers
– Additional resource being mentored to widen the skill sets and spread
available hands
– Review of previous night’s PEWs with full root-cause analysis of issues and
over-runs and mitigations put in place to minimise re-occurrences
11
PEW IMPROVEMENTS
You Said
Were Working on
Delays in the affected features being seen within
BBCR.
Carrying out e2e reviews of the PEW process with BT
Design, Operate & Wholesale to understand this in
more detail.
Different PEW communication processes (20C v 21C)
Carrying out e2e reviews of the PEW process with BT
Design, Operate & Wholesale to understand this in
more detail and align if possible.
Not always notified if a PEW is cancelled or replanned
Manual process put in place Jan 11, new automated
process being introduced Release W 15th May.
12
PEW IMPROVEMENTS
You Said
We Did
BTW don’t notify us when a PEW has been re-scheduled.
From January 2011 we now notify you when a PEW has been re-scheduled.
PEW notifications do not always include the location / exchange details.
We now include the location / exchange details within the PEW
communications.
Lack of detail & understanding as to why the large increase in BTW PEWs.
Set up weekly calls with Industry reviewing a 90 day rolling PEW programme
explaining the key upgrade programmes, their drivers & impact.
We now link the PEW communications to the PEW programme of works for
CP’s to refer to for further information.
Duplicate PEWs under one PEW reference can last a number of days
meaning it’s difficult for CP’s to effectively manage these PEWs with their
own customers.
We now issue separate PEW’s for these works so CP’s know the impact and
when these works will take place.
We have seen duplicated PEW communications (Network & System) for the
same works at the same time and location causing confusion / duplicate
handling.
Investigated & resolved the issue by removing a flag that was being
generated against both these types of PEW notifications. (Network &
System)
Can we notify you if a PEW programme is as a result of a recent MSO to
prevent future incidents in the network if common cause found.
From February 2011 we now notify CP’s if the PEW is related to a MSO.
The PEW gadget on My Page is really slow to download the relevant PEW
information so not a tool we can utilise.
We have resolved the speed issues meaning the PEW gadget now
downloads much quicker for CP’s to utilise & view PEWs & the affected
features.
The PEW gadget on My Page doesn’t show all PEW notifications.
Resolved some data issues to now show all PEWs however there is a delay
in the data uploading to the PEW gadget (approximately a few hours) so
not a LIVE gadget.
Receiving PEW notifications for infrastructure not impacted and some cases
being received late
New internal process now in place to remove inappropriate notifications
and measures taken to ensure minimum notification period is adhered to.
13