Transcript No Slide Title

FEP Diversity Project
RTS System Admin Group
RTUs, Communications
Agenda
1.
Substation – EMS Communications Overview
2.
Traditional 4-wire communications
3.
FEP Diversity Project
a. Motivators
b. Pilot Project - POC
c. Technical Solutions
i. Communications Architecture
ii. AREVA Database Modifications/Changes
d. Additional Benefits
4.
Future Direction – Expansion to other Substation
needs
Real Time Systems
2
Substation – EMS Communications Overview
Real Time Systems
3
Substation to EMS data flow
Energy Management System Applications
FEP
SMP – SCADA Management
Platform
Analogs
Accumulators
Statuses
Controls
SCADA
Subsystem
RTUs
Estimated Values
Frequency, time,
plant & tie-line
measurements,
unit controls,
ACE, system load
Unit Controls
Network
Subsystem
Analogs & Statuses
External unit MW schedules,
pump unit limits,
incremental cost curves
Generation
Subsystem
Unit security limits,
loss sensitivities,
AGC participation
factors
Real-Time Control System Environment
HABITAT NETIO Web-FG
Utilities: Configuration Manager MRS
Substation
Alarm
Operating System
Unix/ Windows 2000
Real Time Systems
4
SCADA protocols

Protocol is a way of organizing and presenting data in a
consistent, repeatable format

Bit and byte oriented protocols
– Byte - Harris, DNP
– Bit - Conitel, CDC2, TRW S9000, TRW S9550
– Special interfaces - Intrac and MOSCAD radio systems

All master station SCADA protocols do the same basic
functions
–
–
–
–
–
analog data conversion
status point monitoring
support “select before operate” function for controls
provide analog output capability
provide different types of scans to capture transient events between scans
and minimize bandwidth usage
– provide some form of “cyclic redundancy check” CRC so that if the
received message has been corrupted during transport it will be rejected.
Real Time Systems
5
FE RTU numbers and types

Approximately 2000 RTUs on FE system

650 in west, 1350 in the east and growing

Diverse hardware platforms

Diverse protocols – including:
–
–
–
–
–
–
–
DNP 3.0
Harris 5000
CDC II
TRW S9000 / 9550
Conitel 300/3000
Modbus
Intrac/Moscad
Real Time Systems
6
RTU communications

Primarily four wire leased circuit
– 1200 baud modem

Radio used extensively in east ~ 650 radio equipped RTUs
– Primarily found at line switch controllers

Dialup - Approximately 100 dialup RTUs in west
– Primarily found at line switch controllers
Real Time Systems
7
Substation to EMS data flow
Energy Management System Applications
FEP
SMP – SCADA Management
Platform
Analogs
Accumulators
Statuses
Controls
SCADA
Subsystem
RTUs
Estimated Values
Frequency, time,
plant & tie-line
measurements,
unit controls,
ACE, system load
Unit Controls
Network
Subsystem
Analogs & Statuses
External unit MW schedules,
pump unit limits,
incremental cost curves
Generation
Subsystem
Unit security limits,
loss sensitivities,
AGC participation
factors
Real-Time Control System Environment
HABITAT NETIO Web-FG
Utilities: Configuration Manager MRS
Substation
Alarm
Operating System
Unix/ Windows 2000
Real Time Systems
8
What is a FEP?

Windows 2k3 server OS

Standard Infrastructure server Hardware
– One exception, the FEP has additional Digi PCI cards so that
server has 128 COM ports

Runs the AREVA e_terraControl suite of applications
–
–
–
–
–
Process Starter
CFEReader – Polls RTU’s
SCADA – Contains the database
ISD – Connects the FEP to the EMS
Ancillary processes:
– AlarmPager (provides method to pass FEP alarms to EMP)
– SpaceSaver
– DBUpdate

Key: No FEP’s = No Data entering the EMS
Real Time Systems
9
Real Time Systems
10
Traditional 4-wire communications
Real Time Systems
11
Traditional 4-Wire Communications
Real Time Systems
12
FEP Diversity Project
Real Time Systems
13
Project Motivators

Issue identified by NERC as a disaster recovery limitation

Loss of DataCenter, either from a physical or communications
perspective, essentially eliminates the EMS system.

Upgrade the method of communications to critical transmission
substation RTU’s.

Network change without causing system software changes
– Keep serial connected FEP ports and serial connected
RTUs

Improve Communications Network Reliability
Real Time Systems
14
Project Overview

Proposal to operate selected critical transmission RTU’s from
physically diverse FEP’s

3rd/4th Quarter 2005 performed R&D proof of concept testing

Production deployment began 1st quarter of 2006

Initial sites include interconnection and critical transmission
sites

By the end of 2006 we successfully cut over 54 production sites

By the end of 2007 we successfully cut over 104 production
sites – All 345 and 138kV Tx sites in FE’s MISO area

111 Transmission sites in PJM are scheduled for completion
though 2012

All new FE RTU sites are being installed on this technology
Real Time Systems
15
Pilot Objectives - What we wanted to prove?

Demonstrate that the AREVA system can support data
acquisition using diversified FEP’s at the same level of reliability
as currently observed with co-located FEP pairs.

Verify the proposed network communications architecture will
support geographically distributed FEP’s.

Demonstrate that in using diversified FEP’s there is no
degradation in AREVA system level performance (FEP and
EMP).

Demonstrate that alarm processing performance is equivalent
to that found with legacy designs - throughput.

Verify system responds properly to the loss of inter-FEP
heartbeat.
Real Time Systems
16
Required work activities in R&D Pilot period

Build new servers (Windows and Unix)

Develop a point lists for each RTU

Develop software configuration for each RTU

Develop a strategy for modeler configurations

Develop a standard network hardware configuration

Configure AREVA Systems

Configure, debug, and test pilot system

Develop Cut-over strategies for production
Real Time Systems
17
Measure of success in all pilot phases?

Burn in test using all RTU’s for a one week period of local
continuous polling is successful
– No FEP process shutdowns
– Minimal missed scans

Alarm propagation and data integrity - 24 hour test
– All status points change state at least once per scan interval
– All analog points change each scan interval
– No missed scans or missed alarms

System stability is maintained for following failover scenarios:
– Loss of one FEP
– Loss of inter FEP heartbeat
– Loss of one EMS server
Real Time Systems
18
FEP Diversity Pilot Phases

Phase 1 - 10/3/05-10/13/05
– Performed local proof of concept testing using new FEP’s and 55
logical RTU’s. Using DYMEC supplied network hardware, failure
analysis was be conducted and logical RTU’s polling statistics
were gathered. FRAME network is all mocked up locally at
DataCenter B.
– End Result- Satisfactory test results based on operational
performance and failure analysis. All testing performed locally at
DataCenter B.
Real Time Systems
19
FEP Diversity Pilot Phases -CONT.

Phase 2 - 10/14/05-10/25/05
– DYMEC Equipment was extended to DataCenter A to allow
controlled diversified FEP testing. Required relocating a FEP
server to DataCenter A, and the associated FRAME network
connectivity.
– End Result- Satisfactory test results based on operational
performance and failure analysis.
Real Time Systems
20
FEP Diversity Pilot Phases - CONT.

Phase 3 -10/26/05-11/16/05
– Selected substations (6 site’s) will be dual ported and connected to
digital circuits. Digital FRAME circuits will appear at DataCenter A
and DataCenter B where the diversified FEP’s and EMS servers
reside.
– Includes site’s on the FE, and the Telco (leased) FRAME network’s
– End Result- Satisfactory test results based on operational
performance and failure analysis.
Real Time Systems
21
Results of Pilot

Project team successfully installed and configured the new versions of the
SMP software on FEP pair.
– Latest Windows O/S
– Latest AREVA SMP Software
– New hardware components

Verified successful RTU communications using current serial
communications.

All three phases of RTU testing completed successfully, including phase
III’s real-life field tests

Developed Cutover Plan
Real Time Systems
22
Production Implementation Project

Order communication circuits as required
– A majority of circuits require High Voltage Protection (Positron)

Make necessary RTU configuration changes
– Baud Rate / Modem settings

Complete AREVA modeler work to allow cutover to occur
– Run ManageSiteAssignments.pl script to use Primary/Secondary SITE ID
– Make PATH changes for new communication circuit

Cutover RTU’s to Diversified FEP
– Perform full station check-out on new circuit
Real Time Systems
23
Project Timeline
Real Time Systems
24
FEP Diversity Architecture
Real Time Systems
25
FEP
DIVERSIFICATION
DYMEC
(Physical)
8 PORT TERMINAL SERVER
(SERIEL CONNECTED max.: 38400
bps
DEVICES i.e: RTU, DFR,
LINE SWITHCES...etc)
10/100
FRAME RELAY
NETWORK
- DSU/CSU (Digital Circuit Interface)
(T1-Fractional, 64k)
DYMEC
-FRAD (Frame-Relay Access Device)
5 PORT ETHERNET SWITCH
(IP ONLY DEVICES i.e.: PC,
IP CAMERA...etc)
- ROUTER
-SUBSTATION HARDENED
(-40C -+75C) no fans
- BUILT-IN DATA ANALYZER
- FRAME/SERIAL ENCAPSULATION RTU PROTOCOL
Network Services Engineering
Confidential
Real Time Systems
TITLE
FEP DIVERSIFICATION PILOT PHASE #1
FILENAME
REVISED
9/14/2006
FULL FILENAME
PAGE
1 OF 1
26
DRAWN BY
REX DUDAS
FEP
DIVERSIFICATION
DYMEC
(LOGICAL)
- FRAME/SERIAL ENCAPSULATION RTU PROTOCOL
(NON-ROUTABLE, ENCRYPTED)
DataCenter A FEPA
RTU MAINTENANCE PORT
RTU
DataCenter B FEPB
FRAME RELAY
NETWORK
8 PORT TERMINAL SERVER
(SERIEL CONNECTED
DEVICES i.e: RTU, DFR,
LINE SWITHCES...etc)
PVC 1
PVC 2
DYMEC
PVC 3
CORP. TRAFFIC
(FIREWALL, ENCRYPTED)
(ROUTER)
IP ROUTABLE PROTOCOL REMOVED OVER WAN
LINK
5 PORT ETHERNET SWITCH
(IP ONLY DEVICES i.e.: PC,
IP CAMERA...etc)
Network Services Engineering
Confidential
TITLE
Real Time Systems
FEP DIVERSIFICATION PILOT PHASE #1
FILENAME
REVISED
9/14/2006
FULLFILENAME
PAGE
1 OF 1
DRAWN BY
REX DUDAS
27
Cut-over Strategy (part1)

Need to have both existing and diversified RTU modeled for
cutover in AREVA EMS
– Model an additional TFE structure for new diversified FEP
– Changes to BAUD, PTMO1, PTMO2 and NoReplyTimeout
– Additional RTUC Record pointing to same RTU record as 4-wire circuit
in production
– Utilize SITE_* and SITE2_* fields to facilitate the cutover
– Each RTU has potentially hundreds of POINT/ANALOG/COUNT
records associated with it.
– These all need their SITE_* field set to the ID_SITE of the new diverse
FEP.
– SITE2_* field set to the ID_SITE of the current 4-wire FEP in production
– Manage those individual data items that do NOT allow automatic
SITE/SITE2 selection.
– Selection is done automatically by SCANNER based on quality. Items
with no status, (CTRL only) do not have a good quality code to follow
(TAP Changers)

Areva TFE tree changes made, No RTU tree changes made.
– Limit the risk of the change – No production affects
Real Time Systems
28
Cut-over Strategy (part2)

RTU Changes
– New FRAME T1 Circuit can be installed at any time on
spare pairs
– Make RTU configuration changes and load onto spare D20
processor board
– Swap out board for cutover, if there are issues original board
can be immediately put back
– No Point mapping changes made
Real Time Systems
29
POST DESIGN BENIFITS

DR Environment
– Remote RTU can be pulled from either DataCenter

Eliminates costly – Out-of-date Analog 4-wire circuits

Increase RTU speed/poll interval
–
1200bps to 9600bps (max.: 38400 bps)

Ability to pull byte and bit-oriented RTUs

Remote Support
–
Allow for mobile workforce support/configuration/data analysis – Secured IP Based

Security
– Serial Connectivity – Isolation of corporate substation traffic

HP/OV Support
– Complete Network end-to-end alarming

“Smart” Substation platform
–

Ability to add additional PVC’s for other needs
– IP enabled Security Cameras for monitoring site
– Swipe card entry systems
CIP Cyber Security full compliance
–
Non-Routable Transport
Real Time Systems
30
FEP Diversification Overview
Real Time Systems
31
FirstEnergy FEP Diversity: Proposed IP Based Network Architecture A
GMS DACs
DC-A
GMS DACs
DC-B
EMS FEPs
DC-A
EMS FEPs
DC-B
FE Corp Network
RPR Backbone
Corp FW
DC-A
Corp FW
DC-D
Dynamic
Failover
RTS FW
DC-A
Consolidate RTU Communication over an
IP Based FE RPR Wan and/or Public
Carrier Frame/MPLS Networks.
Substation RTU Network
RPR Exchange WAN
BG #TBD
Sub-RTR
DC-A
Dynamic
Failover
Sub-RTR
DC-B
to
stE ard
a
F LC
tE t o
Fa s ard
M
ML C
T1
Dynamic
Failover
Public Carrier
FR/MPLS
TCP/IP
Network
Fa
ML stE t
Ca o
rd
Aggregate
RTU Router
Site #4
tE t o
Fa s ard
ML C
Aggregate
RTU Router
Site #3
Sub-RTR
DC-C
T1
FEIsolatedWAN
SONET/RPR
Aggregate
RTU Router
Site #2
RTS FW
DC-C
Electronic Security Perimeter
Fractional T1
Substation
RTU
Site #1b
RTS FW
DC-B
Dynamic
Failover
T1
Aggregate
RTU Router
Site #1
EMS SONET/RPR
Backbone
tE t o
Fa s ard
ML C
Substation
RTU
Site #1a
ElectronicSecurityPerimeter
GMS SONET/RPR
Backbone
EMS FEPs
DC-C
Substation
RTU
Site #1
Substation
RTU Router
Site #5
Real Time Systems
Substation
RTU
Site #2
Substation
RTU
Site #5
Substation
RTU
Site #4
Substation
RTU
Site #3
32
FirstEnergy FEP Diversity: Proposed IP Based Network Architecture B
GMS DACs
DC-A
GMS DACs
DC-B
EMS SONET/RPR
Backbone
FE Corp Network
RPR Backbone
Corp FW
DC-A
Corp FW
DC-D
RTS FW
DC-A
EMS FEPs
DC-B
EMS FEPs
DC-C
DYMEC
DYMEC
DYMEC
Dynamic
Failover
Consolidate RTU Communication over an
IP Based FE RPR Wan and/or Public
Carrier Frame/MPLS Networks.
Substation RTU Network
RPR Exchange WAN
BG #TBD
Sub-RTR
DC-A
Dynamic
Failover
Sub-RTR
DC-B
to
stE ard
a
F LC
tE t o
Fa s ard
M
ML C
T1
Dynamic
Failover
Public Carrier
FR/MPLS
TCP/IP
Network
Fa
ML stE t
Ca o
rd
Aggregate
RTU Router
Site #4
tE t o
Fa s ard
ML C
Aggregate
RTU Router
Site #3
Sub-RTR
DC-C
T1
FEIsolatedWAN
SONET/RPR
Aggregate
RTU Router
Site #2
RTS FW
DC-C
Electronic Security Perimeter
Fractional T1
Substation
RTU
Site #1b
RTS FW
DC-B
EMS FEPs
DC-A
T1
Aggregate
RTU Router
Site #1
Dynamic
Failover
tE t o
Fa s ard
ML C
Substation
RTU
Site #1a
ElectronicSecurityPerimeter
GMS SONET/RPR
Backbone
Substation
RTU
Site #1
Substation
RTU Router
Site #5
Real Time Systems
Substation
RTU
Site #2
Substation
RTU
Site #5
Substation
RTU
Site #4
Substation
RTU
Site #3
33
Questions?
?
Real Time Systems
34