Transcript Industrial Ethernet Performance Metrics

Performance of Industrial Ethernet
Applied to Networked Control,
Diagnostics and Safety
James Moyne
[email protected]
December 2006
ERC for Reconfigurable Manufacturing Systems ERC/RMS
University of Michigan
Copyright © 2006 NSF Engineering Research Center for Reconfigurable Manufacturing Systems
College of Engineering, University of Michigan. All rights reserved
James Moyne
1
Abstract
As industrial Ethernet becomes more prevalent on the manufacturing floor it is
increasingly being considered as a universal networking solution. While the capability of
Ethernet has been proven acceptable in high level manufacturing environments, issues
of Ethernet performance must be considered as we push this technology into domains
of control, diagnostics and safety, and into applications such as I/O, motion control, failto-safe , and security. The University of Michigan’s Engineering Research Center for
Reconfigurable Manufacturing Systems (ERC-RMS) has been exploring the applicability
of Ethernet to the factory floor. Results include:
–
–
–
–
An identification of node software performance as a dominant factor in evaluating overall
Ethernet system performance.
An quantification of the delay overhead associated with common factory Ethernet protocols
such as UDP, VPN and OPC for data transport, security and diagnostics respectively.
A comparative evaluation of common industrial Ethernet protocols of EtherNet/IP and
PROFINET
Development of a cost-based evaluation methodology for determining the need for
partitioning of networks across functionalities of control, diagnostics and (especially) safety
This presentation explores the results achieved in the investigation of industrial Ethernet
at the ERC-RMS and identifies key factors that should be considered in evaluating
industrial Ethernet performance.
Copyright © 2006 NSF Engineering Research Center for Reconfigurable Manufacturing Systems
College of Engineering, University of Michigan. All rights reserved
James Moyne
2
Outline
• Background
– The Engineering Research Center for Reconfigurable
Manufacturing Systems (ERC-RMS)
– Reconfigurable Factory Testbed (RFT)
– Networks Performance Evaluation Efforts
• Performance Metrics for Industrial
Ethernet in Control, Diagnostics and
Safety Domains
– Experimental Analysis
• Key Points
Copyright © 2006 NSF Engineering Research Center for Reconfigurable Manufacturing Systems
College of Engineering, University of Michigan. All rights reserved
James Moyne
3
The UM ERC for RMS
• University of Michigan Engineering Research Center
for Reconfigurable Manufacturing Systems “ERC-RMS”
– Established 1996 (11 year funding from NSF)
– Annual Budget of approx. $3M
– Funding sources
• NSF (ramping down)
• State of Michigan
• Industry membership
• The ERC/RMS is the only ERC today with
Traditional manufacturing focus
– Guided by industry membership
Copyright © 2006 NSF Engineering Research Center for Reconfigurable Manufacturing Systems
College of Engineering, University of Michigan. All rights reserved
James Moyne
4
ERC Membership (partial list)
LIVERNOIS®
Copyright © 2006 NSF Engineering Research Center for Reconfigurable Manufacturing Systems
College of Engineering, University of Michigan. All rights reserved
James Moyne
5
Motivation Responsive Systems
Lean-Manufacturing
Six-Sigma
Mass-Production
Responsive to:
• Demand Changes
• Product Changes
• Opportunities for new products
• System failures – keep production up
The Need:
A factory designed to evolve in response to market demands
Copyright © 2006 NSF Engineering Research Center for Reconfigurable Manufacturing Systems
College of Engineering, University of Michigan. All rights reserved
James Moyne
6
ERC Focus Areas
TA2
Networks
for Control
Software
Infrastruct.
Machine
Diagnostics
Reconfig.
Scheduling
(TA2)
(TA2)
(TA4)
(TA1)
Modular
Logic
Control (TA2)
IMS
Watchdog
(TA4)
RFT
Education
Undergrad Courses
Workshops
Remote Collaborations
Tech Xfer
Platform
• Controls:
• Rockwell, Pilz, Siemens
• Networks:
• DeviceNet, SafetyBus p, Profibus,
Ethernet, Wireless
• Software:
• Brooks, Oracle, Indusoft, iConics
Copyright © 2006 NSF Engineering Research Center for Reconfigurable Manufacturing Systems
College of Engineering, University of Michigan. All rights reserved
James Moyne
7
The Reconfigurable Factory Testbed (RFT)
Serial-Parallel Line
Remote HMI
Virtual Factory
…
NETWORK
AGV
RFID
CMMS
MiddleWare
MiddleWare
System-Level
Controller
Software
Modules
Software Infrastructure &
Control Workflow Manager
Copyright © 2006 NSF Engineering Research Center for Reconfigurable Manufacturing Systems
College of Engineering, University of Michigan. All rights reserved
James Moyne
8
RFT: Control and Diag. Networks
Networks,
Monitoring &
Diagnostics
System Level Database/Web Server
Controller (SLC)
Remote HMIs
OPC:
for Monitoring
and Diagnostics
Ethernet LAN
Virtual Assembly &
Machining Cells
Profibus
DNet
PLC
Soft
PLC
DeviceNet
Copyright © 2006 NSF Engineering Research Center for Reconfigurable Manufacturing Systems
College of Engineering, University of Michigan. All rights reserved
PLC
James Moyne
9
RFT: Safety Networks
Master PSS
Safety
Networking
Bridges
SafetyNet
Cell PSS
I/O block
E-stop
Copyright © 2006 NSF Engineering Research Center for Reconfigurable Manufacturing Systems
College of Engineering, University of Michigan. All rights reserved
James Moyne
10
ERC-TA2 Networking Projects: Summary
• Enterprise industrial networking solutions
– ERC core activity with Reconfigurable Factory Testbed
– Control, Diagnostics and Safety multi-tier networks
• Industrial Ethernet best practices
– Technology trade-offs, and performance analysis
– Network Performance Workshop: April 2006
• Industrial network time synchronization
– IEEE 1588, approaches, benefits,
current issues with implementation
– Analysis of delay associated with common Ethernet protocols
(e.g., UDP, VPN and OPC)
– Factory simulation of network traffic; analysis of impact of time
synchronization
• Network partitioning for safety, control and diagnostics
– Trade-offs of putting safety and control on a dedicated versus
shared network
Copyright © 2006 NSF Engineering Research Center for Reconfigurable Manufacturing Systems
College of Engineering, University of Michigan. All rights reserved
James Moyne
11
Outline
• Background
– The Engineering Research Center for Reconfigurable
Manufacturing Systems (ERC-RMS)
– Reconfigurable Factory Testbed (RFT)
– Networks Performance Evaluation Efforts
• Deriving Performance Metrics for
Industrial Ethernet
– Experimental Analysis
• Key Points
Copyright © 2006 NSF Engineering Research Center for Reconfigurable Manufacturing Systems
College of Engineering, University of Michigan. All rights reserved
James Moyne
12
Network Advantages
Centralized point-to-point
Distributed common bus
Advantages of Distributed Common Bus
• Decrease wiring/maintenance cost
• Increase reliability
• Expand flexibility of control architecture
• Increase reconfigurability
• Expand diagnostic capabilities
• Increase interchangeability
Copyright © 2006 NSF Engineering Research Center for Reconfigurable Manufacturing Systems
College of Engineering, University of Michigan. All rights reserved
James Moyne
13
Performance:
Tracking Error
Worse
Network End-to-End Performance Cost
Out of Control
Ideal
Operating
Region
Networked
Control
???
Digital
Control
Unacceptable
Better
Acceptable
A
B
C
Continuous
Control
PA
PB
Larger
PC
Smaller
Sampling Time
Impact of
sampling
Network
saturation
inducing longer
delays
Impact of
sampling &
additional delays
Copyright © 2006 NSF Engineering Research Center for Reconfigurable Manufacturing Systems
College of Engineering, University of Michigan. All rights reserved
James Moyne
14
Examples of Issues Driving Industrial Network
Performance Assessment
Can I use Ethernet down
to the I/O level?
Should I partition my
networks at different
levels?
Where is the delay and
delay variability
occurring?
What are the industry
defacto standards?
Should I put safety, control
and diagnostics on one, two or
three networks?
What is the tradeoff
cost of a decision?
What is the tradeoff
complexity of a decision?
What is the performance cost
of security (e.g., VPN) or
application level protocols
(e.g., OPC)?
Copyright © 2006 NSF Engineering Research Center for Reconfigurable Manufacturing Systems
College of Engineering, University of Michigan. All rights reserved
James Moyne
15
Performance Issues That Should Be Addressed in
Applying Industrial Ethernet (especially at the I/O level)
• Software delays and delay variability
• Overhead for control/diagnostic and security protocols
– E.g., OPC and VPN
• Architecture best practices
• Comparison of industrial Ethernet solutions
– EtherNET/IP, Modbus/TCP, PROFINET …
• Network partitioning for safety, control and diagnostics
– Trade-offs of putting safety and control on a dedicated versus shared
network
• Industrial network time synchronization
– IEEE 1588, approaches, benefits, current issues with implementation
All Research Projects at UM-ERC
Partnering with Industry
Copyright © 2006 NSF Engineering Research Center for Reconfigurable Manufacturing Systems
College of Engineering, University of Michigan. All rights reserved
James Moyne
16
Software Delays and Delay Variability:
Where are the Sources of Delay?
100m
100m
64-byte
packet
Switched
100Mbps
Ethernet
Node A
Node B
Propagation
0.5
Transmission
5.12
Node time
330
Network
Switch latency
10
Transmission
5.12
Propagation
0.5
Copyright © 2006 NSF Engineering Research Center for Reconfigurable Manufacturing Systems
College of Engineering, University of Michigan. All rights reserved
James Moyne
17
Overhead for Control/Diagnostics Protocols
on Top of Ethernet
• UDP round-trip delays (100Mb/s
switched network)
• OPC round-trip delays (100Mb/s
switched network)
Delay and
Delay Variability
• Mean = 0.33ms, max = 1.89ms
• Stdev = 0.03ms
• Network round-trip time: 0.035ms
• Mean = 1.5ms, max = 16.8ms
• Stdev = 0.81ms
Copyright © 2006 NSF Engineering Research Center for Reconfigurable Manufacturing Systems
College of Engineering, University of Michigan. All rights reserved
James Moyne
18
Overhead for Security
Protocols on Top of Ethernet
• UDP round-trip delays (100Mb/s • VPN round-trip delays (100Mb/s
switched network)
switched network)
• Slope = 0.411s/bit
– Theory: 0.32 s/bit
• Intercept: 0.285ms
• Slope = 0.848s/bit
• Intercept: 1.07ms
• Encryption, no compression
Copyright © 2006 NSF Engineering Research Center for Reconfigurable Manufacturing Systems
College of Engineering, University of Michigan. All rights reserved
James Moyne
19
Ethernet Fieldbus Best Practices
and Comparative Analysis
• Objective
– Comparing EtherNet/IP and PROFINET in the areas of
architecture principles, technologies incorporated,
performance, diagnostic, capability, and network
management capability
– Determine best practices, technology tradeoffs and
issues associated with GM migration to Ethernet
• Deliverables
– Technology Analysis (Literature search, knowledge base,
etc.) Reports
– Peer-to-Peer Test Report
– Network Management and HMI Test Report
• Benefits
– Best practices for migration to Ethernet fieldbus
on the factory floor, including technology tradeoffs
– Address network control system technology deployment
issues before they appear on the factory floor
– Reduction of downtime due to network control system
performance and lack of technology maturity issues
– Reduction of technology and deployment ramp-up time
Copyright © 2006 NSF Engineering Research Center for Reconfigurable Manufacturing Systems
College of Engineering, University of Michigan. All rights reserved
James Moyne
20
Networks Lab
Ethernet/IP
&
ProfiNet
Project
•
•
Features
Performance
Current
Industry
NCS
Practices
PLC
Loop
Zone
PLC
Switch
Switch
Switch
PLC
Station
Switch
Switch
Switch
PLC
PLC
PLC
HMI
HMI
HMI
I/O
I/O
I/O
I/O
I/O
I/O
Copyright © 2006 NSF Engineering Research Center for Reconfigurable Manufacturing Systems
College of Engineering, University of Michigan. All rights reserved
James Moyne
21
PROFINET Observations
Copyright © 2006 NSF Engineering Research Center for Reconfigurable Manufacturing Systems
College of Engineering, University of Michigan. All rights reserved
James Moyne
22
EtherNet/IP Observations
Copyright © 2006 NSF Engineering Research Center for Reconfigurable Manufacturing Systems
College of Engineering, University of Michigan. All rights reserved
James Moyne
23
Industrial Ethernet Performance Workshop
(Apr 06): Workshop Goals
• Format
– Presentations by suppliers and users
– Round table discussion and consensus building
• Identify and reach consensus on network system
performance metrics that are practically important
– Agree on definitions for these metrics
• Develop consensus on basic statements regarding
network systems implementation and operation best
practices with respect to these metrics
• Identify ways in which the ERC can contribute to
improving network system best practices
Copyright © 2006 NSF Engineering Research Center for Reconfigurable Manufacturing Systems
College of Engineering, University of Michigan. All rights reserved
James Moyne
24
Industrial Ethernet Performance Workshop
(Apr 06):Workshop Output  Top performance issues
•
Node performance is most important performance
consideration
–
–
•
Speed, jitter, reliability, determinism
Prioritization on the network is key
Ease of use and diagnostic tools
–
•
From a plant-floor personnel perspective
Cost versus security trade-off
–
•
E.g., wireless safety, security and cost trade-offs
Time synchronization support
–
•
From a plant-floor personnel perspective
Cost of complexity
–
E.g., one or two networks for safety and control?
•
Network handling of power and topology
•
Fault handling
Copyright © 2006 NSF Engineering Research Center for Reconfigurable Manufacturing Systems
College of Engineering, University of Michigan. All rights reserved
James Moyne
25
Ethernet Performance: Control, Diagnostics
and Safety Domains
• Ethernet I/O functionality domains
– Control:
– Diagnostics:
– Safety:
Medium data volumes, high determinism
High data volumes, high speeds, low determinism
Small data volumes, very high determinism
• There is a desire to put control, diagnostics and safety on the
same industrial ethernet wire
– Functional consolidation just like vertical consolidation of Ethernet
• Since safety and control can be guaranteed with best practices
and layers on top of Ethernet it all comes down to an issue of cost
• … but the results are not as obvious as you might think…
Copyright © 2006 NSF Engineering Research Center for Reconfigurable Manufacturing Systems
College of Engineering, University of Michigan. All rights reserved
James Moyne
26
Ethernet, Performance and Control,
Diagnostics and Safety: A Three-Way Trade-off
System
Complexity
Engineering and
Maintenance Cost
Performance
Cost
Network
Delays
You need to determine the
optimal cost solution for you
For splitting up control,
diagnostics and safety
Hardware and
Installation Cost
# of Networks,
# of Nodes
And
Cost of Nodes
Costtotal = Costhardware + Costperformance+ CostEngineering
Copyright © 2006 NSF Engineering Research Center for Reconfigurable Manufacturing Systems
College of Engineering, University of Michigan. All rights reserved
James Moyne
27
Two-Level Cost Calculator
• In order to generalize the cost calculator to take
into account different units of measurement a
two tier normalized weighted cost calculator
was developed:
W
W
H
E
WC

C

C
tota
H
(
W

W

W
)
(
W

W

W
)
H
E
P
H
E
P
W
W
W
H
E
P
WCo

Co

C

C
otal
H
E
P
(
W

W

W
)
(
W

W

W
)
(
W

W

W
)
H
E
P
H
E
P
H
E
P
Costs:
H = Hardware and Installation; E = Engineering and Maintenance
P = Performance
Copyright © 2006 NSF Engineering Research Center for Reconfigurable Manufacturing Systems
College of Engineering, University of Michigan. All rights reserved
James Moyne
28
Cost Calculator: Use Case 1
• Hardware costs for many
shared nodes initially
weighted high
• A shared network is more
cost effective here
Increasing Cost 
• In this case, high initial
costs of hardware
dominate with respect to
the other factors
Shared Networks
Dedicated Networks
Shared Network
is better in this
application
Increasing Number of Nodes 
Copyright © 2006 NSF Engineering Research Center for Reconfigurable Manufacturing Systems
College of Engineering, University of Michigan. All rights reserved
James Moyne
29
Cost Calculator: Use Case 2
• Hardware costs for many
shared nodes initially
weighted lower
• The optimal choice of
dedicated or shared is not
clear
Increasing Cost 
• With initial costs weighted
lower, value over an
amortization period of
time for shared vs.
dedicated networks
remains similar
Shared Networks
Dedicated Networks
Here it could go
either way
Increasing Number of Nodes 
Copyright © 2006 NSF Engineering Research Center for Reconfigurable Manufacturing Systems
College of Engineering, University of Michigan. All rights reserved
James Moyne
30
Cost Calculator: Use Case 3
• Dedicated networks stays
lower with an increasing
amount of nodes.
• Dedicated networks are
more cost effective here
Shared Networks
Dedicated Networks
Increasing Cost 
• Weighted hardware cost
where there are few
shared nodes, a longer
amortization period and
dominant performance
costs
Dedicated
Networks are
better in this
application
Increasing Number of Nodes 
Copyright © 2006 NSF Engineering Research Center for Reconfigurable Manufacturing Systems
College of Engineering, University of Michigan. All rights reserved
James Moyne
31
Outline
• Background
– The Engineering Research Center for Reconfigurable
Manufacturing Systems (ERC-RMS)
– Reconfigurable Factory Testbed (RFT)
– Networks Performance Evaluation Efforts
• Deriving Performance Metrics for
Industrial Ethernet
– Experimental Analysis
• Key Points
Copyright © 2006 NSF Engineering Research Center for Reconfigurable Manufacturing Systems
College of Engineering, University of Michigan. All rights reserved
James Moyne
32
Key Points
• A key issue of concern in the move to Ethernet I/O is end-to-end
performance
– Consider not only network delay, but software delay
– Consider not only delay, but also delay variability (jitter)
• Understand that Ethernet application protocol overhead (e.g.,
OPC and VPN) adds to the delay consideration
• When moving to Ethernet I/O consider that there is control,
diagnostics and safety networking potential
– Determine whether this should be one, two or three networks
• High-speed Ethernet I/O applications such as motor control may
require time synchronization (IEEE 1588) and control
methodologies that leverage the associated synchronization
and time stamping
Copyright © 2006 NSF Engineering Research Center for Reconfigurable Manufacturing Systems
College of Engineering, University of Michigan. All rights reserved
James Moyne
33
For Additional Information
•
James Moyne: [email protected] / Dawn Tilbury: [email protected]
•
UM-ERC-RMS: http://erc.engin.umich.edu
•
References
–
–
–
–
–
–
F.-L. Lian, J. R. Moyne, and D. M. Tilbury, "Performance Evaluation of Control Networks: Ethernet,
ControlNet, and DeviceNet," IEEE Control Systems Magazine, 21(1), pp. 66-83, February 2001.
F.-L. Lian, J. R. Moyne, and D. M. Tilbury, "Network Design Consideration for Distributed Control Systems,"
IEEE Transactions on Control Systems Technology, 10(2), pp. 297-307, March 2002.
J. R. Moyne and D. M. Tilbury, "The Emergence of Industrial Control Networks for Manufacturing Control,
Diagnostics, and Safety Data," accepted for publication in IEEE Proceedings, December 2005.
P. G. Otanez, J. T. Parrott, J. R. Moyne, and D. M. Tilbury, "The Implications of Ethernet as a Control
Network," in Proceedings of the Global Powertrain Congress, Ann Arbor, September 2002.
J. T. Parrott, J. R. Moyne, D. M. Tilbury, "Experimental Determination of Network Quality of Service in
Ethernet: UDP, OPC, and VPN," Proceedings of the American Control Conference, Minneapolis, MN, June
2006.
11th IEEE International Conference on Emerging Technologies and Factory Automation (ETFA ’06), Prague,
Czech Republic, (Sept. 2006).
•
Marco Antolovi , Kristen Acton, Naveen Kalappa, Siddharth Mantri, Jonathan Parrott, Jon Luntz, James
Moyne & Dawn Tilbury, “PLC Communication using PROFINET: Experimental Results and Analysis”
•
Naveen Kalappa, Kristen Acton, Marco Antolovic, Siddharth Mantri, Jonathan Parrott, Jon Luntz, James
Moyne and Dawn Tilbury, “Experimental Determination of Real Time Peer to Peer Communication
Characteristics of EtherNet/IP”
•
Bradley Triden, Siddharth Mantri, Kyle Schroeder, Aditya Thomas, James Moyne, and Dawn Tilbury ,
“Dedicated vs. Shared Networks for Safety and Controls: An analysis of the trade-offs involved”
Copyright © 2006 NSF Engineering Research Center for Reconfigurable Manufacturing Systems
College of Engineering, University of Michigan. All rights reserved
James Moyne
34