Transcript PRESENTATION TITLE/SIZE 30

High Availability
Campus Networks
Tyler Creek
Consulting Systems Engineer
Southern California
© 2005 Cisco Systems, Inc. All rights reserved.
1
Agenda
• Campus High Availability
Design Principles
• Foundation Services
Flexibility
Convergence
Mobility
• Multi-Layer Design
Si
• Routed Access Design
• Summary
© 2005 Cisco Systems, Inc. All rights reserved.
Availability
Si
Security
Architectural Foundation
Hierarchical Campus Design
2
What Is High Availability?
Availability
DPM
Downtime Per Year (24x365)
99.000%
10000
3 Days
15 Hours
36 Minutes
99.500%
5000
1 Day
19 Hours
48 Minutes
99.900%
1000
8 Hours
46 Minutes
99.950%
500
4 Hours
23 Minutes
99.990%
100
53 Minutes
99.999%
10
5 Minutes
99.9999%
1
30 Seconds
More than just revenue impacted
Revenue loss
Productivity loss
Impaired financial performance
Damaged reputation
Recovery expenses
Industry Sector
Revenue/Hour
Revenue/
EmployeeHour
Energy
$2,817,846
$ 569
Telecommunications
$2,066,245
$ 186
Manufacturing
$1,610,654
$ 134
Financial Institution
$1,495,134
$1,079
Insurance
$1,202,444
$ 370
Retail
$1,107,274
$ 244
Transportation
$ 668,586
$ 107
Average
$1,010,536
$ 205
DPM—Defects per Million
To achieve five-nines
availability or better, seconds
or even milliseconds count
© 2005 Cisco Systems, Inc. All rights reserved.
3
Systematic, End-to-End Approach:
Targeting Downtime
SYSTEM LEVEL
RESILIENCY
HARDWARE
RESILIENCY
EMBEDDED
MANAGEMENT
SOFTWARE
RESILIENCY
Reliable, robust
hardware designed for
high availability
Cisco IOS® Software
functionality that
mitigates the impact
of faults
SOFTWARE
RESILIENCY
Automation
and local
action
NETWORK LEVEL
RESILIENCY
SOFTWARE
RESILIENCY
Cisco IOS Software
features for
faster network
convergence, protection,
and restoration
INVESTMENT PROTECTION IS A KEY COMPONENT
© 2005 Cisco Systems, Inc. All rights reserved.
4
System Level Resiliency Overview
Eliminate single points of failure for hardware and
software components
Control/data plane resiliency
CONTROL PLANE
• Separation of control and
forwarding plane
ACTIVE
Link resiliency
Line Card
Line Card
• Seamless software and hardware
upgrades
Micro-Kernel
Line Card
Planned outages
Line Card
• Reduced impact of line card hardware
and software failures
MANAGEMENT PLANE
• Seamless restoration of Route
Processor control and data plane
failures
STANDBY
• Fault isolation and containment
FORWARDING/DATA PLANE
© 2005 Cisco Systems, Inc. All rights reserved.
5
Network Level Resiliency Overview
Hierarchical Network Design
Service Provider
Core
• Scalability to expand/shrink without
affecting network behavior
• Predictable performance under
normal conditions and failure
conditions
Service Provider
Point of Presence
Convergence and Self-Healing
• Reduce convergence times for major
network protocols—EIGRP, OSPF,
IS-IS, BGP
Enterprise
Edge
• Leverage in network wherever
redundant paths exist
Intelligent Protocol Fabric
• Embed NSF intelligence network-wide
in Service Provider and Enterprise
networks
© 2005 Cisco Systems, Inc. All rights reserved.
Data Center
Building Block
Enterprise
Campus Core
Campus
Distribution
Layer
Campus
Access
Layer
6
Cisco Campus Architecture
One Architecture with Multiple Design Options
Enterprise Campus
Intelligent Switching
• Commonality:
Cisco Campus
Architecture
Intelligent switching
Simplified configuration
Future
Campus
Design
Options
Ω
Multi-Layer
Design
Routed
Campus
Design
Reduced network complexity
Improved network availability
Reduced management complexity
Intelligent Switching
(Hybrid of L2 + L3 features)
© 2005 Cisco Systems, Inc. All rights reserved.
7
Hierarchical Campus Design
Without a Rock Solid Foundation the Rest Doesn’t Matter
Access
Si
Distribution
Si
Core
Si
VLANs
Distribution
Access
Si
Si
Routing
Si
Spanning
Tree
Data Center
© 2005 Cisco Systems, Inc. All rights reserved.
8
Hierarchical Campus Design
Building Blocks
Access
Distribution
Core
• Offers hierarchy—each layer has
specific role
• Modular topology—building blocks
• Easy to grow, understand, and
troubleshoot
• Creates small fault domains—clear
demarcations and isolation
• Promotes load balancing and
redundancy
• Promotes deterministic traffic patterns
• Incorporates balance of both Layer 2 and
Layer 3 technology, leveraging the
strength of both
• Can be applied to all campus designs;
Multi-Layer L2/L3 and Routed Access
designs
Si
Si
Si
Si
Si
Si
Si
Si
Distribution
Si
Si
Si
Si
Si
Si
Access
WAN
© 2005 Cisco Systems, Inc. All rights reserved.
Data Center
Internet
9
Multi-Layer Reference Design
Layer 2/3 Distribution with Layer 2 Access
HSRP or GLBP
VLANs 20,120,40,140
Layer 3
Si
Si
HSRP or GLBP
VLANs 20,120,40,140
Layer 2
Distribution
Reference
Model
10.1.20.0
10.1.120.0
•
•
•
•
•
VLAN 20 Data
VLAN 120 Voice
10.1.40.0
10.1.140.0
Access
VLAN 40 Data
VLAN 140 Voice
Consider fully utilizing uplinks via GLBP
Distribution-to-distribution link required for route summarization
STP convergence not required for uplink failure/recovery
Map L2 VLAN number to L3 subnet for ease of use/management
Can easily extend VLANs across access layer switches when required
© 2005 Cisco Systems, Inc. All rights reserved.
10
Routed Campus Design
Layer 3 Distribution with Layer 3 Access
EIGRP/OSPF
EIGRP/OSPF
Si
Layer 3
Layer 3
Si
Layer 2
EIGRP/OSPF
EIGRP/OSPF
GLBP Model
10.1.20.0
10.1.120.0
VLAN 20 Data
VLAN 120 Voice
10.1.40.0
10.1.140.0
Layer 2
VLAN 40 Data
VLAN 140 Voice
• Move the Layer 2/3 demarcation to the network edge
• Upstream convergence times triggered by hardware detection
of link lost from upstream neighbor
• Beneficial for the right environment
© 2005 Cisco Systems, Inc. All rights reserved.
11
Optimal Redundancy
• Core and distribution
engineered with
redundant nodes
and links to
provide maximum
redundancy and
optimal convergence
• Network bandwidth
and capacity
engineered to
withstand node
or link failure
• Sub-Second converge
around most failure
events
Access
Distribution
Si
Si
Si
Si
Si
Si
Redundan
t
Nodes
Core
Si
Si
Distribution
Si
Si
Si
Si
Si
Si
Access
WAN
© 2005 Cisco Systems, Inc. All rights reserved.
Data Center
Internet
12
Campus Network Resilience
Sub-Second Convergence
Seconds
Convergence Times for Campus Best Practice Designs
2
1.8
1.6
1.4
1.2
1
0.8
0.6
0.4
0.2
0
L2 Access (Rapid PVST+ HSRP)
L3 Access
Multi-Layer Multi-Layer
L2 Access
L2 Access
OSPF Core* EIGRP Core
Routed
Campus
OSPF
Access*
Routed
Campus
EIGRP
Access
Worst Case Convergence for Any
Link or Platform Failure Event
*OSPF Results Require Sub-Second Timers
© 2005 Cisco Systems, Inc. All rights reserved.
13
ESE Campus Solution Test Bed
Verified Design Recommendations
Total of 68 Access Switches,
2950, 2970, 3550, 3560, 3750,
4507 SupII+, 4507SupIV, 6500
Sup2, 6500 Sup32, 6500 Sup720
and 40 APs (1200)
Three Distribution Blocks
6500 with Redundant Sup720
Si
Si
Si
Si
Si
Si
4507 with Redundant SupV
6500 with Redundant Sup720s
Si
Si
Three Distribution Blocks
6500 with Redundant Sup720s
Si
Si
Si
Si
8400 Simulated Hosts
10,000 Routes
End-to-End Flows:
TCP, UDP, RTP, IPmc
Si
Si
7206VXR NPEG1
4500 SupII+, 6500 Sup720,
FWSM, WLSM, IDSM2, MWAM
WAN
© 2005 Cisco Systems, Inc. All rights reserved.
Data Center
Internet
14
Agenda
• Campus High Availability
Design Principles
Flexibility
• Foundation Services
• Multi-Layer Design
Convergence
Mobility
Si
• Routed Access Design
• Summary
© 2005 Cisco Systems, Inc. All rights reserved.
Availability
Si
Security
Architectural Foundation
Hierarchical Campus Design
15
Best Practices—Layer 3 Routing Protocols
• Used to quickly re-route around
failed node/links while providing
load balancing over redundant
paths
• Build triangles not squares for
deterministic convergence
• Only peer on links that you
intend to use as transit
• Insure redundant L3 paths to
avoid black holes
• Summarize distribution to core
to limit EIGRP query diameter or
OSPF LSA propagation
• Tune CEF L3/L4 load balancing
hash to achieve maximum
utilization of equal cost paths
(CEF polarization)
• Utilized on both Multi-Layer and
Routed Access designs
Si
Si
Layer 3 Equal
Cost Link’s
Si
Si
Si
Si
Si
Si
Layer 3 Equal
Cost Link’s
Si
Si
WAN
© 2005 Cisco Systems, Inc. All rights reserved.
Si
Si
Si
Si
Data Center
Internet
16
Best Practice—Build Triangles Not Squares
Deterministic vs. Non-Deterministic
Triangles: Link/Box Failure Does NOT
Require Routing Protocol Convergence
Si
Si
Si
Si
Model A
Squares: Link/Box Failure Requires
Routing Protocol Convergence
Si
Si
Si
Si
Model B
• Layer 3 redundant equal cost links support fast convergence
• Hardware based—fast recovery to remaining path
• Convergence is extremely fast (dual equal-cost paths: no need for
OSPF or EIGRP to recalculate a new path)
© 2005 Cisco Systems, Inc. All rights reserved.
17
Best Practice—Passive Interfaces for IGP
Limit OSPF and EIGRP Peering Through the Access Layer
Limit unnecessary peering
Without passive interface:
Distribution
Si
Si
Routing
Updates
• Four VLANs per wiring closet,
• 12 adjacencies total
• Memory and CPU requirements
increase with no real benefit
• Creates overhead for IGP
Access
OSPF Example:
EIGRP Example:
Router(config)#router ospf 1
Router(config-router)#passiveinterface Vlan 99
Router(config)#router eigrp 1
Router(config-router)#passiveinterface Vlan 99
Router(config)#router ospf 1
Router(config-router)#passiveinterface default
Router(config-router)#no passiveinterface Vlan 99
Router(config)#router eigrp 1
Router(config-router)#passiveinterface default
Router(config-router)#no passiveinterface Vlan 99
© 2005 Cisco Systems, Inc. All rights reserved.
18
CEF Load Balancing
Avoid Underutilizing Redundant Layer 3 Paths
• The default CEF hash
‘input’ is L3
• CEF polarization: In a
multi-hop design, CEF
could select the same
left/left or right/right path
• Imbalance/overload
could occur
• Redundant paths are
ignored/underutilized
Redundant
Paths
Ignored
Access
Default L3 Hash
Distribution
Default L3 Hash
Si
L
Core
Default L3 Hash
Distribution
Default L3 Hash
Si
Si
L
Si
R
Si
Si
R
Access
Default L3 Hash
© 2005 Cisco Systems, Inc. All rights reserved.
19
CEF Load Balancing
Avoid Underutilizing Redundant Layer 3 Paths
• With defaults, CEF could
select the same left/left or
right/right paths and ignore
some redundant paths
• Alternating L3/L4 hash and
default L3 hash will give us
the best load balancing
results
• The default is L3 hash—no
modification required in core
or access
Distribution
L3/L4 Hash
Si
Si
L R
Core
Default L3 Hash
Distribution
L3/L4 Hash
• Use:
All Paths
Used
Access
Default L3 Hash
Si
L
L R
Si
R
Si
Si
mls ip cef load-sharing full
in the distribution switches to
achieve better redundant path
utilization
© 2005 Cisco Systems, Inc. All rights reserved.
Access
Default L3 Hash
L
Left Side
Shown
20
Single Points of Termination
SSO/NSF Avoiding Total Network Outage
L2 = SSO
L3 =
SSO/NSF
Access
Distribution
Si
Si
Si
Si
Si
Si
Core
Si
Si
• The access layer and other single points of failure are candidates for
supervisor redundancy
• L2 access layer SSO
• L3 access layer SSO and NSF
• Network outage until physical replacement or reload vs
one to three seconds
© 2005 Cisco Systems, Inc. All rights reserved.
21
Campus Multicast
Which PIM Mode—Sparse or Dense
“Sparse mode Good! Dense
mode Bad!”
Source: “The Caveman’s Guide to IP Multicast”, ©2000, R. Davis
© 2005 Cisco Systems, Inc. All rights reserved.
22
PIM Design Rules for Routed Campus
• Use PIM sparse mode
• Enable PIM sparse mode on
ALL access, distribution and
core layer switches
• Enable PIM on ALL interfaces
• Use Anycast RPs in the core
for RP redundancy and fast
convergence
• Define the Router-ID to prevent
Anycast IP address overlap
• IGMP-snooping is enabled
when PIM is enabled on a
VLAN interface (SVI)
• (Optional) use garbage can
RP to black-hole unassigned
IPmc traffic
Si
Si
RP-Left
10.122.100.1
Si
Si
Si
Si
Si
Si
Si
Si
Si
Si
RP-Right
10.122.100.1
Si
Si
Call Manager IP/TV Server
w/MoH
Internet
WAN
IPmc Sources
© 2005 Cisco Systems, Inc. All rights reserved.
23
Multicast in the Campus
interface loopback 0
ip address 10.0.0.1 255.255.255.255
interface loopback 0
ip address 10.0.0.1 255.255.255.255
interface loopback 1
ip address 10.0.0.3 255.255.255.255
!
ip msdp peer 10.0.0.2 connect-source loopback 1
ip msdp originator-id loopback 1
!
interface TenGigabitEthernet M/Y
ip address 10.122.0.X 255.255.255.252
ip pim sparse-mode
interface loopback 1
ip address 10.0.0.2 255.255.255.255
!
ip msdp peer 10.0.0.3 connect-source loopback 1
ip msdp originator-id loopback 1
!
interface TenGigabitEthernet M/Y
ip address 10.122.0.X 255.255.255.252
ip pim sparse-mode
!
!
MSDP
ip pim rp-address 10.0.0.1
Core
Layer 3
ip pim rp-address 10.0.0.1
Si
Si
Core-A
Core-B
Distribution
Layer 2/3
Distribution-A
ip pim rp-address 10.0.0.1
!
interface Y
description GigE to Access/Core
ip address
10.122.0.Y 255.255.255.252
Access
ip pim sparse-mode
!<snip>
Distribution-B
Si
© 2005 Cisco Systems, Inc. All rights reserved.
IGMP snooping
on by default
Si
ip pim rp-address 10.0.0.1
!
interface Y
description GigE to Access/Core
ip address 10.122.0.Y 255.255.255.252
ip pim sparse-mode
!<snip>
24
Best Practices—UDLD Configuration
• Typically deployed on any
fiber optic interconnection
• Use UDLD aggressive
mode for best protection
• Turn on in global
configuration to avoid
operational
error/“misses”
Si
Si
Si
Si
Si
Si
Fiber Interconnections
Layer 3 Equal
Cost Link’s
Layer 3 Equal
Cost Link’s
Si
Si
• Config example
Cisco IOS Software: udld
aggressive
Si
Si
Si
Si
Si
Si
CatOS: set udld enable
set udld aggressive-mode
enable <mod/port>
WAN
© 2005 Cisco Systems, Inc. All rights reserved.
Data Center
Internet
25
Best Practices—
EtherChannel Configuration
• Typically deployed in
distribution to core, and core
to core interconnections
• Used to provide link
redundancy—while reducing
peering complexity
• Tune L3/L4 load balancing
hash to achieve maximum
utilization of channel members
Si
Si
Si
Si
Si
Si
Layer 3 Equal
Cost Link’s
Layer 3 Equal
Cost Link’s
Si
Si
• Match CatOS and Cisco IOS
Software PAgP settings
• 802.3ad LACP for interop if you
need it
Si
Si
Si
Si
Si
Si
• Disable unless needed
CatOS: set port host
Cisco IOS Software: switchport
host
© 2005 Cisco Systems, Inc. All rights reserved.
WAN
Data Center
Internet
27
EtherChannel Load Balancing
Avoid Underutilizing Redundant Layer 2 Paths
L3 Hash
Link 0 load—68%
• Network did not load
balance using default L3
load balancing hash
Common IP addressing scheme
Si
72 access subnets addressed
uniformly from 10.120.x.10 to
10.120.x.215
Si
Link 1 load—32%
L4 Hash
Link 0 load—52%
• Converted to L4 load
balancing hash and
achieved better load sharing
Si
Si
Link 1 Load—48%
cr2-6500-1(config)#port-channel load-balance src-dst-port
© 2005 Cisco Systems, Inc. All rights reserved.
28
PAgP Tuning
PAgP Default Mismatches
Matching EtherChannel Configuration on Both Sides
Improves Link Restoration Convergence Times
set port channel <mod/port> on/off
Time to Converge in
Seconds
7
6
As Much as
Seven Seconds
of Delay/Loss
Tuned Away
5
4
3
6500 (CatOS)
4006 (CatOS)
2
1
0
PAgP Mismatch
© 2005 Cisco Systems, Inc. All rights reserved.
PAgP Off
29
Mitigating Plug and Players
Protecting Against Well-Intentioned Users
Cisco Secure
ACS
Network Instability
Unauthorized
Switch
Unauthorized
Switch Incorrect
STP Info
BPDU Guard
Root Guard
Enterprise
Server
Enterprise
Server
Authorized
Switch
Authorized
Switch
PROBLEM:
• Well-intentioned users place
unauthorized network devices
on the network possibly
causing instability
© 2005 Cisco Systems, Inc. All rights reserved.
SOLUTION:
• Cisco Catalyst® switches
support rogue BPDU filtering:
BPDU Guard, Root Guard
30
BPDU Guard
Prevent Loops via WLAN (Windows XP Bridging)
PROBLEM:
• WLAN AP’s do not
forward BPDU’s
• Multiple Windows XP
machines can create a
loop in the wired VLAN
via the WLAN
STP Loop
Formed
BPDU Guard
Disables Port
SOLUTION:
• BPDU Guard configured
on all end station switch
ports will prevent loop
from forming
BPDU
Generated
Win XP
Bridging
Enabled
© 2005 Cisco Systems, Inc. All rights reserved.
BPDU
Discarded
Win XP
Bridging
Enabled
31
Cisco Catalyst Integrated Security Features
• Port security prevents
MAC flooding attacks
• DHCP snooping
prevents client attack on
the switch and server
• Dynamic ARP Inspection
adds security to ARP
using DHCP snooping
table
IP Source Guard
Dynamic ARP Inspection
DHCP Snooping
Port Security
• IP source guard adds
security to IP source
address using DHCP
snooping table
© 2005 Cisco Systems, Inc. All rights reserved.
32
Cisco Catalyst 6500 High Availability Leadership
Maximizing Uptime
Cisco IOS Software Modularity
New!
• Subsystem In-Service Software Upgrades (ISSU)
• Stateful Process Restarts
• Fault Containment, Memory Protection
Non-Stop Forwarding/
Stateful Switch Over (NSF/SSO)
• Traffic continues flowing after a
primary supervisor failure
• Sub-second recovery in L2 and L3
networks
Catalyst
6500
Generic Online Diagnostics
(GOLD)
• Proactively detect and address
potential hardware and software
faults in the switch before they
adversely impact network traffic
Physical Redundancy
• Redundant supervisors, power supplies,
switch fabrics, and clocks
© 2005 Cisco Systems, Inc. All rights reserved.
33
Catalyst 6500 with IOS Modularity
Infrastructure Enhancements
Cisco IOS Software Modularity
• IOS with Modularity
Fault Containment
etc
INETD
CDP
FTP
EEM
UDP
TCP
Routing
Base
Protected Memory
Restartable Processes
– 20+ independent processes
High Availability Infrastructure
– Remaining feature subsystems live in
IOS Base process
Network Optimized Microkernel
Catalyst 6500 Data Plane
Subsystem ISSU
• Embedded Event Manager
Create TCL policy scripts to program the
Catalyst 6500
– When detect event X, then do action Y
• Generic On-Line Diagnostics (GOLD)
supports pro-active diagnosis of faults
before they become a problem…
© 2005 Cisco Systems, Inc. All rights reserved.
34
Software Modularity
Minimize Unplanned Downtime
etc
INETD
CDP
FTP
EEM
UDP
TCP
Routing
Base
Cisco IOS Software Modularity
If an error occurs in a
modular process…
• HA subsystem determines the
best recovery action
High Availability Infrastructure
Restart a modular process
Network Optimized Microkernel
Switchover to standby supervisor
Cisco Catalyst 6500 Data Plane
Remove the system from the
network
• Process restarts with no
impact on the data plane
Traffic forwarding continues during
unplanned process restarts
Utilizes Cisco Nonstop
Forwarding (NSF) where
appropriate
State Checkpointing allows quick
process recovery
© 2005 Cisco Systems, Inc. All rights reserved.
35
Software Modularity
Simplify Software Changes
•
etc
INETD
CDP
FTP
EEM
UDP
TCP
Routing
Base
Routing
Cisco IOS Software Modularity
If the software needs to be upgraded
(for example, to protect against a
new security vulnerability)…
The change can be made
available as an individual
patch which reduces code
certification time
High Availability Infrastructure
Network Optimized Microkernel
Code Certification
Code Deployment
Catalyst 6500 Data Plane
Time
•
Traffic forwarding continues during
planned software changes
Subsystem In-Service Software
Upgrade (ISSU)* allows the
change to be applied with no
service disruption
*for all modularized processes
© 2005 Cisco Systems, Inc. All rights reserved.
36
Agenda
• Campus High Availability
Design Principles
Flexibility
• Foundation Services
• Multi-Layer Design
Convergence
Mobility
• Routed Access Design
Si
Availability
Si
Security
• Summary
Architectural Foundation
Hierarchical Campus Design
© 2005 Cisco Systems, Inc. All rights reserved.
37
Why Multi-Layer Campus Design?
Non-Looped
Looped
Layer 3
Si
Layer 2
Si
Layer 2
Si
Si
Distribution
Layer 2
Access
• Most widely deployed campus design
• Supports the spanning of VLANs and Subnets across multiple
access layer switches
• Leverages the strength of both Layer 2 and Layer 3
capabilities
• Supported on all models of Cisco Catalyst Switches
© 2005 Cisco Systems, Inc. All rights reserved.
38
Multi-Layer Design
Best Practices—Spanning VLANs
• ONLY when you have to!
• More common in the
data center
• Required when a VLAN spans
access layer switches
• Required to protect against
‘user side’ loops
• Use Rapid PVST+ for best
convergence
• Take advantage of the
Spanning Tree Toolkit
Same VLAN
Same VLAN
Layer2 Loops
Si
Si
Si
Si
Si
Si
Si
Si
Si
WAN
Si
Si
Layer 3 Equal
Cost Link’s
Layer 3 Equal
Cost Link’s
Si
© 2005 Cisco Systems, Inc. All rights reserved.
Same VLAN
Si
Si
Data Center
Internet
39
PVST+ and Rapid PVST+, MST
Spanning Tree Toolkit, 802.1d, 802.1s, 802.1w
A
B
• 802.1D-1998: Classic Spanning Tree Protocol (STP)
• 802.1D-2004: Rapid Spanning Tree Protocol (RSTP = 802.1w)
• 802.1s: Multiple Spanning Tree Protocol (MST)
• 802.1t: 802.1d Maintenance, 802.1Q: VLAN Tagging (Trunking)
• PVST+: an instance of STP (802.1D-1998) per VLAN + Portfast, Uplinkfast,
BackboneFast, BPDUGuard, BPDUFilter, RootGuard, and LoopGuard
• Rapid PVST+: an instance of RSTP (802.1D-2004 = 802.1w) per VLAN +
Portfast, BPDUGuard, BPDUFilter, RootGuard, and LoopGuard
• MST (802.1s): up to 16 instances of RSTP (802.1w); combining many VLANS
with the same physical and logical topology into a common RSTP instance;
additionally Portfast, BPDUGuard, BPDUFilter, RootGuard, and LoopGuard
are supported with MST
© 2005 Cisco Systems, Inc. All rights reserved.
40
Spanning Tree Toolkit
• PortFast*: Bypass listening-learning
phase for access port
Root
• UplinkFast: Three to five seconds
convergence after link failure
Si
• BackboneFast: Cuts convergence time
by Max_Age for indirect failure
F
Distribution
Switches
F
F
F
• LoopGuard*: Prevents alternate or root
port to become designated in absence
of BPDUs
• RootGuard*: Prevents external switches
from becoming root
• BPDUGuard*: Disable PortFast enabled
port if a BPDU is received
Si
X
F
Wiring
B Closet
Switch
• BPDUFilter*: Do not send or receive
BPDUs on PortFast enabled ports
* Also Supported with MST and Rapid PVST+
© 2005 Cisco Systems, Inc. All rights reserved.
41
Layer 2 Hardening
Spanning Tree Should Behave the Way You Expect
• Place the Root where you want it
Root Primary/Secondary Macro
• The root bridge should stay
where you put it
Rootguard
Loopguard
UplinkFast (Classic STP only)
UDLD
• Only end station traffic should
be seen on an edge port
BPDU Guard
Root Guard
PortFast
Port-security
Loopguard
STP Root
Si
Si
Rootguard
Loopguard
UplinkFast
BPDU Guard or
Rootguard
PortFast
© 2005 Cisco Systems, Inc. All rights reserved.
42
Optimizing Convergence: PVST+ or Rapid PVST+
802.1d + Extensions or 802.1s + Extensions
• Rapid-PVST+ greatly improves the restoration times for any
VLAN that requires a topology convergence due to link UP
• Rapid-PVST+ also greatly improves convergence time over
backbone network fast for any indirect link failures
Timed to Converge in
Seconds
35
To Access
To Server Farm
30
25
30 Seconds of
Delay/Loss
Tuned Away
20
15
10
5
0
PVST+
© 2005 Cisco Systems, Inc. All rights reserved.
Rapid PVST+
43
Multi-Layer Design
Best Practices—Trunk Configuration
• Typically deployed on
interconnection between
access and distribution layers
802.1q Trunks
• Use VTP transparent mode to
decrease potential for
operational error
• Hard set trunk mode to on and
encapsulation negotiate off for
optimal convergence
Si
Si
Layer 3 Equal
Cost Link’s
Si
Si
Si
Si
Si
Si
Layer 3 Equal
Cost Link’s
• Change the native VLAN to
something unused to avoid
VLAN hopping
• Manually prune all VLANS
except those needed
Si
Si
Si
Si
Si
Si
• Disable on host ports:
CatOS: set port host
Cisco Cisco IOS: switchport host
© 2005 Cisco Systems, Inc. All rights reserved.
WAN
Data Center
Internet
44
Optimizing Convergence: Trunk Tuning
Trunk Auto/Desirable Takes Some Time
• DTP negotiation tuning improves link up convergence time
CatOS> (enable) set trunk <port> nonegotiate dot1q <vlan>
IOS(config-if)# switchport mode trunk
IOS(config-if)# switchport nonegotiate
Time to Converge in Seconds
2.5
3550 (Cisco IOS)
4006 (CatOS)
4507 (Cisco IOS)
6500 (CatOS)
2
1.5
Si
Two Seconds
of Delay/Loss
Tuned Away
1
0.5
Voice Data
0
Trunking Desirable
© 2005 Cisco Systems, Inc. All rights reserved.
Trunking Nonegotiate
45
Multi-Layer Design
Best Practices—First Hop Redundancy
• Used to provide a resilient
default gateway/first hop
address to end stations
1st Hop Redundancy
• HSRP, VRRP, and GLBP
alternatives
• VRRP, HSRP and GLBP
provide millisecond timers
and excellent convergence
performance
Si
Si
Si
Si
Si
Si
Layer 3 Equal
Cost Link’s
Layer 3 Equal
Cost Link’s
Si
Si
• VRRP if you need multivendor interoperability
• GLBP facilitates uplink load
balancing
Si
Si
Si
Si
Si
Si
• Tune preempt timers to avoid
black-holed traffic
WAN
© 2005 Cisco Systems, Inc. All rights reserved.
Data Center
Internet
46
Optimizing Convergence: HSRP Timers
HSRP Millisecond Convergence
• HSRP = default gateway
redundancy; effects traffic
out of the access layer
Layer 2 Link’s
Si
Si
Si
Si
Layer 3 Equal
Cost Link’s
Layer 3 Equal
Cost Link’s
Si
Si
Si
Si
Si
Si
WAN
Si
Si
interface Vlan5
description Data VLAN for 6k-access
ip address 10.1.5.3 255.255.255.0
ip helper-address 10.5.10.20
no ip redirects
ip pim query-interval 250 msec
ip pim sparse-mode
logging event link-status
standby 1 ip 10.1.5.1
standby 1 timers msec 200 msec 750
standby 1 priority 150
standby 1 preempt
standby 1 preempt delay minimum 180
Si
Si
Data Center
Internet
© 2005 Cisco Systems, Inc. All rights reserved.
47
Optimizing Convergence: HSRP Preempt Delay
Preempt Delay Needs to Be Longer Than Box Boot Time
Without Increased Preempt Delay HSRP Can Go
Active Before Box Completely Ready to Forward
Traffic L1 (Boards), L2 (STP), L3 (IGP Convergence)
standby 1 preempt delay minimum 180
Time to Converge in Seconds
Test Tool Timeout—30 Seconds
30
25
More Than
30 Seconds of
Delay/Loss
Tuned Away
20
15
10
3550 (Cisco IOS)
2950 (Cisco IOS)
4506 (CatOS)
4507 (Cisco IOS)
6500 (CatOS)
6500 (Cisco IOS)
5
0
No Preempt Delay
© 2005 Cisco Systems, Inc. All rights reserved.
Prempt Delay Tuned
48
First Hop Redundancy with GLBP
Cisco Designed, Load Sharing, Patent Pending
R1- AVG; R1, R2 Both Forward Traffic
• All the benefits of
HSRP plus load
balancing of default
gateway  utilizes all
available bandwidth
• A group of routers
function as one virtual
router by sharing one
virtual IP address but
using multiple virtual
MAC addresses for
traffic forwarding
• Allows traffic from a
single common subnet
to go through multiple
redundant gateways
using a single virtual
IP address
GLBP AVF,SVF
GLBP AVG/AVF,SVF
IP:
10.0.0.253
MAC: 0000.0C78.9abc
vIP: 10.0.0.10
vMAC: 0007.b400.0102
IP:
10.0.0.254
MAC: 0000.0c12.3456
vIP: 10.0.0.10
vMAC: 0007.b400.0101
R1
Si
Si
Distribution-B
GLPB AVF,SVF
Distribution-A
GLBP AVG/AVF, SVF
© 2005 Cisco Systems, Inc. All rights reserved.
Access-a
IP:
MAC:
GW:
ARP:
10.0.0.1
aaaa.aaaa.aa01
10.0.0.10
0007.B400.0101
IP:
MAC:
GW:
ARP:
10.0.0.2
aaaa.aaaa.aa02
10.0.0.10
0007.B400.0102
IP:
MAC:
GW:
ARP:
10.0.0.3
aaaa.aaaa.aa03
10.0.0.10
0007.B400.0101
49
Optimizing Convergence: VRRP, HSRP,
GLBP Mean, Max, and Min—Are There Differences?
• VRRP does not have sub-second timers and all flows go through a
common VRRP peer; mean, maximum, and minimum are equal
• HSRP has sub-second timers; however all flows go through same
HSRP peer so there is no difference between mean, maximum, and
minimum
• GLBP has sub-second timers and distributes the load amongst
the GLBP peers; so 50% of the clients are not effected by an
uplink failure
Si
Si
Time in Seconds to Converge
Distribution to Access Link Failure
Access to Server Farm
1.2
VRRP
HSRP
GLBP
50% of Flows
Have ZERO
Loss W/ GLBP
1
0.8
GLBP Is 50%
Better
0.6
0.4
0.2
0
Longest
© 2005 Cisco Systems, Inc. All rights reserved.
Shortest
Average
50
If You Span VLANS Tuning Required
By Default Half the Traffic Will Take a Two Hop L2 Path
• Both distribution switches act as default gateway
• Blocked uplink caused traffic to take less than optimal path
Core
Layer 3
Distribution
Layer 2/3
Core
Distribution-A
GLBP Virtual
MAC 1
Distribution-B
GLBP Virtual MAC 2
Si
Si
F: Forwarding
Access
Layer 2
B: Blocking
Access-a
Access-b
VLAN 2
VLAN 2
© 2005 Cisco Systems, Inc. All rights reserved.
51
GLBP + STP turning
Change the Blocking Interfaces + VLAN per Access
1.
2.
Force STP to block the interface between the distribution switches
Use the fewest possible VLANs per access switch
Core
Layer 3
Distribution
Layer 2/3
Core
Distribution-A
GLBP Virtual
MAC 1
Si
Distribution-B
GLBP Virtual MAC 2
Si
STP Port
Cost
Increased
Access
Layer 2
Access-a
Access-b
VLAN 2
VLAN 3
© 2005 Cisco Systems, Inc. All rights reserved.
52
Asymmetric Routing (Unicast Flooding)
• Affects redundant
topologies with
shared L2 access
• One path upstream
and two paths
downstream
• CAM table entry
ages out on
standby HSRP
Asymmetric
Equal Cost
Return Path
CAM Timer Has
Aged out on
Standby HSRP
Si
Si
• Without a CAM
Downstream
entry packet is
Packet
Flooded
flooded to all ports
in the VLAN
VLAN 2
© 2005 Cisco Systems, Inc. All rights reserved.
VLAN 2
VLAN 2
Upstream Packet
Unicast to Active
HSRP
VLAN 2
53
Best Practices Prevent Unicast Flooding
• Assign one unique
voice and as few
data VLAN’s as
possible to each
access switch
• Traffic is now only
flooded down
one trunk
• Access switch
unicasts correctly;
no flooding to
all ports
• If you have to:
Tune ARP and CAM
aging timers; CAM
timer exceeds
ARP timer
Bias routing metrics
to remove equal
cost routes
Asymmetric
Equal Cost
Return Path
Downstream
Packet
Flooded on
Single Port
VLAN 3
© 2005 Cisco Systems, Inc. All rights reserved.
Si
VLAN 4
Si
VLAN 5
Upstream Packet
Unicast to Active
HSRP
VLAN 2
54
Keep Redundancy Simple
“If Some Redundancy Is
Good, More Redundancy
Is NOT Better”
• Root placement?
• How many
blocked links?
• Convergence?
• Complex fault resolution
© 2005 Cisco Systems, Inc. All rights reserved.
55
But Not Too Simple…
What Happens if You Don’t Link the Distributions?
• STP’s slow convergence can cause
considerable periods of traffic loss
Core
• STP could cause non-deterministic
traffic flows/link load engineering
• STP convergence will cause
Layer 3 convergence
STP Root and
HSRP Active
STP Secondary
Root and HSRP
Standby
Hellos
Si
Si
F 2
B
2
• STP and Layer 3 timers are
independent
• Unexpected Layer 3 convergence
and re-convergence could occur
• Even if you do link the distribution
switches dependence on STP and
link state/connectivity can cause
HSRP irregularities and
unexpected state transitions
© 2005 Cisco Systems, Inc. All rights reserved.
Access-a
VLAN 2
Traffic
Dropped Until
Transition to
Forwarding;
As much as 50
Seconds
Access-b
VLAN 2
Traffic
Dropped Until
MaxAge
Expires Then
Listening and
Learning
56
What If You Don’t?
Black Holes and Multiple ‘Transitions’…
• Aggressive HSRP
Core
Layer 3
Distribution
Layer 2/3
Core
STP Root and
HSRP Active
STP
Secondary
Root and
HSRP Standby
timers limit black
hole #1
• Backbone fast limits
HSRPtime
Active
(30 seconds) to
(Temporarily)
event #2
Hellos
Si
Si
• Even with Rapid
PVST+ at least one
second before event #2
F: Forwarding
Access
Layer 2
B: Blocking
Access-a
Access-b
VLAN 2
VLAN 2
MaxAge
Seconds Before
Failure Is
Detected….Then
Listening and
Learning
• Blocking link on access-b will take 50 seconds to move to forwarding  traffic black hole
until HSRP goes active on standby HSRP peer
• After MaxAge expires (or backbone fast or Rapid PVST+) converges HSRP preempt
causes another transition
• Access-b used as transit for access-a’s traffic
© 2005 Cisco Systems, Inc. All rights reserved.
57
What If You Don’t?
Return Path Traffic Black Holed…
Core
Layer 3
Distribution
Layer 2/3
Core
• 802.1d: up to 50
STP
Secondary
Root and
HSRP Standby
seconds
• PVST+: backbone
STP Root and
HSRP Active
• Rapid PVST+:
Hellos
address by the
protocol (one
second)
Si
Si
fast 30 seconds
F: Forwarding
Access
Layer 2
B: Blocking
Access-a
Access-b
VLAN 2
VLAN 2
• Blocking link on access-b will take 50 seconds to move to
forwarding return traffic black hole until then
© 2005 Cisco Systems, Inc. All rights reserved.
58
Layer 2 Distribution Interconnection
Redundant Link from Access Layer Is Blocked
HSRP Active
and STP Root
VLAN 20,140
Layer 2
Si
Trunk
Si
HSRP Active
and STP Root
VLAN 40,120
Distribution
Layer 2
Links
Layer 2
Links
STP Model
Access
10.1.20.0
10.1.120.0
VLAN 20 Data
VLAN 120 Voice
10.1.40.0
10.1.140.0
VLAN 40 Data
VLAN 140 Voice
• Use only if Layer 2 VLAN spanning flexibility required
• STP convergence required for uplink failure/recovery
• More complex as STP root and HSRP should match
• Distribution-to-distribution link required for route summarization
© 2005 Cisco Systems, Inc. All rights reserved.
59
Layer 3 Distribution Interconnection
No Spanning Tree—All Links Active
Layer 3
HSRP Active
VLAN 20,140
Si
Si
HSRP Active
VLAN 40,120
Distribution
Layer 2
Links
Layer 2
Links
HSRP Model
Access
10.1.20.0
10.1.120.0
VLAN 20 Data
VLAN 120 Voice
10.1.40.0
10.1.140.0
VLAN 40 Data
VLAN 140 Voice
• Recommended ‘best practice’—tried and true
• No STP convergence required for uplink failure/recovery
• Distribution-to-distribution link required for route summarization
• Map L2 VLAN number to L3 subnet for ease of use/management
© 2005 Cisco Systems, Inc. All rights reserved.
60
Layer 3 Distribution Interconnection
GLBP Gateway Load Balancing Protocol
Layer 3
GLBP Active
VLAN 20,120,40,140
Si
Si
GLBP Active
VLAN 20,120, 40, 140
Distribution
Layer 2
Links
Layer 2
Links
GLBP Model
Access
10.1.20.0
10.1.120.0
VLAN 20 Data
VLAN 120 Voice
10.1.40.0
10.1.140.0
VLAN 40 Data
VLAN 140 Voice
• Fully utilize uplinks via GLBP
• Distribution-to-distribution required for route summarization
• No STP convergence required for uplink failure/recovery
© 2005 Cisco Systems, Inc. All rights reserved.
61
Layer 3 Distribution Interconnection
Reference Design—No VLANs Span Access Layer
• Tune CEF load balancing
• Match CatOS/IOS Etherchannel
settings and tune load
balancing
Si
Si
Core
• Summarize routes
towards core
• Limit redundant IGP peering
• STP Root and HSRP primary
tuning or GLBP to load balance
on uplinks
Layer 3
Si
P-t-P Link
Distribution
Si
• Set trunk mode on/nonegotiate
• Disable Etherchannel
unless needed
• Set Port Host
on access layer ports:
Disable Trunking
Disable Etherchannel
Enable PortFast
• RootGuard or BPDU-Guard
• Use security features
Access
VLAN 20 Data
10.1.20.0/24
VLAN 120 Voice
10.1.120.0/24
© 2005 Cisco Systems, Inc. All rights reserved.
VLAN 40 Data
10.1.40.0/24
VLAN 140 Voice
10.1.140.0/24
62
Layer 2 Distribution Interconnection
Some VLANs Span Access Layer
• Tune CEF load balancing
• Match CatOS/IOS Etherchannel
settings and tune load balancing
• Summarize routes towards core
Si
Si
• Limit redundant IGP peering
• STP Root and HSRP primary or GLBP
and STP port cost tuning to load
balance on uplinks
Layer 2
• Set trunk mode on/nonegotiate
Trunk
Si
Si
• Disable Etherchannel unless needed
• RootGuard on downlinks
• LoopGuard on uplinks
• Set port host
on access Layer ports:
Disable trunking
Disable Etherchannel
Enable PortFast
VLAN 20 Data
VLAN 40 Data
• RootGuard or
10.1.20.0/24
10.1.40.0/24
BPDU-Guard
VLAN 120 Voice
VLAN 140 Voice
10.1.120.0/24
10.1.140.0/24
• Use security features
VLAN 250 WLAN
Core
Distribution
Access
10.1.250.0/24
© 2005 Cisco Systems, Inc. All rights reserved.
63
Agenda
• Campus High Availability
Design Principles
• Foundation Services
• Multi-Layer Design
Flexibility
• Routed Access Design
EIGRP Design Details
Convergence
Mobility
Si
Availability
Si
Security
OSPF Design Details
• Summary
© 2005 Cisco Systems, Inc. All rights reserved.
Architectural Foundation
Hierarchical Campus Design
64
Why Routed Access Campus Design?
Si
Si
Distribution
Layer 3
Si
•
•
•
•
Layer 2
Si
Access
Most Cisco Catalyst routers support L3 switching today
EIGRP/OSPF routing preference over spanning tree
IGP enhancements; stub router/area, fast reroute, etc..
Single control plane and well known tool set
Traceroute, show ip route, show ip eigrp neighbor, etc…
• It is another design option available to you
© 2005 Cisco Systems, Inc. All rights reserved.
65
Ease of Implementation
• Less to get right:
No STP feature placement core
to distribution
LoopGuard
RootGuard
STP Root
No default gateway redundancy
setup/tuning
No matching of STP/HSRP priority
No L2/L3 multicast topology
inconsistencies
© 2005 Cisco Systems, Inc. All rights reserved.
66
Ease of Troubleshooting
• Routing troubleshooting tools
Show IP route
Traceroute
Ping and extended pings
Extensive protocol debugs
Consistent troubleshooting; access, dist, core
• Bridging troubleshooting tools
Show ARP
Show spanning-tree, standby, etc…
Multiple show CAM dynamic’s to find a host
• Failure differences
Routed topologies fail closed—i.e. neighbor loss
Layer 2 topologies fail open—i.e. broadcast and unknowns flooded
© 2005 Cisco Systems, Inc. All rights reserved.
67
Routed Campus Design
Seconds
Resiliency Advantages? Yes, with a Good Design
2
1.8
1.6
1.4
1.2
1
0.8
0.6
0.4
0.2
0
Si
Si
Si
Si
Upstream
Downstream
Multilayer
RPVST+
Routed Access
OSPF
Routed Access
EIGRP
• Sub-200 msec convergence for EIGRP and OSPF
• OSPF convergence times dependent on timer tuning
A
B
• RPVST+ convergence times dependent on GLBP/HSRP tuning
© 2005 Cisco Systems, Inc. All rights reserved.
68
Routed Access Considerations
• Do you have any Layer 2 VLAN adjacency
requirements between access switches?
• IP addressing—Do you have enough address
space and the allocation plan to support a
routed access design?
• Platform requirements;
Cisco Catalyst 6500 requires an MSFC in the access to get all the
necessary switchport and routing features
Cisco Catalyst 4500 requires a SUP4/5 for EIGRP or OSPF support
Cisco Catalyst 3500s and 3700s require an enhanced Cisco IOS Software
image for IGRP and OSPF
© 2005 Cisco Systems, Inc. All rights reserved.
69
EIGRP vs. OSPF as Your Campus IGP
DUAL vs. Dijkstra
• Convergence:
Within the campus environment,
both EIGRP and OSPF provide
extremely fast convergence
2
EIGRP requires summarization
1.8
1.6
OSPF requires summarization and
timer tuning for fast convergence
1.4
1.2
• Flexibility:
EIGRP supports multiple levels of
route summarization and route
filtering which simplifies migration
from the traditional Multi-Layer L2/L3
campus design
OSPF area design restrictions need
to be considered
Upstream
Downstream
1
0.8
0.6
0.4
0.2
0
OSPF
OPSF 12.2S
EIGRP
• Scalability:
Both protocols can scale to support
very large enterprise network
topologies
© 2005 Cisco Systems, Inc. All rights reserved.
70
Routed Access Design
High-Speed Campus Convergence
• Convergence is the time needed
for traffic to be rerouted to the
alternative path after the
network event
• Network convergence requires
all affected routers to process
the event and update the
appropriate data structures used
for forwarding
Si
Si
Si
Si
• Network convergence is the time
required to:
Detect the event
Propagate the event
Process the event
Update the routing table/FIB
© 2005 Cisco Systems, Inc. All rights reserved.
71
Agenda
• Campus High Availability
Design Principles
• Foundation Services
• Multi-Layer Design
Flexibility
• Routed Access Design
EIGRP Design Details
Convergence
Mobility
Si
Availability
Si
Security
OSPF Design Details
• Summary
© 2005 Cisco Systems, Inc. All rights reserved.
Architectural Foundation
Hierarchical Campus Design
72
Strengths of EIGRP
• Advanced distance vector
• Maps easily to the traditional Multi-Layer design
• 100% loop free
• Fast convergence
• Easy configuration
• Incremental update
• Supports VLSM and discontiguous network
• Classless routing
• Protocol independent
IPv6, IPX and AppleTalk
• Unequal cost paths load balancing
• Flexible topology design options
© 2005 Cisco Systems, Inc. All rights reserved.
73
EIGRP Design Rules for HA Campus
Similar to WAN Design, But…
• EIGRP design for the campus
follows all the same best
practices as you use in the WAN
with a few differences
Si
Si
No BW limitations
Lower neighbor counts
Direct fiber interconnects
Lower cost redundancy
Si
Si
HW switching
• WAN  stability and speed
• Campus  stability, redundancy,
load sharing, and high speed
© 2005 Cisco Systems, Inc. All rights reserved.
74
EIGRP in the Campus
Conversion to an EIGRP Routed Edge
• The greatest advantages of
extending EIGRP to the access
are gained when the network
has a structured addressing
plan that allows for use of
summarization and stub routers
• EIGRP provides the ability to
implement multiple tiers of
summarization and route
filtering
• Relatively painless to migrate
to a L3 access with EIGRP if
network addressing scheme
permits
10.10.0.0/16
Si
Si
10.10.128.0/17
Si
10.10.0.0/17
Si
Si
Si
• Able to maintain a deterministic
convergence time in very large
L3 topology
© 2005 Cisco Systems, Inc. All rights reserved.
75
EIGRP Design Rules for HA Campus
Limit Query Range to Maximize Performance
• EIGRP convergence is
largely dependent on query
response times
Si
Si
• Minimize the number of
queries to speed up
convergence
• Summarize distribution block
routes upstream to the core
Si
Si
Upstream queries are returned
immediately with infinite cost
• Configure all access switches
as EIGRP stub routers
No downstream queries are
ever sent
© 2005 Cisco Systems, Inc. All rights reserved.
76
EIGRP Neighbors
Event Detection
• EIGRP neighbor relationships are created
when a link comes up and routing adjacency
is established
• When physical interface changes state, the
routing process is notified
Si
Si
Routed
Interface
Carrier-delay should be set as a rule because
it varies based upon the platform
Hello’s
• Some events are detected by the
routing protocol
Neighbor is lost, but interface is UP/UP
• To improve failure detection
Use Routed Interfaces and not SVIs
Decrease interface carrier-delay to 0
Decrease EIGRP hello and hold-down timers
Hello = 1
Hold-down = 3
Si
Si
L2 Switch
or VLAN Interface
Si
interface GigabitEthernet3/2
ip address 10.120.0.50 255.255.255.252
ip hello-interval eigrp 100 1
ip hold-time eigrp 100 3
carrier-delay msec 0
© 2005 Cisco Systems, Inc. All rights reserved.
77
EIGRP Query Process
Queries Propagate the Event
• EIGRP is an advanced distant vector;
it relies on its neighbor to provide
routing information
Reply
Query
Reply
Query
Access
• If a route is lost and no feasible
successor is available, EIGRP
actively queries its neighbors for
the lost route(s)
Reply
Query
Reply
Query
Si
Distribution
• The router will have to receive replies
back from ALL queried neighbors
before the router calculates
successor information
Reply
Query
Reply
Query
Core
• If any neighbor fails
to reply, the queried route
is stuck in active and the
router resets the neighbor
that fails to reply
• The fewer routers and routes
queried, the faster EIGRP converges;
solution is to limit query range
© 2005 Cisco Systems, Inc. All rights reserved.
Si
Si
Query
Si
Reply
Query
Si
Si
Reply
Query
Si
Distribution
Access
Reply
78
EIGRP Query Process
With Summarization
No Queries
to Rest of Network
from Core
• When we summarize from
distribution to core for the
subnets in the access we can
limit the upstream query/
reply process
• In a large network this could be
significant because queries will
now stop at the core; no
additional distribution blocks
will be involved in the
convergence event
Reply∞
Si
Reply∞
Si
Summary
Route
• The access layer is still queried
Summary
Route
Query
Si
Query
Reply
Si
interface gigabitethernet 3/1
ip address 10.120.10.1 255.255.255.252
ip summary-address eigrp 1 10.130.0.0 255.255.0.0
Query
Reply
© 2005 Cisco Systems, Inc. All rights reserved.
Reply
79
EIGRP Stubs
Distribution
• A stub router signals (through the
hello protocol) that it is a stub and
should not transit traffic
• Queries that would have been
generated towards the stub routers
are marked as if a “No path this
direction” reply had been received
Si
Si
D1
D2
“I’m Not Going
to Send You Any
Queries Since
You Said That!”
• D1 will know that stubs cannot be
transit paths, so they will not have
any path to 10.130.1.0/24
• D1 simply will not query the stubs,
reducing the total number of
queries in this example to 1
“Hello, I’m
a Stub…”
• These stubs will not pass D1’s
advertisement of 10.130.1.0/24 to D2
• D2 will only have one path to
10.130.1.0/24
© 2005 Cisco Systems, Inc. All rights reserved.
10.130.1.0/24
Access
80
EIGRP Query Process
With Summarization and Stub Routers
No Queries
to Rest of Network
from Core
• When we summarize from
distribution to core for the subnets
in the access we can limit the
upstream query/reply process
• In a large network this could be
significant because queries will
now stop at the core; no additional
distribution blocks will be involved
in the convergence event
• When the access switches are
EIGRP stub’s we can further
reduce the query diameter
• Non-stub routers do not query stub
routers—so no queries will be sent
to the access nodes
• No secondary queries—and only
three nodes involved in
convergence event
© 2005 Cisco Systems, Inc. All rights reserved.
Reply∞
Si
Reply∞
Si
Summary
Route
Summary
Route
Reply
Query
Si
Si
Stub
Stub
81
EIGRP Route Filtering in the Campus
Control Route Advertisements
• Bandwidth is not a constraining
factor in the campus but it is still
advisable to control the number
of routing updates advertised
Si
Si
• Remove/filter routes from the
core to the access and inject a
default route with distribute-lists
• Smaller routing table in access
is simpler to troubleshoot
Si
Si
• Deterministic topology
router eigrp 100
network 10.0.0.0
distribute-list Default out <mod/port>
ip access-list standard Default
permit 0.0.0.0
© 2005 Cisco Systems, Inc. All rights reserved.
82
EIGRP Routed Access Campus Design
Summary
• Detect the event:
Set hello-interval = 1 second and
hold-time = 3 seconds to detect soft
neighbor failures
Si
Si
Set carrier-delay = 0
Summary
Route
• Propagate the event:
Configure all access layer switches
as stub routers to limit queries
from the distribution layer
Summarize the access routes from
the distribution to the core to limit
queries across the campus
• Process the event:
Si
Si
Stub
Summarize and filter routes to
minimize calculating new
successors for the RIB and FIB
© 2005 Cisco Systems, Inc. All rights reserved.
83
Agenda
• Campus High Availability
Design Principles
• Foundation Services
• Multi-Layer Design
Flexibility
• Routed Access Design
EIGRP Design Details
Convergence
Mobility
Si
Availability
Si
Security
OSPF Design Details
• Summary
© 2005 Cisco Systems, Inc. All rights reserved.
Architectural Foundation
Hierarchical Campus Design
84
Open Shortest Path First (OSPF) Overview
• OSPFv2 established in 1991 with RFC 1247
• Goal—a link-state protocol more efficient and
scaleable than RIP
• Dijkstra Shortest Path First (SPF) algorithm
• Metric—path cost
• Fast convergence
• Support for CIDR, VLSM, authentication, multipath
and IP unnumbered
• Low steady state bandwidth requirement
• OSPFv3 for IPv6 support
© 2005 Cisco Systems, Inc. All rights reserved.
85
Hierarchical Campus Design
OSPF Area’s with Router Types
Internal’s
Internal’s
Access
Distribution
Core
Area 10
ABR’s
Si
Area 0
Si
Area 20
ABR’s
Si
ABR’s
ABR’s
Si
Si
Si
Si
Si
Area 0
Backbone
Backbone
Si
Si
Distribution
Area 30
ASBR’s
ABR’s
Si
Si
Si
Si
Area 300
Area 100
Access
Area 200
WAN
© 2005 Cisco Systems, Inc. All rights reserved.
Data Center
Internet
BGP
86
OSPF Design Rules for HA Campus
Where Are the Areas?
Area 100
Area 110
Area 120
Si
Si
Si
• Area size/border is bounded by
the same concerns in the
campus as the WAN
• In campus, the lower number of
nodes and stability of local
links could allow you to build
larger areas however…
Si
Si
Si
Si
Si
• Keep area 0 for core
infrastructure; do not extend to
the access routers
Si
Si
WAN
© 2005 Cisco Systems, Inc. All rights reserved.
Si
Area 0
• Area design also based on
address summarization
• Area boundaries should define
buffers between fault domains
Si
Si
Si
Data Center
Internet
87
Regular Area
ABRs Forward All LSAs from Backbone
External Routes/LSA Present in Area 120
Si
Si
Backbone Area 0
An ABR Forwards the
Following into an Area
Summary LSAs (Type 3)
ASBR Summary (Type 4)
Specific Externals (Type 5)
Area Border Router
Si
Area 120
Si
Distribution Config
router ospf 100
summary-address 10.120.0.0 255.255.0.0
network 10.120.0.0 0.0.255.255 area 120
network 10.122.0.0 0.0.255.255 area 0
Access Config:
router ospf 100
network 10.120.0.0 0.0.255.255 area 120
© 2005 Cisco Systems, Inc. All rights reserved.
88
Stub Area
Consolidates Specific External Links—Default 0.0.0.0
Eliminates External Routes/LSA Present in
Area (Type 5)
Si
Backbone Area 0
Si
Stub Area ABR Forwards
Summary LSAs
Summary 0.0.0.0 Default
Area Border Router
Si
Area 120
Si
Distribution
Config
router ospf 100
area 120 stub
summary-address 10.120.0.0 255.255.0.0
network 10.120.0.0 0.0.255.255 area 120
network 10.122.0.0 0.0.255.255 area 0
Access Config:
router ospf 100
area 120 stub
network 10.120.0.0 0.0.255.255 area 120
© 2005 Cisco Systems, Inc. All rights reserved.
89
Totally Stubby Area
Use This for Stable—Scalable Internetworks
Minimize the Number of LSA’s and the Need
for Any External Area SPF Calculations
Si
Backbone Area 0
Si
A Totally Stubby Area
ABR Forwards
Summary Default
Area Border Router
Si
Area 120
Si
Distribution
Config
router ospf 100
area 120 stub no-summary
summary-address 10.120.0.0 255.255.0.0
network 10.120.0.0 0.0.255.255 area 120
network 10.122.0.0 0.0.255.255 area 0
Access Config:
router ospf 100
area 120 stub no-summary
network 10.120.0.0 0.0.255.255 area 120
© 2005 Cisco Systems, Inc. All rights reserved.
90
Summarization Distribution to Core
Reduce SPF and LSA Load in Area 0
Minimize the Number of LSA’s and the Need for
Any SPF Recalculations at the Core
Si
Si
Backbone Area 0
ABR’s Forward
Summary 10.120.0.0/16
Area Border Router
Si
Area 120
Si
Distribution Config
router ospf 100
area 120 stub no-summary
summary-address 10.120.0.0 255.255.0.0
network 10.120.0.0 0.0.255.255 area 120
network 10.122.0.0 0.0.255.255 area 0
Access Config:
router ospf 100
area 120 stub no-summary
network 10.120.0.0 0.0.255.255 area 120
© 2005 Cisco Systems, Inc. All rights reserved.
91
OSPF Default Route to Totally Stubby Area
• Totally stubby area’s are used
to isolate the access layer
switches from route
calculations due to events in
other areas
Si
• This means that the ABR (the
distribution switch) will send a
default route to the access
layer switch when the neighbor
relationship is established
Si
Si
Si
• The default route is sent
regardless of the distribution
switches ability to forward
traffic on to the core (area 0)
• Traffic could be black holed
until connectivity to the core is
established
A
B
Note: Solution to this anomaly is being investigated.
© 2005 Cisco Systems, Inc. All rights reserved.
92
OSPF Timer Tuning
High-Speed Campus Convergence
• OSPF by design has a number of
throttling mechanisms to prevent
the network from thrashing
during periods of instability
Si
Reduce Hello
Interval
Si
• Campus environments are
candidates to utilize OSPF timer
enhancements
Sub-second hellos
Generic IP (interface) dampening
mechanism
Back-off algorithm for LSA generation
Si
Si
Reduce
LSA and SPF
Interval
Exponential SPF backoff
Configurable packet pacing
© 2005 Cisco Systems, Inc. All rights reserved.
93
Subsecond Hello’s
Neighbor Loss Detection—Physical Link Up
• OSPF hello/dead timers detect
neighbor loss in the absence of
physical link loss
• Useful in environments where an
L2 device separates L3 devices
(Layer 2 core designs)
• Aggressive timers are needed to
quickly detect neighbor failure
• Interface dampening is
recommended if sub-second hello
timers are implemented
Si
Si
OSPF
Processing
Failure
(Link Up)
Si
Si
Access Config:
interface GigabitEthernet1/1
dampening
ip ospf dead-interval minimal hello-multiplier 4
router ospf 100
area 120 stub no-summary
timers throttle spf 10 100 5000
timers throttle lsa all 10 100 5000
timers lsa arrival 80
© 2005 Cisco Systems, Inc. All rights reserved.
A
B
94
OSPF LSA Throttling
• By default, there is a 500ms delay before
generating router and network LSA’s; the wait is
used to collect changes during a convergence
event and minimize the number of LSA’s sent
Si
Si
• Propagation of a new instance of the LSA is
limited at the originator
timers throttle lsa all <start-interval>
<hold-interval> <max-interval>
• Acceptance of a new LSAs is limited by the
receiver
timers lsa arrival <milliseconds>
Si
Si
Access Config:
interface GigabitEthernet1/1
ip ospf dead-interval min hello-multi 4
router ospf 100
area 120 stub no-summary
timers throttle spf 10 100 5000
timers throttle lsa all 10 100 5000
timers lsa arrival 80
© 2005 Cisco Systems, Inc. All rights reserved.
A
B
95
OSPF SPF Throttling
• OSPF has an SPF throttling timer
designed to dampen route
recalculation (preserving CPU
resources) when a link bounces
Si
Si
• 12.2S OSPF enhancements let us tune
this timer to milliseconds; prior to
12.2S one second was the minimum
• After a failure, the router waits for
the SPF timer to expire before
recalculating a new route; SPF timer
was one second
Si
Si
Access Config:
interface GigabitEthernet1/1
ip ospf dead-interval min hello-multi 4
router ospf 100
area 120 stub no-summary
timers throttle spf 10 100 5000
timers throttle lsa all 10 100 5000
timers lsa arrival 80
© 2005 Cisco Systems, Inc. All rights reserved.
A
B
96
OSPF Routed Access Campus Design
Overview—Fast Convergence
• Detect the event:
Decrease the hello-interval and deadinterval to detect soft neighbor failures
Si
Si
Enable interface dampening
Backbone
Area
0
Set carrier-delay = 0
• Propagate the event:
Summarize routes between areas
to limit LSA propagation across
the campus
Tune LSA timers to minimize LSA
propagation delay
• Process the event:
Si
Si
Stub
Area
120
Tune SPF throttles to decrease
calculation delays
© 2005 Cisco Systems, Inc. All rights reserved.
97
OSPF Routed Access Campus Design
Overview—Area Design
• Use totally stubby areas to minimize
routes in Access switches
• Summarize area routes to backbone
Area 0
• These recommendations will reduce
number of LSAs and SPF
recalculations throughout the
network and provide a more robust
and scalable network infrastructure
router ospf 100
area 120 stub no-summary
summary-address 10.120.0.0 255.255.0.0
network 10.120.0.0 0.0.255.255 area 120
network 10.122.0.0 0.0.255.255 area 0
Si
Si
Area Routes
Summarized
Si
Si
Configured as
Totally Stubby
Area
router ospf 100
area 120 stub no-summary
network 10.120.0.0 0.0.255.255 area 120
© 2005 Cisco Systems, Inc. All rights reserved.
98
OSPF Routed Access Campus Design
Overview—Timer Tuning
• In a hierarchical design, the key
tuning parameters are SPF throttle
and LSA throttle
Si
Reduce Hello
Interval
Si
• Need to understand other LSA tuning
in the non-optimal design
• Hello and dead timers are secondary
failure detection mechanism
router ospf 100
area 120 stub no-summary
area 120 range 10.120.0.0 255.255.0.0
timers throttle spf 10 100 5000
timers throttle lsa all 10 100 5000
timers lsa arrival 80
network 10.120.0.0 0.0.255.255 area 120
network 10.122.0.0 0.0.255.255 area 0
Si
Si
Reduce SPF and
LSA Interval
interface GigabitEthernet5/2
ip address 10.120.100.1 255.255.255.254
dampening
ip ospf dead-interval minimal hello-multiplier 4
© 2005 Cisco Systems, Inc. All rights reserved.
99
Agenda
• Campus High Availability
Design Principles
• Foundation Services
Flexibility
Convergence
Mobility
• Multi-Layer Design
Si
• Routed Access Design
• Summary
© 2005 Cisco Systems, Inc. All rights reserved.
Availability
Si
Security
Architectural Foundation
Hierarchical Campus Design
100
Campus High Availability
Non-Stop Application Delivery
Access
Si
Distribution
Si
Core
Distribution
Si
Si
Si
Si
Si
Si
Si
Si
Si
Si
Si
Si
Access
WAN
Data Center
Internet
Hierarchical, systematic approach
• System level resiliency for switches and routers
• Network resiliency with redundant paths
• Supports integrated services and applications
• Embedded management
© 2005 Cisco Systems, Inc. All rights reserved.
101
Multi-Layer Campus Design
Reference Design—No VLANs Span Access Layer
• Tune CEF load balancing
• Match CatOS/IOS
Etherchannel settings and
tune load balancing
• Summarize routes
towards core
Si
Si
Core
• Limit redundant IGP peering
• STP Root and HSRP primary
tuning or GLBP to load
balance on uplinks
Layer 3
Si
• Set trunk mode
on/nonegotiate
P-t-P Link
Distribution
Si
• Disable Etherchannel
unless needed
• Set Port Host
on access layer ports:
Disable Trunking
Disable Etherchannel
Enable PortFast
• RootGuard or BPDU-Guard
Access
VLAN 20 Data
10.1.20.0/24
VLAN 120 Voice
10.1.120.0/24
VLAN 40 Data
10.1.40.0/24
VLAN 140 Voice
10.1.140.0/24
• Use security features
© 2005 Cisco Systems, Inc. All rights reserved.
102
Multi-Layer Campus Design
Some VLANs Span Access Layer
• Tune CEF load balancing
• Match CatOS/IOS Etherchannel
settings and tune load balancing
• Summarize routes towards core
• Limit redundant IGP peering
Si
• STP Root and HSRP primary or
GLBP and STP port cost tuning to
load balance on uplinks
• Set trunk mode on/nonegotiate
• Disable Etherchannel unless
needed
• RootGuard on downlinks
Si
Layer 2
Si
Trunk
Core
Distribution
Si
• LoopGuard on uplinks
• Set Port Host
on access Layer ports:
Disable trunking
Disable Etherchannel
Enable PortFast
• RootGuard or
BPDU-Guard
• Use security features
Access
VLAN 20 Data
VLAN 40 Data
10.1.20.0/24
10.1.40.0/24
VLAN 120 Voice
VLAN 140 Voice
10.1.120.0/24
10.1.140.0/24
VLAN 250 WLAN
10.1.250.0/24
© 2005 Cisco Systems, Inc. All rights reserved.
103
Routed Access Campus Design
No VLANs Span Access Layer
• Use EIGPR or OSPF
• Use Stub routers or Stub Areas
• With OSPF tune LSA and SPF
timers
• Summarize routes towards core
Si
Si
• Filter routes towards the
access
• Tune CEF load balancing
• Disable Etherchannel
unless needed
• Set Port Host
on access layer ports:
Core
Distribution
Si
Layer 3
Si
P-t-P Link
Disable Trunking
Disable Etherchannel
Enable PortFast
• RootGuard or BPDU-Guard
• Use security features
© 2005 Cisco Systems, Inc. All rights reserved.
Access
VLAN 20 Data
10.1.20.0/24
VLAN 120 Voice
10.1.120.0/24
VLAN 40 Data
10.1.40.0/24
VLAN 140 Voice
10.1.140.0/24
104
Cisco Campus Architecture
Multiple Design Options supporting Integrated Services
Enterprise Campus
The Right Design for Each Customer
• High Availability
• IP Communications
• WLAN Integration
High Availability
• Integrated Security
Cisco Campus Architecture
Future
Campus
Design
Options
Multi-Layer
Campus
Design
Routed
Campus
Design
• IPv6
• Virtualization
• Future Services
Intelligent Switching
(Hybrid of L2 + L3 features)
© 2005 Cisco Systems, Inc. All rights reserved.
105
RST-2031
11207_05_2005_c1
© 2005 Cisco Systems, Inc. All rights reserved.
106