Transcript Document

Network
Troubleshooting
Accessing the WAN – Chapter 8
Version 4.0
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Public
1
Objectives

Establish a network baseline

Describe troubleshooting methodologies and
troubleshooting tools

Describe the common issues that occur during WAN
implementation

Troubleshoot enterprise network implementation
issues
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Public
2
Establish a Network Baseline
 Explain the importance of network documentation
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Public
3
Establish a Network Baseline
 When you document your network, you may have to
gathr information directly from routers and switches.
 Commands that are useful to the network
documentation process include:
The ping command is used to test connectivity with
neighboring devices. Pinging to other PCs in the network
also initiates the MAC address auto-discovery process.
The telnet command is used to log in remotely to a
device for accessing configuration information.
The show ip interface brief is used to display the up or
down status and IP address of all interfaces.
The show ip route command is used to display the
routing table in a router to learn the directly connected
neighbors, more remote devices (through learned
routes), and the routing protocols.
The show cdp neighbor detail command is used to
obtain detailed information about directly connected
Cisco neighbor devices.
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Public
4
Establish a Network Baseline
 Explain the purpose for measuring normal network
performance when creating a baseline
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Public
5
Establish a Network Baseline
 Describe the steps for establishing a network baseline
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Public
6
Describe Troubleshooting Methodologies
and Troubleshooting Tools
 When you document your network, you may have to
gather information directly from routers and switches.
 Commands that are useful to the network
documentation process include:
The ping command is used to test connectivity with
neighboring devices. Pinging to other PCs in the network
also initiates the MAC address auto-discovery process.
The telnet command is used to log in remotely to a
device for accessing configuration information.
The show ip interface brief is used to display the up or
down status and IP address of all interfaces.
The show ip route command is used to display the
routing table in a router to learn the directly connected
neighbors, more remote devices (through learned
routes), and the routing protocols.
The show cdp neighbor detail command is used to
obtain detailed information about directly connected
Cisco neighbor devices.
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Public
7
Describe Troubleshooting Methodologies
and Troubleshooting Tools
 There are three main methods for troubleshooting:
 Bottom-Up Troubleshooting Method
In bottom-up troubleshooting you start with the physical
components of the network and move up through the layers.
Bottom-up troubleshooting is a good approach to use when the
problem is suspected to be a physical one.
 Top-Down Troubleshooting Method
In top-down troubleshooting your start with the end-user
applications and move down the layers of the OSI model.
Use this approach for simpler problems or when you think the
problem is with a piece of software.
 Divide-and-Conquer Troubleshooting Method
In divide-and-conquer troubleshooting you start by collecting
user experience of the problem, document the symptoms
and then, using that information, make an informed guess as
to which OSI layer to start your investigation.
For example, if users can't access the web server and you can
ping the server, then you know that the problem is above Layer 3.
If you can't ping the server, then you know the problem is likely at
a lower OSI layer.
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Public
8
Describe Troubleshooting Methodologies
and Troubleshooting Tools
 To quickly resolve network problems,
take the time to select the most effective
troubleshooting method.
Use the process shown in the figure to
help you select the most efficient
troubleshooting method.
 For example: Two IP routers are not
exchanging routing information. The last
time this type of problem occurred it was
a protocol issue. So you choose the
divide-and-conquer troubleshooting
method.
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Public
9
Gathering Symptoms
 Step 1. Analyze existing symptoms
Analyze symptoms gathered from the trouble ticket or
users to form a definition of the problem.
 Step 2. Determine ownership
If problem is within your system, move onto next stage.
If the problem is outside the boundary of your control, for
example, lost Internet connectivity you need to contact
an administrator for the external system.
 Step 3. Narrow the scope
Determine if the problem is at the core, distribution, or
access layer of the network.
 Step 4. Gather symptoms from suspect devices
Use knowledge and experience to determine if the
problem is a hardware or software problem.
 Step 5. Document symptoms
Sometimes the problem can be solved using the
documented symptoms. If not, begin the isolating phase
of the general troubleshooting process.
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Public
10
Gathering Symptoms
 Use the Cisco IOS commands to gather
symptoms about the network.
Although the debug command is an
important tool for gathering symptoms it
generates a large amount of console
message traffic and the performance of a
network device can be noticeably affected.
Make sure you warn network users that a
troubleshooting effort is underway and that
network performance may be affected.
Remember to disable debugging when you
are done.
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Public
11
Describe Troubleshooting Methodologies
and Troubleshooting Tools
 NMS Tools
Network management system (NMS) tools
include device-level monitoring, configuration,
and fault management tools.
Network monitoring software graphically
displays a physical view of network devices,
allowing network managers to monitor remote
devices without physically checking them.
Examples are CiscoView, HP Openview, Solar
Winds, and What's Up Gold.
 Knowledge Bases
On-line network device vendor knowledge
bases have become indispensable sources of
information.
When vendor-based knowledge bases are
combined with Internet search engines like
Google, a network administrator has access to
a vast pool of experience-based information.
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Public
12
Software Troubleshooting Tools
 Baselining Tools
For example they can help you draw network
diagrams, help you to keep network software and
hardware documentation up-to-date and help you to
cost-effectively measure baseline network bandwidth
use.
Many tools for automating the network
documentation and baselining process are available.
The figure shows a screen chapter of the SolarWinds
LAN surveyor and CyberGauge software.
 Protocol Analyzers
A protocol analyzer decodes the various protocol
layers in a recorded frame and presents this
information in a relatively easy to use format.
The figure shows a screen capture of the Wireshark
protocol analyzer.
Most protocol analyzers can filter traffic that meets
certain criteria so that, for example, all traffic to and
from a particular device can be captured.
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Public
13
Hardware Troubleshooting Tools
 Network Analysis Module
A network analysis module (NAM) can be
installed in Cisco Catalyst 6500 series switches
and Cisco 7600 series routers to provide a
graphical representation of traffic.
 Digital Multimeters
Digital multimeters (DMMs) are test instruments
that are used to directly measure electrical values
of voltage, current, and resistance.
 Cable Testers
Cabling testers can be used to detect broken
wires, crossed-over wiring, shorted connections,
and improperly paired connections.
These devices can be inexpensive continuity
testers, moderately priced data cabling testers, or
expensive time-domain reflectometers (TDRs).
TDRs are used to test the distance to a break in a
cable.
TDRs used to test fiber optic cables are known as
optical time-domain reflectometers (OTDRs).
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Public
14
Hardware Troubleshooting Tools
 Cable Analyzers
Cable analyzers are multifunctional handheld devices that
are used to test and certify copper and fiber cables for
different services and standards.
The more sophisticated tools include advanced
troubleshooting diagnostics that measure distance to
performance defect (NEXT, RL), identify corrective
actions, and graphically display crosstalk and impedance
behavior.
 Portable Network Analyzers
Portable devices that are used for troubleshooting
switched networks and VLANs.
By plugging the network analyzer in anywhere on the
network, a network engineer can see the switch port to
which the device is connected and the average and peak
utilization.
The analyzer can also be used to discover VLAN
configuration, identify top network talkers, analyze
network traffic, and view interface details.
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Public
15
Describe the Common Issues that Occur
During WAN Implementation
 WAN technologies function at the lower three
layers of the OSI reference model.
 A communications provider normally owns the
data links that make up a WAN.
The links are made available to subscribers for a
fee and are used to interconnect LANs or connect
to remote networks.
WAN data transfer speed (bandwidth) is
considerably slower than the common LAN
bandwidth.
The charges for link provision are the major cost
element, therefore the WAN implementation must
aim to provide maximum bandwidth at acceptable
cost.
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Public
16
Describe the Common Issues that Occur
During WAN Implementation
 WAN connectivity is important to business and expensive,
these are the steps for designing or modifying a WAN:
 Step 1. Locate LANs - Establish the source and destination
endpoints that will connect through the WAN.
 Step 2. Analyze traffic - Know what data traffic must be
carried, its origin, and its destination.
 Step 3. Plan the topology - A high requirement for availability
requires extra links that provide alternative data paths for
redundancy and load balancing.
 Step 4. Estimate the required bandwidth - Traffic on the links
may have varying requirements for latency and jitter.
 Step 5. Choose the WAN technology - Suitable link
technologies must be selected.
 Step 6. Evaluate costs - When all the requirements are
established, installation and operational costs for the WAN
can be determined and compared with the business need
driving the WAN implementation.
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Public
17
WAN Traffic Considerations
 The table in the figure shows
the wide variety of traffic
types and their varying
requirements of bandwidth,
latency, and jitter that WAN
links are required to carry.
To determine traffic flow
conditions and timing of a WAN
link, you need to analyze the
traffic characteristics specific to
each LAN that is connected to
the WAN.
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Public
18
Describe the Common Issues that Occur
During WAN Implementation
 Describe the considerations for designing a WAN
topology
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Public
19
Describe the Common Issues that Occur
During WAN Implementation
 Describe common WAN implementation issues
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Public
20
Describe the Common Issues that Occur
During WAN Implementation
 Describe the recommended steps for troubleshooting a
WAN
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Public
21
Troubleshoot Enterprise Network
Implementation Issues
 Explain how network diagrams are used for
troubleshooting
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Public
22
To isolate problems at the physical layers
 Check for bad cables or connections
Verify that the cable is properly connected and is in good
condition. Your cable tester might reveal an open wire.
 Check that the correct cabling standard is adhered to
throughout the network
Verify that the proper cable is being used. For example, in
the figure, the Fluke meter detected that a cable was good
for Fast Ethernet, it is not qualified to support 1000BASE-T.
 Check that devices are cabled correctly
Check that cables are connected to their correct ports.
This is where having a neat and organized wiring closet saves
you a great deal of time.
 Verify proper interface configurations
Check that all switch ports are set in the correct VLAN and,
speed, and duplex settings are correctly configured.
 Check operational statistics and data error rates
Use Cisco show commands to check for statistics such as
collisions and input and output errors.
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Public
23
Symptoms of Data Link Layer Problems
 Common symptoms at the data link layer include:
 No functionality or connectivity at network layer or above
Some Layer 2 problems can stop the frames across a link.
 Network is operating below baseline performance levels
There are two types of suboptimal Layer 2 operation:
Frames take an illogical path to their destination but do arrive.
An example of a problem which could cause frames to take a
suboptimal path is a poorly designed Layer 2 spanning-tree.
Some frames are dropped. An extended or continuous ping also
reveals if frames are being dropped.
 Excessive broadcasts
Excessive broadcasts result from one of the following:
Poorly programmed or configured applications
Large Layer 2 broadcast domains
Underlying network problems, such as STP loops.
 Console messages
In some instances, a router recognizes a Layer 2 problem
has occurred and sends alert messages to the console.
The most common console is line protocol down message.
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Public
24
Troubleshooting Layer 3 Problems
 In most networks, static routes are used in combination with
dynamic routing protocols.
Improper configuration of static routes can lead to less than optimal
routing and, in some cases the network to become unreachable.
 Here are some possible problems involving routing protocols:
 General network issues
Often a change in the topology, such as a down link, may have
affects on other areas that might not be obvious at the time.
 Connectivity issues
Check for any equipment problems, cabling, and ISP problems.
 Neighbor issues
Check if there are any problems with the routers forming neighbor.
 Topology database
Check the topology table, for any missing or unexpected entries.
 Routing table
Check the routing table for anything missing or unexpected routes.
Use debug commands to view routing updates and maintenance.
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Public
25
Transport Layer Troubleshooting: Access List Issues
1. Selection of traffic flow
ACL must be applied to the correct interface, and correct
traffic direction must be selected to function properly.
If the router is running both ACLs and NAT, the order in
which each of these technologies is applied is important:
Inbound traffic is processed by the inbound ACL before being
processed by outside-to-inside NAT.
Outbound traffic is processed by the outbound ACL after being
processed by inside-to-outside NAT.
2. Order of access control elements
The elements ACL should be from specific to general.
3. Implicit deny all
Forgetting about this implicit access control element may be
the cause of an ACL misconfiguration.
4. Addresses and wildcard masks
Complex wildcard masks provide significant improvements
in efficiency, but are more subject to configuration errors.
The address 10.0.32.0 and wildcard mask 0.0.32.15 to select
the first 15 host addresses in either the 10.0.0.0 or 10.0.32.0
network.
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Public
26
Transport Layer Troubleshooting: Access List Issues
5. Selection of transport layer protocol
When configuring ACLs, it is important that only the correct
transport layer protocols [TCP, UDP] be specified.
6. Source and destination ports
Address and port information for traffic generated by a replying
host is the mirror address and port from the source host.
7. Use of the established keyword
If the keyword is applied to an outbound ACL, unexpected
results may occur.
8. Uncommon protocols
Uncommon protocols that are gaining popularity are VPN and
encryption protocols.
 Troubleshooting Access Control Lists
A useful command for viewing ACL operation is the log
keyword on ACL entries.
This keyword instructs the router to place an entry in the system log
whenever that entry condition is matched.
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Public
27
Application Layer Overview
 The most widely known application layer protocols:
Telnet - Enables users to establish terminal session
connections with remote hosts.
HTTP - Supports the exchanging of text, graphic, sound,
video, and other multimedia files on the web.
FTP - Performs interactive file transfers between hosts.
TFTP - Performs basic interactive file transfers typically
between hosts and networking devices.
SMTP - Supports basic message delivery services.
POP - Connects to mail servers and downloads e-mail.
Simple Network Management Protocol (SNMP) Collects management information from network devices.
DNS - Maps IP addresses to the names assigned to
network devices.
Network File System (NFS) - Enables computers to
mount drives on remote hosts and operate them as if
they were local drives.
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Public
28
Symptoms of Application Layer Problems
 A problem at the application layer can result in
unreachable or unusable resources when the
physical, data link, network, and transport layers are
functional.
It is possible to have full network connectivity, but the
application simply cannot provide data.
 Another type of problem at the application layer
occurs when the physical, data link, network, and
transport layers are functional, but the data transfer
and requests for network services from a single
network service or application do not meet the normal
expectations of a user.
 A problem at the application layer may cause users to
complain that the network or the particular application
that they are working with is sluggish or slower than
usual when transferring data or requesting network
services.
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Public
29
Troubleshooting Application Layer Problems
 The steps for troubleshooting application layer problems are:
 Step 1. Ping the default gateway.
If successful, Layer 1 and Layer 2 services are functioning properly.
 Step 2. Verify end-to-end connectivity.
If Layers 1-3 functioning properly, the issue exist at a higher layer.
 Step 3. Verify access list and NAT operation.
If the ACLs and NAT are functioning as expected, the problem must
lie in a higher layer.
 Step 4. Troubleshoot upper layer protocol connectivity.
Upper layer protocol, such as FTP, HTTP, or Telnet ride on top of the
basic IP transport but are subject to protocol-specific problems relating
to packet filters and firewalls.
Troubleshooting an upper layer protocol connectivity problem requires
understanding the process of the protocol.
This information is usually found in the latest RFC for the protocol or
on the developer web page.
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Public
30
Correcting Application Layer Problems
 The steps for correcting application layer problems are:
 Step 1: Make a backup.
Ensure that a valid configuration has been saved.
 Step 2: Make an initial configuration change.
Make only one change at a time.
 Step 3: Evaluate each change and its results.
If the results of any problem-solving steps are unsuccessful,
immediately undo the changes.
 Step 4: Determine if the change solves the problem.
Verify that the change actually resolves the problem without
introducing any new problems.
If the problem is not solved, undo all the changes.
 Step 5: Stop when the problem is solved.
 Step 6: If necessary, get assistance from outside resources.
This may be a co-worker, a consultant, or TAC.
 Step 7: Document.
Once the problem is resolved, document the solution.
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Public
31
Summary

Network Baseline
How a network is expected to perform under normal conditions

Network documentation should include:
– Network configuration table
– End-system configuration table
– Network topology diagram

Planning for the 1st baseline
– Determine what type of data to collect
– Identify devices and ports of interest
– Determine baseline duration
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Public
32
Summary
 3 stages of the troubleshooting process
–Gather symptoms
–Isolate problem
–Correct problem
 3 main methods for troubleshooting a network
–Bottom up
–Top down
–Divide & conquer
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Public
33
Summary
 Software troubleshooting tools
–Cisco view
–Solar winds
–HP Open view
 Hardware troubleshooting tools
–Network analysis mode
–Digital multi-meters
–Cable testers
–Network analyzer
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Public
34
Summary
 Common WAN implementation issues include
–QoS
–Reliability
–Security
–Latency
–Confidentiality
–Public or Private
 Using a layered approach to troubleshooting aids in
isolating and solving the problem
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Public
35
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Public
36