Network Operations
Download
Report
Transcript Network Operations
Network Management
and
Network Operations
I have a network, now what?
Abha Ahuja
<[email protected]>
some slides based on the netmgt talk in T4-98 by Scott Bradner
[email protected]
Network Management and Network
Operations
1
Outline
What is network management?
Fault Management
• Fault detection and tracking
• Basic Network Operations
• What are typical network problems?
Other parts of network management
[email protected]
Network Management and Network
Operations
2
Outline (con)
Network Management Tools
• what do I need?
• what is available?
• Pros and Cons of various tools
A day in the life of Merit’s NOC
[email protected]
Network Management and Network
Operations
3
Network Management - What is it?
Making sure the network is up, running and
performing well
Parts of Network Management
•
•
•
•
•
fault management
performance management
security management
trouble tracking
statistics and accounting
[email protected]
Network Management and Network
Operations
4
Who is Merit?
Educational and Research Institution based in
Michigan, U.S.
Affiliated with University of Michigan
Run State-wide ISP - MichNet
• 300+ routers
• 24/7 Network Operations Center
[email protected]
Network Management and Network
Operations
5
Fault Management
one of the most important parts of network
management
detect network problems
• transient/persistent
• failure/overload
– examples: router down, serial link down
detect server problems
isolating problems
[email protected]
Network Management and Network
Operations
6
Fault Management (con)
reporting mechanism
• link to help desk
• notify on-call personnel
setup & control alarm procedures
repair/recovery procedures
ticket system
[email protected]
Network Management and Network
Operations
7
Fault Management - Fault Detection
Who notices a problem with the network?
• Network Operations Center w/ 24x7 operations staff
– open trouble ticket to track problem
– preliminary troubleshooting
– escalate to engineer or call carrier
[email protected]
Network Management and Network
Operations
8
Fault Management Fault Detection (con)
How can you tell if there is a problem with the
network?
• Network Monitoring Tools
– common utilities
ping
traceroute
snmp
• Report state or unreachability
– detect node down
– routing problems
[email protected]
Network Management and Network
Operations
9
Fault Management Fault Detection (con)
• “Alert” shows up for NOC
– rover
– spectrum
– NOCol
– HP Openview
– other
• Other methods
– customer complaint via phone/email
– another ISP notices problem
[email protected]
Network Management and Network
Operations
10
Fault Detection Example Using Rover
Rover = network monitoring system
• http://www.merit.edu/internet.tools/rover/
Keep it Simple
add nodes and tests to hostfile
run Display to see status
NOC notices alert on board for failed node
• opens ticket
• investigates
[email protected]
Network Management and Network
Operations
11
Fault Management Ticket System (Why all the fuss?)
Very Important!
Need mechanism to track:
• failures
• current status of outage
• carrier ticket #s
[email protected]
Network Management and Network
Operations
12
Fault Management Ticket Systems (Why all the fuss?)
system provides for:
•
•
•
•
•
•
short term memory & communication
scheduling and work assignment
referrals and dispatching
oversight
statistical analysis
long term accountability
[email protected]
Network Management and Network
Operations
13
Fault Management Ticket Systems (Why all the fuss?)
Goal: make your NOC the communication and
coordination center!
Central repository for all information
• current status
• troubleshooting information
Engineers can coordinate their work through the
NOC
[email protected]
Network Management and Network
Operations
14
Fault Management - Ticket Usage
create a ticket on ALL calls
create a ticket on ALL problems
create a ticket for ALL scheduled events
copy of ticket mailed to reporter and mailing
list(s)
all milestones in resolution of problem create a
new ticket entry with reference to original
ticket stays "open" until problem resolved
according to problem reporter
[email protected]
Network Management and Network
Operations
15
Fault Management - Ticket Example
sample opening ticket
TT0000033975 has been OPENED.
Here is the trouble ticket contents:
Create-date
: 06/09/99 12:46:42
Ticket ID
: TT0000033975
Node +
: rs2.mae-west.rsng.net
Equipment Type
: host
NOC Customer
: RA
Trouble Reported
: Unreachable
Next Action
: Investigate
Next Action Date
: 06/09/99 12:46:42
Outage type
: unscheduled
Source of Report
: Noc/roverStatus
Assigned
Assigned-to
: Noc
Contact Name
: rsng
Group Member
:
Contact pager#/email address :
Contact Phone
: .
Carrier Ticket History
:
Carrier
:
Carrier Phone
:
Ticket information log
:
06/09/99 12:46:42
noc-op
[email protected] said ...
:
11 Wed12:23 rs2MW_O/C 198.32.136.2 PING
[email protected]
Network Management and Network
Operations
16
Fault Management - Ticket Example
sample progress ticket
TT0000033975 has been MODIFIED.
changed:
Here are the fields that have been
CopyOfTime
: 5
TTC Temp
: 0
Ticket information log : [email protected] said ...
While I was investigating this, Debbie from UUNet called (via Merit
main
number) to tell us they were seeing it down. She can be reached at
xxx-xxxx. The UUNet ticket is xxxxx..
[email protected]
Network Management and Network
Operations
17
Fault Management - Ticket Example
sample closing ticket
• includes previous ticket contents plus resolution
T0000033975 has been CLOSED.
Here is the trouble ticket contents:
01/15/99 12:50:06
noc-op
[email protected] said ...
Email response from Abha suggesting contacting peers directly -- see
internal log.
01/15/99 14:25:22
noc-op
[email protected] said ...
The alerts cleared shortly before 14:00. I called MCI/Worldcom for an
update, and found out their ticket was closed. According to them the
outage was due solely to a power problem.
Closing.
Last-modified-by
Modified-date
Submitter
[email protected]
: noc-op
: 01/15/99 14:25:22
: btracy
Network Management and Network
Operations
18
Fault Management - typical failures
Node unpingable
• no ip connectivity to router
• possible reasons:
– serial link down
call telco
– router down/hardware problem
call engineer
– routing problem
troubleshoot with traceroute
routeviews machine
[email protected]
Network Management and Network
Operations
19
Performance Management
evaluate the behavior of network elements
information used in planning
– interface stats
– throughput
– error rates
– software stats
– usage
– queues
– system load
– disk space
– percent availability
[email protected]
Network Management and Network
Operations
20
Security Management
tends to be host-based
protect your stats, data and NOC info
protect other services
security required to operate network and protect
managed objects
security services
• Kerberos
• PGP key server
• secure time
[email protected]
Network Management and Network
Operations
21
Security Management (con)
security tools
•
•
•
•
•
cops - host configuration checker (www.cert.org)
swatch - email reports of activity on machine
tcpwrappers
ssh/skey
tripwire
distribute security information
• bug reports
– CERT advisories
• bug fixes
• intruder alerts
[email protected]
Network Management and Network
Operations
22
Security Management (con)
reporting procedure for security events
• e.g. break-ins
• abuse email address for customers to report
complaints ([email protected])
control internal and external gateways
• control firewalls (external and internal)
security logs
privacy issues a conflict
[email protected]
Network Management and Network
Operations
23
Security Management
Network based security
• Types of attacks
– DOS - Denial of Service
ping floods
smurf
attacks that make your network unusable
– Spoofing
packets with “spoofed” source address
[email protected]
Network Management and Network
Operations
24
What types of problems?
Blocking and tracing denial of service attacks
Tracing incoming forged packets back to their
source
Blocking outgoing forged packets
Most other security problems are not specific to
backbone operators
Deal with complaints
[email protected]
Network Management and Network
Operations
25
smurf
attacker sends many ping request packets:
• from forged (victim) source address
• to broadcast address on “amplifier” network
many ping responses from systems on amplifier
network
attacker on dialup modem can saturate victim’s
T1 using a T3-connected amplifier
http://users.quadrunner.com/chuegen/smurf/
[email protected]
Network Management and Network
Operations
26
Protection against smurf
configure “no directed-broadcast” on all
interfaces
• so you can’t be used as an amplifier
trace forged packets back, hop by hop
block outgoing forged packets from your
customers
limit the bandwidth that can be used by ICMP
traffic
[email protected]
Network Management and Network
Operations
27
Smurf Attack
132.34.65.1
victim
src IP=132.34.65.1
253*5*100
dst IP= 215.23.16.255
5*100 byte packets
attacker
24.3.2.1
[email protected]
amplifier
215.23.16.0/24
Network Management and Network
Operations
28
SYN flooding
attacker sends many TCP SYN packet from
forged source address
victim sends SYN+ACK packets to invalid
address
• gets no response
• connection hangs in half open state
• wastes OS resources, possibly crashing system
[email protected]
Network Management and Network
Operations
29
Protection against SYN flooding
Make operating system more robust
• not a backbone problem, except on routers
Trace and block forged packets
Limit bandwidth that can be used by TCP SYN
traffic
[email protected]
Network Management and Network
Operations
30
Syn attack
230.55.65.1
src IP=230.55.65.1
dst IP=132.16.12.5
connection request packets
attacker
( syn packets)
24.13.51.2
[email protected]
Replies go to
spoofed IP
victim
132.16.12.5
Network Management and Network
Operations
31
Notice a pattern?
Forged packets
Need a way of preventing customers from
sending forged packets
Need a way of tracing where forged packets
really come from
[email protected]
Network Management and Network
Operations
32
Tracing forged packets
Start on router near victim
Find how packets get to that router
Repeat on next router
Continue until edge of your AS
Ask next AS to trace further
Need cooperation
IMPORTANT - Should have a 24hour security
contact!
[email protected]
Network Management and Network
Operations
33
Security Management
Protecting your network
• traffic shapers
– use CAR to limit ICMP traffic
• anti-spoofing filters
– RFC 2267 (Network Ingress Filtering)
– for singly-homed customers
IF packet's source address from within your network
THEN forward as appropriate
IF packet's source address is anything else
THEN deny packet
– Filter on the outbound
[email protected]
Network Management and Network
Operations
34
Preventing forged packets from
customers
packet filters!
you know what IP addresses are used (at east for
dialup and statically routed customers)
make a filter for each customer that denies other
source addresses
very recent cisco code has “ip verify sourceaddress”
[email protected]
Network Management and Network
Operations
35
Preventing forged packets from you to
outside world
you might know all the IP addresses that are
used in your AS
• if your connections to the outside world and your
transit arrangements are not too complicated
make a filter that denies other source addresses
apply that filter to all links from you to other
Ases
[email protected]
Network Management and Network
Operations
36
Configuration and Name Management
track network vitals
• ip addresses, interfaces, console phone numbers, etc
NOC needs valid contact info for nodes
network state information
• network topology
• operation status of network elements
– including resources
• network element configuration
[email protected]
Network Management and Network
Operations
37
Configuration and Name Management
control network elements
• start/stop
• modification of network attributes
• addition of new features
configuration modification
• allocation and addition of network resources
• reconfiguration if dictated by link outages
[email protected]
Network Management and Network
Operations
38
Configuration and Name Management
inventory management
• database of network elements
• history of changes & problems
directory maintenance
• all hosts & applications
• nameserver database
host and service naming coordination
• "Information is not information if you can't find it"
[email protected]
Network Management and Network
Operations
39
Config. Mgmt. - Network State Info.
e.g. SNMP driven display
wjh12
mghgw
generali
husc6
harvard
talcott
wjhgw1
harvisr
huelings
geo
pitirium
nnhvd
nngw
oitgw1
sphgw1
lmagw1
dfch
[email protected]
Network Management and Network
Operations
tch
tch
40
Network Management Tools
many use SNMP
ping
traceroute
References:
•
•
•
•
•
MON - http://www.kernel.org/software/mon/
NOCol - ftp://ftp.navya.com/pub/vikas/nocol.tar.gz
Sysmon - ftp://puck.nether.net/pub/jared
Rover - http://www.merit.edu/~rover
Concord - http://www.concord.com
[email protected]
Network Management and Network
Operations
41
What is SNMP? (the quick version...)
Simple Network Management Protocol
query - response system
• can obtain status from a device
• standard queries
• enterprise specific
uses database defined in MIB
• management information base
[email protected]
Network Management and Network
Operations
42
What do we use SNMP for?
query routers for:
•
•
•
•
in and out bytes per second
CPU load
uptime
BGP peer session status
query hosts for:
• network status
[email protected]
Network Management and Network
Operations
43
SNMP Network Management Tools
mrtg
(http//:www.ee.ethz.ch/~oetiker/webtools/mrtg
• why we like it
– simple to use and configure
– quickly determine spikes/drops in traffic
ping floods
• in/out bps
• uptime
• supplement to monitoring tools
http://israel.t1.isocws.isoc.org/mrtg/7200
[email protected]
Network Management and Network
Operations
44
MRTG
Traffic Analysis for Hssi1/0/0
System:
msu.mich.net in
Maintainer:
Interface:
Hssi1/0/ 0 (2)
IP:
hssi1-0-0.msu.mich.net (198.108.22.102)
Max Speed:
5630.6 kBytes/s (propPointToPointSerial)
[email protected]
Network Management and Network
Operations
45
Spectrum
commercial package
Used by various networks
configurable alarms
GUI interface - view network topology
auto-discovery
difficult to use
[email protected]
Network Management and Network
Operations
46
Netscarf/Scion
free
snmp collector and analyzer package
• collects snmp data
• display on web pages
http://www.merit.net/~netscarf
[email protected]
Network Management and Network
Operations
47
Other Network Tools
netflow
•
•
•
•
•
•
•
cflowd (http://www.caida.org/Tools/Cflowd)
collects flow information from cisco routers
AS to AS information
src and destination ip and port information
useful for accounting and statistics
how much of my traffic is port 80?
how much of my traffic goes to AS237?
[email protected]
Network Management and Network
Operations
48
Netflow examples
Top ten lists (or top five)
##### Top 5 AS's based on number of bytes #######
srcAS dstAS
pkts
bytes
6461 237
4473872
3808572766
237 237
22977795
3180337999
3549 237
6457673
2816009078
2548 237
5215912
2457515319
##### Top 5 Nets based on number of bytes ######
Net Matrix
---------number of net entries: 931777
SRCNET/MASK DSTNET/MASK
PKTS
165.123.0.0/16 35.8.0.0/13
745858
207.126.96.0/19 198.108.98.0/24
708205
206.183.224.0/19 198.108.16.0/22
740218
35.8.0.0/13 128.32.0.0/16
671980
##### Top 10 Ports #######
input
port
packets
bytes
119
10863322 2808194019
80
36073210
862839291
20
1079075 1100961902
7648
1146864
419882753
25
1532439
97294492
[email protected]
BYTES
1036296098
907577874
861538792
467274801
output
packets
bytes
5712783
427304556
17312202 1387817094
614910
62754268
1147081
414663212
2158042
722584770
Network Management and Network
Operations
49
More Tools!
http://www.caida.org/Tools/
• OC3Mon/Coral
http://www.merit.edu/~ipma
• RouteTracker
• IRRj
• ASExplorer
http://www.geektools.com/
http://www.merit.edu/ipma/tools/other.html
[email protected]
Network Management and Network
Operations
50
ASexplorer
[email protected]
Network Management and Network
Operations
51
Route Flap Stats
[email protected]
Network Management and Network
Operations
52
Looking Glass Tools
http://www.merit.edu/~ipma/tools/lookingglass.h
tml
route-views.oregon-ix.net>show ip bgp 35.0.0.0
BGP routing table entry for 35.0.0.0/8, version 56135569
Paths: (17 available, best #12)
11537 237
198.32.8.252 from 198.32.8.252
Origin incomplete, localpref 100, valid, external
Community: 11537:900 11537:950
2914 5696 237
129.250.0.3 (inaccessible) from 129.250.0.3
Origin IGP, metric 0, localpref 100, valid, external
Community: 2914:420
2914 5696 237
129.250.0.1 (inaccessible) from 129.250.0.1
Origin IGP, metric 0, localpref 100, valid, external
Community: 2914:420
3561 237 237 237
204.70.4.89 from 204.70.4.89
Origin IGP, localpref 100, valid, external
267 1225 237
204.42.253.253 from 204.42.253.253
Origin IGP, localpref 100, valid, external
Community: 267:1225 1225:237
[email protected]
Network Management and Network
Operations
53
More Looking Glass Tools
Traceroute servers
http://www.merit.edu/ipma/tools/trace.html
Query: trace
Addr: www.isoc.org
Translating "www.isoc.org"...domain server (206.205.242.132) [OK]
Type escape sequence to abort.
Tracing the route to info.isoc.org (198.6.250.9)
1
2
3
4
5
6
7
8
9
iad1-core2-fa5-0-0.atlas.digex.net (165.117.129.2) 0 msec 0 msec 4 msec
dca5-core2-s5-0-0.atlas.digex.net (165.117.53.41) 0 msec 4 msec 0 msec
dca5-core1-fa5-1-0.atlas.digex.net (165.117.56.117) 4 msec 0 msec 4 msec
Hssi3-1-0.BR1.DCA1.ALTER.NET (209.116.159.98) 0 msec 0 msec 4 msec
101.ATM2-0.XR1.DCA1.ALTER.NET (146.188.160.226) [AS 701] 4 msec 0 msec 4 msec
195.ATM7-0.XR1.TCO1.ALTER.NET (146.188.160.102) [AS 701] 4 msec 0 msec 0 msec
193.ATM8-0-0.GW1.TCO1.ALTER.NET (146.188.160.33) [AS 701] 4 msec 4 msec 4 msec
charlie.isoc.org (198.6.250.1) [AS 701] 8 msec 8 msec 8 msec
info.isoc.org (198.6.250.9) [AS 701] 8 msec * 12 msec
[email protected]
Network Management and Network
Operations
54
Accounting Management
what do you account for?
if you count packets sent
• it can inhibit anonymous ftp & web sites
• QoS differences in the future
want to charge "user" of service
• application dependent determination of "user"
if count hosts
• is a PC equal to a mainframe?
• cost?
[email protected]
Network Management and Network
Operations
55
Accounting, Cont.
usage-based billing may become common as
telcos take over Internet service providers
when "little provider" meets "big provider”, QoS
...
fred's
bits to go
MCI
UNET
NAP
PSI
SPRINT
[email protected]
Network Management and Network
Operations
56
getoctets
simple traffic stats collector
cron-driven shell procedure
• get-octets router1 router2 router3
• figures out interface list for each router
then gets
• ifInOctets, ifOutOctets, ifInUcastPkts, ifOutUcastPkts
• ifInNUcastPkts, ifOutNUcastPkts, system.sysUpTime
ftp://ndtl.harvard.edu/pub/SNMPoll/octets.tar
• needs cmu snmp package
[email protected]
Network Management and Network
Operations
57
getoctets, contd.
makes separate stats file for each interface
• example filename: 128.103.1.2.WJHgw1
example data
1997,06,23,160,09,1,00,02,37,EDT,1764089502,1045789221,99138769,92200835,10,628226,758006814
1997,06,23,160,09,1,00,22,37,EDT,1766362487,1047093977,99151676,92213338,10,628281,758126831
1997,06,23,160,09,1,00,42,36,EDT,1768439726,1048266407,99163118,92224546,10,628342,758246748
processing a bit hard
• must deal with counter wrap & router reboots
• sample period must be < 59 min for an Ethernet
link utilization calculation complex
• must include link encapsulation etc
[email protected]
Network Management and Network
Operations
58
getoctets, processing
UpDate routine
• bug in 32 bit versions of perl (gives bad results)
example output
week
ending
1997.06.01
[email protected]
millions of bits per second
peak in peak out 95% in 95% out
5.0976
0.9330
1.3389
0.4104
Network Management and Network
Operations
millions of octets
in
out
total
18782
13752
32534
59
Accounting, Cont.
usage-based billing may become common as
telcos take over Internet service providers
when "little provider" meets "big provider”, QoS
...
fred's
bits to go
MCI
UNET
NAP
PSI
SPRINT
[email protected]
Network Management and Network
Operations
60
Accounting, Cont.
could do settlements based in routing
information
• try to minimize size of routing tables
Telco model
• everyone shares in revenue
• call an 800 number from a pay phone
– 800 destinations pays pay phone owner
• receive a long distance call to your own switch
– you get free for local delivery
[email protected]
Network Management and Network
Operations
61
Importance of Network Statistics
Accounting
Troubleshooting
Long-term trend analysis
Capacity Planning
Two different types
• active measurement
• passive measurement
Management Tools have statistical functionality
[email protected]
Network Management and Network
Operations
62
Management for Real
A few basic tools
echo request
•
•
•
•
ping on IP
checks path & basic node function
can return round trip time
normally not higher node function
oolbeans% ping -s www.cisco.com
PING cio-sys.cisco.com: 56 data bytes
64 bytes from cio-sys.cisco.com (192.31.7.130): icmp_seq=0. time=69. ms
64 bytes from cio-sys.cisco.com (192.31.7.130): icmp_seq=1. time=68. ms
64 bytes from cio-sys.cisco.com (192.31.7.130): icmp_seq=2. time=68. ms
64 bytes from cio-sys.cisco.com (192.31.7.130): icmp_seq=3. time=70. ms
64 bytes from cio-sys.cisco.com (192.31.7.130): icmp_seq=4. time=69. ms
64 bytes from cio-sys.cisco.com (192.31.7.130): icmp_seq=5. time=68. ms
^C
----cio-sys.cisco.com PING Statistics---5 packets transmitted, 5 packets received, 0% packet loss
round-trip (ms) min/avg/max = 68/68/70
[email protected]
Network Management and Network
Operations
63
Management for Real, Cont.
traceroute - finds path to node with delays
• detect reachability
• detect routing problems
– example of routing loop (next slide)
[email protected]
Network Management and Network
Operations
64
dfalk@unagi [Thu 15:07] 5 /usr/home/jdfalk>traceroute -m 255 www.monkeys.com
traceroute to www.monkeys.com (207.212.142.41), 255 hops max, 40 byte packets
1 thermal-detonator.explosive.net (209.133.38.1) 3.428 ms 2.032 ms 2.915
ms
2 explosive-gate.bungi.com (207.126.96.81) 14.158 ms 6.082 ms 6.239 ms
3 above-gw1.above.net (207.126.96.249) 18.889 ms 23.423 ms 13.275 ms
4 core2-main.sjc.above.net (207.126.96.133) 20.749 ms 22.295 ms 26.260 ms
5 pbnap.ibm.net (198.32.128.49) 31.658 ms 21.513 ms 10.753 ms
6 sfra1sr1-4-0-0.ca.us.ibm.net (165.87.13.5) 22.046 ms 46.370 ms 11.730 ms
7 sfo-pacbell-pop-sc.ca.us.ibm.net (165.87.225.9) 14.978 ms 31.752 ms
15.835 ms
8 ded1-fa0-1-0.pbi.net (216.102.176.229) 16.619 ms 26.949 ms 14.992 ms
9 pbi.scrm01.foothill.net (206.13.15.82) 47.453 ms 41.492 ms 55.562 ms
10 inyo.E0.foothill.net (206.170.175.12) 25.009 ms 42.198 ms 46.245 ms
11 fhaub.foothill.net (207.212.142.2) 26.434 ms 26.344 ms 28.052 ms
12 aub2-aub.foothill.net (207.212.142.18) 124.096 ms 101.107 ms 116.097 ms
13 yellowstone.foothill.net (209.77.125.7) 60.986 ms 65.366 ms 62.531 ms
14 black.foothill.net (209.77.125.5) 54.999 ms 54.907 ms 75.083 ms
15 den-edge-03.inet.qwest.net (205.171.2.81) 60.018 ms 65.658 ms 70.363 ms
16 den-core-01.inet.qwest.net (205.171.16.101) 74.909 ms 65.983 ms 53.476
ms
17 kcm-core-01.inet.qwest.net (205.171.5.49) 122.825 ms 122.386 ms 109.227
ms
18 chi-core-03.inet.qwest.net (205.171.5.209) 105.897 ms 124.867 ms *
19 chi-brdr-01.inet.qwest.net (205.171.20.66) 157.154 ms 135.603 ms 112.038
ms
20 ameritech-nap.ibm.net (198.32.130.48) 97.206 ms 287.921 ms 118.020 ms
21 scha1br2-0-0-0.il.us.ibm.net (165.87.34.162) 127.120 ms 94.150 ms
108.502 ms
22 sfra1br2-at-2-0-1-4.ca.us.ibm.net (165.87.230.238) 121.666 ms 106.453 ms
137.678 ms
23 sfra1sr1-12-0-0.ca.us.ibm.net (165.87.13.9) 134.660 ms 121.347 ms
134.990 ms
24 sfo-pacbell-pop-sc.ca.us.ibm.net (165.87.225.9) 110.007 ms 118.412 ms
25 ded1-fa0-1-0.pbi.net (216.102.176.229) 110.922 ms 121.757 ms 120.744 ms
26 pbi.scrm01.foothill.net (206.13.15.82) 168.531 ms 120.297 ms 126.005 ms
27 inyo.E0.foothill.net (206.170.175.12) 139.673 ms 132.929 ms 127.300 ms
28 fhaub.foothill.net (207.212.142.2) 141.649 ms 122.945 ms 129.213 ms
Management for Real, Cont.
network monitors/analyzers
local systems
• take unit to problem
• don't depend on working network
• wide range of cost & function
remote systems
• leave unit on problem or key network
• remote control & viewing of information
privacy & security issues
[email protected]
Network Management and Network
Operations
66
Management for Real, Cont.
management agents
• SNMP agents in all "gateway" devices
• SNMP agents in all servers
• binary + "analog" reports
need something that knows what it is looking at
it
[email protected]
Network Management and Network
Operations
67
Management for Real
Which tools should I use? What do I really
need?
• Keep it simple!
• Need to consider engineers working remotely
• Don’t want to spend too much time maintaining the
tool (it should be helping you!)
• Different tools for NOC and engineers
• Different tools for statistics
• RELIABILITY!
[email protected]
Network Management and Network
Operations
68
Monitoring
simple monitoring tools do 95% of task
• e.g. ftp://ndtl.harvard.edu/pub/SNMPoll
• e.g. http://www.merit.edu/internet.tools/rover/
monitor should be both poll & trap based for
best reliability
• but just polling will do better than just traps
• and will work fine other than response latency
simple, terse, messages on problems
[email protected]
Network Management and Network
Operations
69
A Day in the Life of Merit’s NOC
running rover
•
•
•
•
prefer because easy to tell when change occurs
quickly can determine type of problem
no sifting through GUIs
quick screen display
alert appears on screen
27 Wed02:07 MCH_MSU:S6/1/7.6-->STOCKBRIDG 198.109.177.41 PING
28 Tue16:00 MCH_STOCKBRIDGE:S0.2-->JACKSO 198.109.177.46 PING
29 Tue16:00 MCH_STOCKBRIDGE:E0-GW 207.74.125.129 PING
30 Tue16:00 MCH_STOCKBRIDGE:S0.1-->MSU 198.109.177.42 PING
[email protected]
Network Management and Network
Operations
70
A Day in the Life of Merit’s NOC
open ticket
investigate
• the two most important questions:
– can you ping it?
– can you trace to it?
• get to the the node from somewhere else in the
network?
• dial-in to the router?
• serial line problem? call telco
If necessary, escalate to engineer
[email protected]
Network Management and Network
Operations
71
Another example - Sluggishness
customer calls NOC - reports sluggishness
open ticket
investigate
• check mrtg
– more traffic now than normal?
• use netflow to determine what type of traffic
– possible denial of service attack
• circuit problem?
– call telco to test
always call customer
back to get okay to close
Network Management and Network
[email protected]
Operations
72
Another example - DOS
Customer reports possible Denial of Service
Open ticket
Investigate
• notice a large amount of packets from one
destination?
– log onto router
– ip accounting
– sho ip route cache flow
• install packet filter
• report to offending ISP
[email protected]
Network Management and Network
Operations
73
Tracing packets on cisco - interface
access-group
cisco access list
• permit everything, but log packets from 10.2.3.4 to
195.176.0.0/16
– access-list 199 permit ip 10.2.3.4 0.0.0.0 195.176.0.0
0.0.255.255 log-input
– access-list 199 permit ip 0.0.0.0 255.255.255.255 0.0.0.0
255.255.255.255
apply access-list to interface
– interface serial3
– ip access-group 199 out
[email protected]
Network Management and Network
Operations
74
Tracing packets on cisco - debug ip
packet
cisco access list
• permit packets from 10.2.3.4 to 195.176.0.0/16, deny
others
– access-list 199 permit ip 10.2.3.4 0.0.0.0 195.176.0.0
0.0.255.255 log-input
– access-list 199 deny ip 0.0.0.0 255.255.255.255 0.0.0.0
255.255.255.255
use access-list with “debug ip packet”
– debug ip packet 199
[email protected]
Network Management and Network
Operations
75
Limiting bandwidth
access-list matches a class of traffic (e.g. ICMP)
use bandwidth management techniques to limit
amount of traffic in that class
• cisco CAR or traffic-shaping
[email protected]
Network Management and Network
Operations
76
Things to Look For
duplicate addresses
network/link load
router/bridge
•
•
•
•
•
CPU load
errors
drops!!
interface resets
collisions (if CSMA/CD network)
[email protected]
Network Management and Network
Operations
77
Things to Do (Defensive)
Filter!!! Filter!!! Filter!
Use the Internet Routing Registry!
• register your routes
• register your policy
• configure your routers off of the database!
– tools available
– http://www.isi.edu/ra/RAToolSet
use the Route Servers!
[email protected]
Network Management and Network
Operations
78
Route Filtering
BBN Planet
MIT
MCI
dial-up
provider
in VA
NAP
SPRINT
[email protected]
Network Management and Network
Operations
79
Things Not to Do
tunnel
complex routing
reconfig on the fly
[email protected]
Network Management and Network
Operations
80
Problems
we are early in the internet management game
• there is still a lot to learn
prices still high for functionality
• many new NMSs will be on the market soon, will
help lower price and expand capabilities
data networks are not "plug and play" with large
scale
nefarious people
[email protected]
Network Management and Network
Operations
81
More Problems
not so good at provoking simple, easy to
understand, warning to non-gurus
should have database & logic about when to cry
wolf
• critical vs, noncritical device, access restrictions,
who to call when
needs to be usable by "normal" people
needs to say when users will complain
[email protected]
Network Management and Network
Operations
82
Even more Problems
training your Network Operations Staff
keeping your database up-to-date
• router configs
• contact information
communication with the telco
[email protected]
Network Management and Network
Operations
83
More things you can do!
secure your router
• tacacs
• radius
• restrict login and snmp access
enable syslog logging
• security
• debugging
[email protected]
Network Management and Network
Operations
84
More things you can do!
Filtering
•
•
•
•
generate your filters off of the IRR
anti-spoofing filters
filter private networks (RFC 1918)
recommended filter list
– http://www.merit.edu/ipma/docs/help.html
[email protected]
Network Management and Network
Operations
85
More things you can do!
educate your NOC
• provide adequate documentation
• escalation procedures
register your routers in DNS
• traceroutes easier to follow
coolbeans% traceroute www.above.net
traceroute to www.above.net (207.126.96.163), 30 hops max, 40 byte packets
1 eth0-2.michnet1.mich.net (198.108.61.1) 1.074 ms 0.888 ms 0.696 ms
2 hssi1-0-0.msu.mich.net (198.108.22.102) 77.602 ms 75.356 ms 12.437 ms
3 aads.above.net (198.32.130.71) 9.981 ms 15.098 ms 11.342 ms
4 chicago-core1.ord.above.net (209.249.0.129) 9.634 ms 9.834 ms 9.590 ms
5 sjc-chicago-oc3.above.net (209.249.0.125) 71.261 ms 71.232 ms 71.305 ms
6 main2-core1-oc3-3.sjc.above.net (209.133.31.97) 123.499 ms 71.512 ms 71.8
7 www.above.net (207.126.96.163) 72.861 ms 72.624 ms 74.529 ms
[email protected]
Network Management and Network
Operations
86
More things you can do!
Prevent excessive route-flapping
• enable route-flap dampening
• use CIDR
• use filters
[email protected]
Network Management and Network
Operations
87
References
http://www.merit.edu/ipma/docs/isp.html
http://www.nanog.org
http://www.caida.org
http://www.nlanr.net
http://www.cisco.com
http://www.amazing.com/internet/
http://www.isp-resource.com/
http://www.merit.edu/ipma
http://www.ripe.net
[email protected]
Network Management and Network
Operations
88