Transcript Performance & Monitoring - Asian Institute of Technology

Netflow Overview
• Developed by Cisco Systems in 1996
• The value of information in the cache was a
secondary discovery
– Initially designed as a switching path
• NetFlow is now the primary network
accounting technology in the industry
• Answers questions regarding IP traffic:
who, what, where, when, and how
• NetFlow version 9 an IETF standard
Traffic Analysis
• What we needs
–
–
–
–
•
•
•
•
•
•
application performance
application-based accounting
network security
Network behavior, application recognition
‘debug ip packet’ in router?
IP Sniffing in shared LAN (or using switch to do so)
Port Span in switch (how about port span in router?)
Circuit Sniffing
Netflow
What we prefer in backbone:
– Embeded
– Fixed length partial packet export
– Real-time filtered packet export
Addressing The Needs with Netflow
Netflow Possible Applications
•
•
•
•
•
•
•
•
•
Network Monitoring
Network planning
Security Analysis
Application Monitoring
User Monitoring
Traffic Engineering
Peering Agreement
Usage-base Billing
Destination sensitive billing
What is a flow?
Defined by seven unique keys:
1. Source IP address
2. Destination IP address
3. Source port
4. Destination port
5. Layer 3 protocol
6. TOS byte (DSCP)
7. Input interface (ifIndex)
A Flow is Unidirectional!
Exported Data
NetFlow Sequence
1.
2.
3.
4.
5.
Create and update flows in NetFlow Cache
Expiration
Aggregation?
Export Version
Transport Protocol
NetFlow Sequence (continued)
SrcIPadd
173.100.21.2
173.100.3.2
173.100.20.2
173.100.6.2
DstIf
Fa0/0
Fa0/0
Fa0/0
Fa0/0
DstIPadd
10.0.227.12
10.0.227.12
10.0.227.12
10.0.227.12
Bytes/Pkt Active
1528
1745
740
41.5
1428
1145.5
1040
24.5
Idle
4
1
3
14
SrcIf SrcIPadd
DstIf DstIPadd
Protocol TOS Flgs Pkts SrcPort SrcMsk SrcAS DstPort DstMsk DstAS NextHop Bytes/Pkt Active
Fa1/0 173.100.21.2 Fa0/0 10.0.227.12 11
80 10 11000 00A2
/24
5
00A2
/24
15
10.0.23.2 1528
1800
Idle
4
step1
step2
Protocol
11
6
11
6
TOS
80
40
80
40
Flgs
10
0
10
0
Pkts
11000
2491
10000
2210
SrcPort SrcMsk SrcAS
00A2
/24
5
15
/26
196
00A1
/24
180
19
/30
180
DstPort
00A2
15
00A1
19
DstMsk
/24
/24
/24
/24
DstAS
15
15
15
15
NextHop
10.0.23.2
10.0.23.2
10.0.23.2
10.0.23.2
• Inactive timer expired (15 sec is default)
• Active timer expired (30 min (1800 sec) is default)
• NetFlow cache is full (oldest flows are expired)
• RST or FIN TCP Flag
step3
e.g. Protocol-Port Aggregation Scheme becomes
Protocol Pkts SrcPort DstPort Bytes/Pkt
11
11000 00A2
00A2
1528
step4
Aggregated Flows – export Version 8 or 9
Non-Aggregated Flows – export Version 5 or 9
step5
Export
Packet
Heade
r
SrcIf
Fa1/0
Fa1/0
Fa1/0
Fa1/0
Payload
(flows)
Netflow Processing Order
PreProcessing
Packet
Sampling
Filtering
Features
And
Services
IP
Multicast
MPLS
IPv6
Post
Processing
Aggregation
schemes
Non-key fields
lookup
Export
Creating Export Packets
Enable NetFlow
Traffic
Core Network
(IP, MPLS)
PE
Export Packets
•
•
•
Approximately 1500 bytes
Typically contain 20-50 flow
records
Sent more frequently if traffic
increases on NetFlow-enabled
interfaces
UDP NetFlow
Export
Packets
Collector (Solaris, HP-UX, or Linux)
Application:
Performance
Billing
Security
NetFlow Principles
• Inbound traffic only (with some exceptions)
• Unidirectional flow
• Accounts for both transit traffic and traffic destined
for the router
• Works with Cisco Express Forwarding (CEF) or fast
switching
• Almost supported on all interfaces and Cisco IOS
Software platforms
• Provides the sub-interface information in the flow
records
• 6500/7600 enables Netflow on all interfaces by default
Comprehensive Platform Support
GSR
12000
ESR
10000
Si
7200/
7500/
AS5300/
5800
3700
3600
1400/
1600/
1700
2500/
2600
4500/
4700
Catalyst
5000/6500/
Catalyst
7600
4500
NetFlow Versions
Version 5 - Flow Format
Usage
• Packet Count
• Byte Count
• Source IP Address
• Destination IP Address
Time
of Day
• Start sysUpTime
• End sysUpTime
• Source TCP/UDP Port
• Destination TCP/UDP
Port
• Input ifIndex
• Output ifIndex
• Next Hop Address
• Source AS Number
• Dest. AS Number
• Source Prefix Mask
• Dest. Prefix Mask
QoS
• Type of Service
• TCP Flags
• Protocol
From/to
Application
Routing
and
Peering
Blue – Key Field (7)
Red - Lookup Field (5)
Black- Value Field (6)
Netflow Configuration Commands
•
ip flow-export version <version> [origin-as | peer-as | bgp-nexthop]
– e.g. ip flow-export version 5
•
ip flow-export destination <address> <port>
– e.g. ip flow-export destination 10.0.0.1 65001
•
ip flow export source <interface>
– default is interface with best route to collector. Recommendation:
configure loopback interface.
•
ip flow-aggregation cache <name of aggregation scheme>
– select the aggregation cache
•
ip flow-cache timeout inactive <seconds>
– sets the seconds an inactive flow will remain in the cache before expiration.
15 seconds is default
•
ip flow-cache timeout active <mintues>
– sets the minutes an active flow will remain in the cache bvefore expiration.
30 minutes is default
•
ip flow-cache entries <number>
– sets the maximum number of flow entries in the cache. The default varies
dependent on platform.
Netflow Show Commands
• show ip cache [verbose] flow
– shows Netflow statistics
• show cache flow aggregation <name of aggregation
scheme>
– shows netflow statistics for the configured aggregation
scheme
• show ip flow export
– shows export statistics
• clear ip cache flow
– clears netflow statistics
• clear ip flow stats
– clears export statistics
Show ip cache flow
IP packet size distribution (2175M total packets):
1-32 64 96 128 160 192 224 256 288 320 352 384 416 448 480
.001 .440 .139 .014 .008 .000 .000 .000 .000 .000 .000 .000 .011 .000 .000
512 544 576 1024 1536 2048 2560 3072 3584 4096 4608
.000 .000 .000 .002 .377 .000 .000 .000 .000 .000 .000
IP Flow Switching Cache, 4456704 bytes
550 active, 64986 inactive, 509378135 added
3145787062 ager polls, 0 flow alloc failures
Active flows timeout in 30 minutes
Inactive flows timeout in 15 seconds
Protocol
Total Flows Packets Bytes Packets Active(Sec) Idle(Sec)
-------Flows /Sec /Flow /Pkt /Sec /Flow /Flow
TCP-WWW
10431912
2.4
10 181 25.7
7.1
20.4
TCP-SMTP
773843
0.1
6 98
1.1
8.3
16.7
…….
Total:
509377507 118.5
4 567 506.4
1.7
15.9
SrcIf
Te7/3
Te7/3
Te7/3
SrcIPaddress
219.245.101.77
84.97.234.47
222.81.87.163
DstIPaddress
Pr
202.205.5.3
tcp
202.204.192.18 udp
202.205.3.203 tcp
SrcP
1444
7692
1172
DstP
1203
2881
Pkts
1
1
Show ip flow export
Router> sh ip flow export
Flow export v5 is enabled for main cache
Exporting flows to 192.168.1.2 (2055) 192.168.2.3 (2054)
Exporting using source interface Loopback0
Version 5 flow records, origin-as
998016649 flows exported in 33267252 udp datagrams
0 flows failed due to lack of export packet
0 export packets were sent up to process level
0 export packets were dropped due to no fib
0 export packets were dropped due to adjacency issues
0 export packets were dropped due to fragmentation failures
0 export packets were dropped due to encapsulation fixup
failures
0 export packets were dropped enqueuing for the RP
0 export packets were dropped due to IPC rate limiting
Version 7
• Adds NetFlow switching support for:
 Cisco Catalyst 5000 Series Switches with an RSM
 Cisco Catalyst 5000 Series Switches with an MSFC
• Uses MultiLayer Switching (MLS) or CEF with Cisco
Catalyst 6000 Series Switches with SUP2
• IP unicast only
 No multicast or IPX, even if MLS can do all three
• MLS cache is the equivalent of the NetFlow cache
Version 8
•
•
•
•
•
Router-based aggregation
Enables router to summarize NetFlow data
Reduces NetFlow Export data volume
Decreases NetFlow Export bandwidth requirements
Currently 11 aggregation schemes
 Five original schemes
 Six new schemes with the TOS byte field
• Several aggregations can be enabled simultaneously
Version 9
 Fixed formats (versions 1, 5, 7, and 8) are not flexible
and adaptable
 Cisco needed to build a new version each time a customer
wanted to export new fields
 When new versions are created, partners need to
reengineer to support the new export format
Solution: Build a flexible and
extensible export format!
Netflow v9 Principles
•
•
•
•
Version 9 is an export format
Still a push model
Sent the template regularly (configurable)
Independent of the underlying protocol, it is
ready for any reliable protocol (ie: TCP,
SCTP)
• Advantage: we can add new technologies
and data types quickly
• E.g. MPLS, IPv6, BGP Next Hop, Multicast
Netflow V9 Template
• NetFlow Version 9 Export format is template based. Version 9
record format consists of a packet header followed by at least
one or more template or data FlowSets. A template FlowSet
(collection of one or more template) provides a description of
the fields that will be present in future data FlowSets.
Templates provide an extensible design to the record format, a
feature that should allow future enhancements to NetFlow
services without requiring concurrent changes to the basic
flow-record format.
• template composed of type and length
• flow records composed of template ID and value
• sent the template regularly (configurable), because
of UDP
Netflow Version 9 Scenario
Netflow v9: Example for Template
Definition
Netflow Version9 Export Packet
Netflow v9: Example for 1 Export
Packet
NetFlow v9 Export Packet
To support technologies such as
MPLS or Multicast, this export format can
be leveraged to easily insert new fields
Header
(version,
# packets,
sequence #,
Source ID)
Template FlowSet
Template
Record
Template ID #1
Template
Record
Template ID #2
(specific Field
types and
lengths)
(specific Field
types and
lengths)
Flows from
Interface A
Data FlowSet
FlowSet ID #1
Data Record
(Field values)
Flows from
Interface B
Data FlowSet
FlowSet ID #2
Option
Template
FlowSet
Data Record
Data Record
Template ID
(Field values)
(Field values)
(specific
Field types
and lengths)
Option Data
FlowSet
FlowSet ID
Option
Data
Record
Option
Data
Record
(Field
values)
(Field
values)
• Matching ID #s is the way to associate Template to the Data Records
• The Header follows the same format as prior NetFlow versions so Collectors will be backward
compatible
• Each Data Record represents one flow
• If exported flows have the same fields then they can be contained in the same Template Record
e.g. unicast traffic can be combined with multicast records
• If exported flows have different fields then they can’t be contained in the same Template Record
e.g. BGP next-hop can’t be combined with MPLS Aware NetFlow records
NetFlow v9 Export
test(config)# ip flow-export version ?
1
5
Configuring Version 9
9
test(config)# ip flow-export version 9
export
.
Export versions available for
standard NetFlow flows
Configuring Version 9 export for an aggregation scheme
test(config)# ip flow-aggregation cache as
test(config-flow-cache)# enabled
test(config-flow-cache)# export ?
destination Specify the Destination IP address
version configure aggregation cache export version
test(config-flow-cache)# export version ?
8
Version 8 export format
9
Version 9 export format
Export versions available for
test(config-flow-cache)# export version 9 aggregated NetFlow flows
IETF: IP Flow information
Export(IPFIX) Working Group
• IPFIX is an effort to:
– Define the notion of a "standard IP flow"
– Devise data encoding for IP flows
– Consider the notion of IP flow information export based
upon packet sampling
– Identify and address any security privacy concerns
affecting flow data
– Specify the transport mapping for carrying IP flow
information(IETF approved congestion-aware transport
protocol)
– Netflow version 9 has been selected as a basis for the IPFIX
protocol
IETF: Packet Sampling WG(PSAMP)
• PSAMP agreed to use IPFIX(Netflow version9)
for export
• PSAMP is an effort to:
– specify a set of selection operations by which
packets are sampled
– describe protocols by which information on
sampled packets is reported to applicatons
• http://www.ietf.org/html.charters/psampcharter.html
• Note: Netflow is already using some
sampling mechanisms
NetFlow Infrastructure
NetFlow Uses
Distribution
• Attack Mitigation
• User (IP)
monitoring
• Application
monitoring
• Billing
• Chargeback
• AS Peer
Monitoring
• Aggregation
Schemes (v8)
• “show ip cache
flow” command
• Arbor Networks
• NetFlow
MPLS Egress
Accounting
• BGP Next-hop
(v9)
• Multicast
NetFlow (v9)
Core
Distribution
Access
• Traffic
Engineering
• Traffic
Analysis
• Billing
• Chargeback
• AS Peer
Monitoring
• Attack Mitigation
• User (IP)
monitoring
• Application
monitoring
• MPLS Aware
NetFlow (v9)
• BGP Next-hop
(v9)
• Sampled
NetFlow
• NetFlow
MPLS Egress
Accounting
• BGP Nexthop (v9)
• Multicast
NetFlow (v9)
• Aggregation
Schemes (v8)
• “show ip cache
flow” command
• Arbor Networks
NetFlow
Features
Applications
Network Layer
Access
Netflow Collector(NFC) 5.0
Netflow on the Network Analysis
Module (NAM)
Netflow Partners
Billing
 Flat-rate billing does not necessarily scale
Competitive pricing models can be created with usagebased billing
 Usage-based billing considerations
Time of day
Within or outside of the network
Application
Distance-based
Quality of Service (QoS) / Class of Service (CoS)
Bandwidth usage
Transit or peer
Data transferred
Traffic class
Tracking Users
 Who are my top N talkers, and what percentage of traffic do they
represent?
 How many users are on the network at a given time?
 When will upgrades affect the least number of users?
 How long do users spend connected to the network?
 Where Internet sites do they use?
 What is a typical pattern of usage between sites?
 Are users staying within an acceptable usage
policy (AUP)?
 Alarm DOS attacks like smurf, fraggle, and SYN flood
 Will watch for these attack, regardless of source / destination
Principle Netflow Benefits
Service Provider
Enterprise
• Traffic Engineering
• Internet access
monitoring (protocol
distribution, where traffic
is going/coming)
• Accounting and billing
• User Monitoring
• Security Monitoring
• Application Monitoring
• Peering arrangements
• Network Planning
• Charge Back billing for
departments
• Security Monitoring
NetFlow – Charge Back Billing
Account per network (rather that per IP addresses)
Example: charge the department
for the cost of the Internet link
Internet
Finance
HR
R&D
NetFlow – Peering Agreement
Account per BGP AS, to
Review Peering Agreements
ISP
NetFlow – Peering Agreement
Public Routers 1, 2, 3 Month of
September—Outbound Traffic
4% 2% 1% 1%1%
6%
1%1%
1%
1%
1%
1%
8%
8%
10%
32%
20%
Uunet
Digex
Erols
BBN
AT&T
AMU
C&W
JHU
PACBell Internet Service
RCN
OARnet
SURAnet
Compuserve
OL
ABSNET
WebTV
WEC
MPLS Aware NetFlow (v9)
Source and destination IP address
Input and output sub-interfaces
IP Fields
Transport layer protocol
Source and destination application port numbers
8 bit IP Type of Service (ToS)
TCP Flags (accumulation from all packets in the flow)
Up to three incoming MPLS labels with experimental (EXP)
bits and end-of-stack (S) bit
MPLS Fields
Position of each of the three labels
Type of the top label
IP address associated with the top label
Traditional
NetFlow Fields
Number of packets
Number of bytes (count either IP or MPLS header / payload)
Time-stamps of first and last packets in the flow
MPLS
Traditional NetFlow
for IP to MPLS traffic
MPLS Aware NetFlow
(version 9)
Egress MPLS NetFlow Accounting
for MPLS to IP traffic
IP
MPLS
IP
PE
P
PE
Traffic Flow
Egress MPLS NetFlow Accounting
• IP information only
• Ideal for billing
• Current availability: Cisco IOS Software Releases 12.0(10)ST and 12.1(5)T
MPLS Aware NetFlow (version 9)
• Exports up to three MPLS labels, and IP packet information
• Ideal for Traffic Engineering
• Will be available in Cisco IOS Software Releases 12.0(24)S, 12.2S, and 12.3
Autonomous System
• Origin-AS
Specifies that export statistics include the origin
autonomous system (AS) for the source and
destination
• Peer-AS
Specifies that export statistics include the peer
AS for the source and destination
3600-4(config)# ip flow-export version 5 ?
origin-as record origin AS
peer-as
record peer AS
<cr>
3600-4(config)#
Autonomous System
NetFlow enabled
AS 101
AS 102
AS 103
Configuring Peer-AS
• Source AS = AS 103
• Destination AS = AS 105
AS 104
AS 105
Router(config)#ip flow-export version 5 peer-as
AS 106
Autonomous System
NetFlow enabled
AS 101
AS 102
AS 103
AS 104
AS 105
Router(config)#ip flow-export version 5 origin-as
Configuring Origin-AS
• Source AS = AS 101
• Destination AS = AS 106
AS 106
BGP next-hop
• Supported only in version 9 export
• For traffic engineering/analysis and possible
billing applications
• Fields that are exported include all those
found in version 5 export
• Will be supported in Cisco IOS Software
Releases 12.0(26)S, 12.2S, and 12.3
BGP next-hop
Netflow BGP next-hop
BGP next-hop Details
• Supported only in version 9 export
• For traffic engineering/analysis (traffic matrix) and
possible billing applications. "What is the Next hop
IP address of my BGP traffic?"
• exported fields include all version 5 fields, including
IP next hop
• Adds 16 bytes to each Netflow flow record (goes
from 64 bytes to 80 bytes), while CPU increase is
negligible
• Edge to Edge traffic matrix for engineering/analysis
and possible billing applications
• Supported in Cisco IOS Software releases 12.0(26)S,
12.2(18)S, and 12.3(1)
BGP next-hop
pamela(config)# ip flow-export version ?
1
5
Configuring Version 9
9
pamela(config)# ip flow-export version 9
export
.
Configuring Version 9 export with BGP next-hop
pamela(config)# ip flow-export version 9 ?
bgp-nexthop record BGP NextHop
origin-as record origin AS
peer-as record peer AS
<cr>
pamela(config)# ip flow-export version 9 bgp-nexthop
Multicast NetFlow
Three types of NetFlow implementations for
Multicast traffic:
1. Traditional NetFlow
2. Multicast NetFlow Ingress
3. Multicast NetFlow Egress
Multicast – Traditional NetFlow
Traditional NetFlow configuration
(S, G) - (10.0.0.2, 224.10.10.100)
Interface Ethernet 0
NetFlow
Collector
server
ip route-cache flow
10.0.0.2
ip flow-export destination 127.0.0.1 9995
127.0.0.1
Eth 0
ip flow-export version 9
Eth 1
Eth 3
Eth 2
Flow Record Created in NetFlow Cache
SrcIf SrcIPadd DstIf DstIPadd
Protocol TOS Flgs SrcPort SrcMsk DstPort DstMsk NextHop Bytes Packets Active Idle
Eth 0 10.0.0.2 Null 224.10.10.100 11
80 10 00A2
/24
00A2
/24
23100 21
1745 4
•
•
•
•
There is only one flow per NetFlow configured input interface
The 7 Key fields that define a unique flow are marked in red
Destination interface is marked as “Null”
Bytes and Packets are the incoming values
Multicast NetFlow Ingress
Multicast NetFlow Ingress configuration
Interface Ethernet 0
(S, G) - (10.0.0.2, 224.10.10.100)
NetFlow
Collector
server
10.0.0.2
ip multicast netflow ingress
127.0.0.1
Eth 0
ip flow-export version 9
Eth 1
Eth 3
ip flow-export destination 127.0.0.1 9995
Eth 2
Flow Record Created in NetFlow Cache
SrcIf SrcIPadd DstIf DstIPadd
Protocol TOS Flgs SrcPort SrcMsk DstPort DstMsk NextHop Bytes Packets Active Idle
Eth 0 10.0.0.2 Null 224.10.10.100 11
80 10 00A2
/24
00A2
/24
69300 63
1745 4
•
•
•
•
There is only one flow per NetFlow configured input interface
The 7 Key fields that define a unique flow are marked in red
Destination interface is marked as “Null”
Bytes and Packets are the outgoing values
Multicast NetFlow Egress
Multicast NetFlow Egress configuration
(S, G) - (10.0.0.2, 224.10.10.100)
NetFlow
Collector
server
Interface Ethernet 1
ip multicast netflow egress
10.0.0.2
Interface Ethernet 2
ip multicast netflow egress
127.0.0.1
Eth 0
Interface Ethernet 3
ip multicast netflow egress
Eth 1
Eth 3
ip flow-export version 9
Eth 2
ip flow-export destination 127.0.0.1 9995
Flow Records Created in NetFlow Cache
SrcIf
Eth 0
Eth 0
Eth 0
SrcIPadd
10.0.0.2
10.0.0.2
10.0.0.2
DstIf
Eth 1
Eth 2
Eth 3
DstIPadd
224.10.10.100
224.10.10.100
224.10.10.100
Protocol
11
11
11
TOS
80
80
80
Flgs
10
10
10
SrcPort
00A2
00A2
00A2
SrcMsk
/24
/24
/24
DstPort
00A2
00A2
00A2
DstMsk NextHop Bytes Packets
/24
23100 21
/24
23100 21
/24
23100 21
Active
1745
1745
1745
• There is one flow per Multicast NetFlow Egress configured output interface
• One of the 7 Key fields that define a unique flow has changed from Source Interface to Destination Interface
• Bytes and Packets are the outgoing values
Idle
4
4
4
Multicast NetFlow – Summary
 Supported via NetFlow version 9 export format
 Availability
Cisco IOS Software Releases 12.0(27)S, 12.2S, and 12.3
 Not supported in 120000
 Performance: Ingress vs. Egress
Multicast NetFlow Ingress and traditional NetFlow will have
similar performance numbers
Multicast NetFlow Egress will have performance impact
that is proportional to the number of interfaces on which it is
enabled (include input interface)
 Cisco Catalyst 6500/7600 Series Switches
Do not currently support the tracking of multicast traffic via
NetFlow due to current ASIC limitation
Will have this support in a future Supervisor
How to Identify a Security Attack?
• Suddenly highly-increased overall traffic in the
network
• Higher CPU and memory utilization of network
devices
• Unexpectedly large amount of traffic generated by
individual hosts
• Increased number of accounting records generated
• Multiple accounting records with abnormal content,
like one packet per flow record (e.g. TCP SYN flood)
• A changed mix of traffic applications, e.g. a sudden
increase of "unknown" applications
• An increase of certain traffic types and messages,
e.g. TCP resets or ICMP messages
• An increasing number of ACL violations
What Does a DOS Attack Look Like?
NetFlow – Mitigating Attacks
1. Cost Saver
•
•
•
•
“sh ip cache flow” command to find top volume flows
Identify source of attack
Write access-list to block
Monitor via “show ip cache flow” & “Null” entry in DestIf field
to show that it is blocked
• Prefix-port aggregation can be configured, while “sh ip cache
flow aggregation prefix-port” is used
2. Most Effective
• Arbor Networks leverages NetFlow to provide a quicker
response and more sophisticated solution
Security Analysis: Best Practices
Quality of Service Example
ToS bits
Precedence bits
DS5
DS4
DS3
DS2
DS1
DS0
ECN
ECN
128
64
32
16
8
4
2
1
DiffServ field
AKA
IP DSCP markings
Early Congestion
Notification
(ECN) bits
Quality of Service Example
TOS byte
DS5 DS4 DS3 DS2 DS1 DS0 ECN ECN
128 64
32
16
8
4
2
1
Precedence bits
1
1
1
1
1
0
1
0
1
1
0
0
0
1
1
0
1
0
0
0
1
0
0
0
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
Delay, Throughput, and Reliability bits
Delay bit
x
x
x
0
x
x
x
x
x
x
1
x
x
x
Throughput bit
x
x
x
x
0
x
x
x
x
x
x
1
x
x
Reliability bit
x
x
x
x
x
0
x
x
x
x
x
x
1
x
x
x
x
x
x
x
x
x
Decimal
224
192
160
128
96
64
32
0
x
x
0
16
Delay - normal
Delay - low
x
x
0
8
Throughput - normal
Throughtput - high
x
x
0
4
Reliability - normal
Reliability - high
Precedence
7
6
5
4
3
2
1
0
Early Congestion Notification (ECN) bits
ECN-capable Transport (ECT) bit
Congestion Experienced (CE) bit
x
x
x
x
x
x
0
0
0
x
x
x
x
x
x
0
1
1
x
x
x
x
x
x
1
0
2
x
x
x
x
x
x
1
1
3
Function
Network Control (link layer keepalives)
Internetwork Control (Routing Protocols)
CRITIC/ECP (Express Forwarding)
Flash Override (Class 4)
Flash (Class 3)
Immediate (Class 2)
Priority (Class 1)
Routine (Best effort)
Not ECN-capable
Endpoints of transport protocol ECN-capable
Endpoints of transport protocol ECN-capable
Congestion experienced
Tracking TOS with NetFlow
Hex
7200-3-netflow# show ip cache verbose flow
SrcIf
SrcIPaddress
DstIf
Port Msk AS
Port Msk AS
SR6/0
210.210.210.2
PO1/0
0000 /0 0
0000 /0 0
SR6/0
210.210.210.2
PO1/0
0000 /0 0
0000 /0 0
DstIPaddress
NextHop
200.200.200.2
0.0.0.0
200.200.200.2
0.0.0.0
Pr TOS Flgs Pkts
B/Pk Active
FF 00 10
21K
1496
665.4
06 C0 00
21K
1496
666.0
7200-3-netflow# show ip cache verbose flow
SrcIf
SrcIPaddress
DstIf
Port Msk AS
Port Msk AS
Et1/1
52.52.52.1
Fd4/0
0000 /8 50
0000 /8 40
Et1/2
52.52.52.1
Fd4/0
0000 /8 50
0000 /8 40
Et1/2
10.1.3.2
Fd4/0
0000 /0 0
0000 /8 40
DstIPaddress
NextHop
42.42.42.1
202.120.130.2
42.42.42.1
202.120.130.2
42.42.42.1
202.120.130.2
Pr TOS Flgs Pkts
B/Pk Active
01 55 10
3748
28
17.8
01 CC 10
3568
28
17.8
01 C0 10
1124
28
17.8
Decimal
Binary
55
85
0101 0101
of transport protocol ECN-capable
Precedence 2 - Immediate (Class 2), Delay - low, Reliability - high, Endpoints
C0
1100 0000
Precedence 6 - Internetwork Control (Routing Protocols)
1100 1100
Precedence 6 - Internetwork Control (Routing Protocols), Throughput - high,
192
CC
204
Reliability - high
Sampled NetFlow
Deterministic
Original type
Cisco 12000 Series Internet Routers
Cisco Catalyst 6500 Series Switches – Release 12.1(13)E
Random (recommended per statistical principles)
Cisco IOS Software Releases 12.0(26)S, 12.2S, and 12.3
Cisco 2500, 2600, 3600, 7200, and 7500 Series Routers
Cisco 12000 Series Internet Routers
Time-based
Cisco Catalyst 6500 Series Switches – Release 12.1(13)E
Trajectory (Hash-based)
in development
Sampling configuration
• GSR 12xxx (IOS Version: 12.0(31)S2:
R1(config)# ip flow-sampling-mode packet-interval 256
R1(config-if)# ip route-cache flow sampled input
R1(config-if)# ip route-cache flow sampled output
bj2-bgw(config)#ip flow-sampling-mode packet-interval ?
<10-16382> Specify the packet interval at which to sample
• 7609: (12.2(18)SXD6)
R1(config)# mls flow ip source
R1(config)# mls nde sender version 5
R1(config)# mls sampling time-based 64
R1(config-if)# ip route-cache flow
R1(config-if)# mls netflow sampling
// 64:1
Cisco Catalyst 6500 and 7600 Series
Switches
• Export is centrally via the supervisor and MSFC, each line card
has its own hardware NetFlow cache and forwarding table, i.e.
distributed platform
Cisco 12000 Series Internet
Routers – NetFlow
• Engine 0 – software support
• Engine 1 – software support
• Engine 2 – supported in ASICs, but lower
priority so beware if running many other
features
• Engine 3 – version 5 support in software,
version 8 support in ASIC
• Engine 4 – not supported
• Engine 4+ – supported in ASICs
Cisco 12000 Series Internet
Routers Sampled NetFlow
Engine
Full NetFlow
Sampled
NetFlow
0
1
2
3
4
4+
Supported
Not supported
Scaling - Memory Utilization
Scaling - Sample Traffic
Deterministic vs. Random Sampling
Sampled Netflow Details
• Deterministic
– Cisco C6500/7600 Series switches(12.1(13)E)
– Cisco 12000 series internet routers (12.0(11)S and
12.0(14)ST)
• Random (select packet to export per statistical
principles)
– Cisco IOS Software Releases 12.0(26)S, 12.2S(18), and
12.3(1)T
– Cisco 800, 1700, 1800, 2600, 2800, 3600, 3700, 3800, 7200,
and 7500 series routers
• Time-based
– Cisco C6500/7600 series Random and Time based sampling
12.1(13)E
Sampled Netflow CPU Reduction
Netflow Multiple Export Destinations
Performance Testing Conclusions
• Additional CPU utilization
Number of Active Flows
Additional CPU Utilization
10,000
<4%
45,000
<12%
65,000
<16%
• NetFlow Data Export (single/dual)
No significant impact
• NetFlow v5 versus v8: little or not impact
• NetFlow Feature Acceleration:
>200 lines of ACLs and/or Policy Based-Routing (PBR)
• NetFlow versus Sampled NetFlow on the Cisco
12000 Series Internet Routers
23% versus 3% (65,000 flows, 1:100)
Performance Testing
NetFlow Version 9
• Similar CPU and throughput numbers result
from configuration of both NetFlow version 5
and 9
• No change in NetFlow performance after the
addition of version 9
Cisco IOS Software Releases 12.0(24)S, 12.2S, and
12.3
• CPU is slightly higher immediately following
initial boot up or configuration
Caused by sending Template Flowsets to Collector
Reducing Performance Impact
Reduce CPU and memory impact on the router,
collector, or network:
•
•
•
•
•
•
•
•
•
•
Aging timers (router)
Sampled NetFlow (router)
Enable NetFlow Feature Acceleration (router)
Flow Masks (only Cat6000/7600)
Enable on specific sub-interface (upcoming router feature)
Aggregation schemes (v8 on router or on collector)
Filters (router or collector)
Data Compression (collector)
Increase collection bucket sizes (collector)
Collector and router can be placed on the same LAN segment
(network)
Netflow Deployment: Rules of Thumb
Netflow Deployment: Considerations
Cisco Netflow MIB
Netflow MIB applications
• Netflow Configuration
• Checking Netflow Configuration
• Monitoring and security
– export statistics
– protocol statistics
– top flows information (top talkers)
Netflow Mib Overview
• Defined groups of objects
1. cnfCacheInfo
• A group of objects related to cache information and configuration
stored per cache configuration.
2. cnfExportInfo
• A group of objects related to Export configuration and information.
4. cnfExportStatistics
• Provides export statistics.
5. cnfProtocolStatistics
• Provides a summary of NetFlow cache statistics per protocol and
port.
6. cnfExportTemplate
• Provides Template based Version 9 flow export information and
statistic.
7. cnfTopFlows
• Provides top Netflow flows.
Netflow MIB Monitoring
Egress Netflow Accounting
Netflow and IPv6
•
•
•
•
•
•
Collects IPv6 flow records
Based on Netflow Version9
Support or both ingress and egress traffic
"Full NetFlow" i.e. non-sampled
Data export is still IPv4
Available in release 12.3(7)T
Netflow Summary
• Netflow is a mature Cisco IOS feature (in
Cisco IOS since 1996)
• Netflow provides input for Accounting,
Performance, Fault, Security, and Billing
Applications
• Cisco has IETF and industry leadership
• Netflow v9 eases the exporting of additional
fields
• A lot of new features have been added
SFlow
• sFlow® is an industry standard technology for monitoring high
speed switched networks, Juniper’s devices support it.
•
•
similar to netflow
NetStream from Huawei Company
• SFlow Packet:










Packet header (eg MAC,IPv4,IPv6,IPX,AppleTalk,TCP,UDP, ICMP)
Sample process parameters (rate, pool etc.)
Input/output ports
Priority (802.1p and TOS)
VLAN (802.1Q)
Source/destination prefix
Next hop address
Source AS, Source Peer AS
Destination AS Path
Communities, local preference
 User IDs (TACACS/RADIUS) for source/destination
 URL associated with source/destination
 Interface statistics (RFC 1573, RFC 2233, and RFC 2358)
Tools for Netflow
• Cisco NFC
• Arbor Peakflow
• Flow tools
• Ntop
– http://ww.ntop.org
• Etc.
Flow-tools
• Flow-tools is library and a collection of programs
used to collect, send, process, and generate reports
from NetFlow data.
• Can be used together on a single server or
distributed to multiple servers for large deployments.
• The flow-tools library provides an API for
development of custom applications for NetFlow
export versions 1,5,6 and the 14 currently defined
version 8 subversions.
• Version 9 is not supported now
Flow-tools utilities
•
•
•
•
•
•
•
•
flow-capture - Collect, compress, store, and manage disk space for
exported flows from a router.
flow-cat - Concatenate flow files. Typically flow files will contain a
small window of 5 or 15 minutes of exports. Flow-cat can be used to
append files for generating reports that span longer time periods.
flow-fanout - Replicate NetFlow datagrams to unicast or multicast
destinations. Flow-fanout is used to facilitate multiple collectors
attached to a single router.
flow-report - Generate reports for NetFlow data sets. Reports include
source/destination IP pairs, source/destination AS, and top talkers.
Over 50 reports are currently supported.
flow-tag - Tag flows based on IP address or AS #. Flow-tag is used to
group flows by customer network. The tags can later be used with
flow-fanout or flow-report to generate customer based traffic reports.
flow-filter - Filter flows based on any of the export fields. Flow-filter
is used in-line with other programs to generate reports based on flows
matching filter expressions.
flow-import - Import data from ASCII or cflowd format.
flow-export - Export data to ASCII or cflowd format.
Flow-tools utilities( Cont.)
• flow-send - Send data over the network using the NetFlow
protocol.
• flow-receive - Receive exports using the NetFlow protocol
without storing to disk like flow-capture.
• flow-gen - Generate test data.
• flow-dscan - Simple tool for detecting some types of network
scanning and Denial of Service attacks.
• flow-merge - Merge flow files in chronoligical order.
• flow-xlate - Perform translations on some flow fields.
• flow-expire - Expire flows using the same policy of flowcapture.
• flow-header - Display meta information in flow file.
• flow-split - Split flow files into smaller files based on size, time,
or tags.
Configuration in Cisco Router
R1(config)# ip flow-export source Loopback0
R1(config)# ip flow-export version 5 origin-as
R1(config)# ip flow-export destination 202.112.xx.xx 9800
R1(config-if)# ip route-cache flow
flow-capture
• Flow-tools most useful and important command
• flow-capture -w /flows/dat -m 255.255.248.0 -E5G 0/10.0.0.1/9800
– Receive flows from the exporter at 10.0.0.1 port 9800.
Maintain 5 Gigabytes of flow files in /flows/dat. Mask the
source and destination IP addresses contained in the flow
exports with 255.255.248.0.
• flow-capture -w /flows/dat 0/0/9800 -S5
– Receive flows from any exporter on port 9800. Do not
perform any flow file space management. Store the exports
in /flows/dat. Emit a stat log message every 5 minutes.
Flow-cat
Flow-print
FreeBSD1# flow-print < ft-v01.2006-09-02.134114+0800
srcIP
202.204.79.253
202.204.79.253
202.204.79.253
202.204.79.253
202.204.79.253
202.204.79.253
dstIP
prot sPort dPort octets
202.204.239.227 6 4414 1433 48
202.204.239.229 6 4450 1433 96
202.204.239.240 6 4535 1433 48
202.204.239.228 6 4443 1433 48
202.204.239.233 6 4472 1433 96
202.204.239.231 6 4461 1433 48
pkts
1
2
1
1
2
1
Flow-stat
Flow-stat exam. 1
% flow-cat -p /flows/dat | flow-stat
IP packet size distribution:
1-32 64 96 128 160 192 224 256 288 320 352 384 416 448 480
.000 .906 .029 .004 .002 .009 .001 .001 .004 .027 .004 .000 .000 .000 .000
512 544 576 1024 1536 2048 2560 3072 3584 4096 4608
.000 .000 .001 .001 .012 .000 .000 .000 .000 .000 .000
Packets per flow distribution:
1 2 4 8 12 16 20 24 28 32 36 40 44 48 52
.812 .157 .010 .013 .006 .001 .000 .000 .000 .000 .000 .001 .000 .000 .000
60 100 200 300 400 500 600 700 800 900 >900
.000 .001 .000 .000 .000 .000 .000 .000 .000 .000 .000
Octets per flow distribution:
32 64 128 256 512 1280 2048 2816 3584 4352 5120 5888 6656 7424 8192
.000 .754 .183 .009 .012 .015 .014 .008 .004 .002 .000 .000 .000 .000 .000
8960 9728 10496 11264 12032 12800 13568 14336 15104 15872 >15872
.000 .000 .000 .000 .001 .000 .000 .000 .000 .000 .001
Flow time distribution:
10 50 100 200 500 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000
.812 .000 .000 .000 .000 .000 .001 .131 .015 .001 .004 .004 .004 .002 .001
12000 14000 16000 18000 20000 22000 24000 26000 28000 30000 >30000
.000 .001 .001 .002 .001 .000 .002 .001 .001 .000 .017
formats
Flow-stat exam. 2
• flow-cat -p /flows/dat | flow-stat -f10 -S4
– Provide a report on top source/destination
IP pairs sorted by octets
# Fields: Total
# Symbols: Disabled
# Sorting: Descending Field 4
# Name:
Source/Destination IP
#
# src IPaddr dst IPaddr
flows
#
202.204.192.1 10.20.0.12
1
202.204.192.1 10.20.0.8
3
202.204.192.1 10.20.0.9
2
202.204.193.1 64.84.7.4
1
202.204.204.148 221.137.69.66
3
216.186.143.246 202.204.227.118 1
202.204.79.253 202.204.239.233
1
octets
packets
3720
3128
3269
390
144
144
96
12
11
11
3
3
3
2
Flow-scan
Netflow in CERNET-POP Traffic
Statistics
Netflow in CERNET-POP PPS
Statistics
Netflow in CERNET-POP Average
Packet Size Statistics
Netflow in CERNET-POP Protocol
Statistics
Thank You!
• Most materials in this PPT is from network,
thanks goes to the authors
• Any Questions?