Transcript Beveiligingsdag - University of Oregon

Bandwidth management and
optimization
BCrouter
14-16 March 2006
Dirk Janssens
ICTS – K.U.Leuven
1
Introduction into introduction
BCrouter is an ongoing network project
 Not all features are already implemented or
ready for 3th party deployment
Constructive feedback
 What do you expect from a good solution
 Try to fulfill as many expectations as possible
2
Overview
1. Introduction



Problem
Expectations
BCrouter solution
2. BCrouter solution




Components
Example network setups
Integration
Security considerations
3. BCrouter





4. BCpolicer




Introduction
Design principles
Policing alternatives
Complete design
5. Case study: KotNet
6. Development



Current status
Future
Wish list
Introduction
Components
Commands and logging
Routing and Netfilter setup
Quota/Bandwidth exceptions
3
Introduction: problem
Bandwidth usage rises rapidly




Increasing Internet population
‘Richer’ content (HTML,Flash,…)
P2P download applications
Video/music streaming
Bandwidth availability is limited
 Expensive uplink
• No alternatives
 Expensive hardware
4
Introduction: problem
 Majority of bandwidth
 used by minority of users
 Minority of users
 cause network congestion
 cause problems for other users
 Example: K.U.Leuven KotNet
 Student network across region of Leuven
 20 000 active students
 5% of users caused 50% of used bandwidth
5
Introduction: problem
 Users are anonymous
 Only known by IP address
• Very easy to change IP address to be anonymous
 Everyone can (ab)use the network
 What to do if external complaints come in?
 User awareness is needed
 Let the user take responsibility of his own network usage
• Give the user a ‘personal credit’ he can use (network quota)
• Notify/block the user if his/her PC acts ‘strange’ and give instructions
 Answer: User authentication
 Makes it possible to map every action on the network to an
individual person
 Prevents unauthorized access
 Makes it possible to use ‘personal’ network settings and actions
6
Introduction: expectations
 Login system
 Each user must authenticate him/herself before using the network
 No extra software or configuration needed on client hosts
 Bandwidth regulation
 Works for all protocols and traffic
 Prevent that a minority of users take away all the bandwidth for the
majority of users
 Allow exceptions to certain (educational) sites
• E.g. OS security updates, e-learning site…
 Maximize responsiveness for interactive traffic
• E.g. Slow down bulk traffic, but don’t touch SSH unless really needed
 Every user and/or IP can have its own personal bandwidth settings
• E.g. Different settings for a lab computer and personal PC
 Distribute the individual bandwidth over the individual active
network connections
7
Introduction: expectations
 Volume quota
 Every user and/or IP is only allowed to use a certain fixed amount
of traffic
• Learns the user how to manage his Internet behavior
 Slow down traffic when a user and/or IP generates too much traffic
 Every user and/or group and/or IP can have its own personal quota
settings
• E.g. personal vs. lab PC, limited guest accounts...
 A user and/or IP is never blocked from the network (real-time
small band)
 If a user and/or IP who is on 'small band' stops downloading for a
few minutes, the user immediately can use a limited amount of
traffic again at normal speed.
8
Introduction: BCrouter solution
Why?
 Didn’t find another solution that fulfills all the
expectations
• No open source projects
• Commercial black boxes not really an option
• It’s interesting, fun and challenging 
 High performance needed
• Old quota/login system was maxed out
• Network usage still increases
9
Introduction: BCrouter solution
 Features
 User login system
• ‘Unlimited’ number of users
• Users can login multiple times at different location
 Group based routing
• ‘Unlimited’ number of user groups possible
• Every group has its own independent routing and policy
 Bandwidth regulation and volume quota
• Individual user/group and IP address based settings with no
performance impact
• Prevent network congestion by dynamically regulating maximum
bandwidth
• Powerful quota and bandwidth exception possibilities
 User friendly
• No user side configuration needed
• Nice user webpage with information and history information
• Automatically redirect to login site for login
10
Introduction: BCrouter solution
 Quota/bandwidth limiting to both user and IP
 Example 1:
• Assign user:
– Quota of 1 Gigabyte
– Refill the quota at rate of 1 Gigabyte/month
– Maximum speed: unlimited
• Assign IP:
– Quota of 10 Mbyte
– Refill the quota at rate of 5 Kilobytes/second
– Maximum speed: 20 Kilobytes/sec
• Result:
– User settings to determine the maximum volume a user can
download each month
– IP settings to limit the ‘real-time’ bandwidth usage
11
Introduction: BCrouter solution
 Quota/bandwidth limiting to both user and IP
 Example 2:
• Assign user:
– Unlimited quota
– Maximum speed: 50 Kilobytes/second
• Assign IP:
– Quota of 10 Mbyte
– Refill the quota at rate of 5 Kilobytes/second
– Maximum speed: 20 Kilobytes/sec
• Result:
– If a user logs in multiple times, the sum of all logins cannot
exceed the maximum user speed. The speed is divided across the
hosts that are logged in.
12
Introduction: BCrouter solution
13
Introduction: BCrouter solution
14
Introduction: BCrouter solution
15
Introduction: BCrouter solution
16
Introduction: BCrouter solution
17
Introduction: BCrouter solution
18
Introduction: BCrouter solution
19
Solution: components
Frontend
 Login server
 Redirect server
Backend
 User database server
 Log/History server
 “BCrouter” router
20
Solution: components
 Login server
 Serves secure web pages to the users
•
•
•
•
Login page
Statistics page
Technical information page
…
 Contacts the user database server for validating user
accounts
 Contacts the history server to gather historical
information about logins and/or quota
 Contacts BCrouter to check current quota and/or login
status and performs login/logout
21
Solution: components
 Redirect server
 Redirects HTTP requests to the login page on the Login
server
 Gets all the traffic that requires a login from nonlogged-in hosts
 Redirect done by a webpage (not TCP level)
 Separate dedicated host because can get DoS
 Real time network anomaly detection
• Detect virus/worm before login… even for 1st time users
• Coupled to automatic user blocking system
22
Solution: components
User database
 Contains all known users
 Contacted by the login server
 Can be any type of server
•
•
•
•
LDAP
Radius
Custom type of authentication
…
23
Solution: components
Log/history server
 Receives logs from BCrouter
 Parses received log files
 Store processed information in a database
• Historical login information
• Historical account information
 Database contacted by the login server
 Possibility to use data mining techniques to
detect suspicious user behavior
24
Solution: components
BCrouter
 Implements the core functionality
 Linux based solution
 Sends detailed quota reports and issued
commands to the log server
 Contacted by the login server
• Get quota information about user and/or IP
• Get login status of user and/or IP
• Perform login and logout operations
25
Solution: internet router setup
Assumptions
 A few 1000’s of users
• Limit by log/history server
 Manage the internet connection
 Auto redirect to login website
 Minimize the used Internet bandwidth
26
Solution: internet router setup
Internal backbone
network
User
database
Log/History
server
Internal
management
network
BCrouter
Login server
Redirect
server
Web cache
NAT
Firewall
Internet
27
Solution: main router setup
Assumptions
 A few 1000’s of users
• Limit by log/history server
 Manage the entire network
 Auto redirect to login website
 Central DHCP server is used to distribute IP
addresses
 Minimize the used Internet bandwidth
28
Solution: main router setup
Internal net
Internal net
Internal net
User
database
Log/History
server
Internal
management
network
Internal net
BCrouter
Login server
Redirect
server
Web cache
NAT
Firewall
DHCP
Internet
DNS
29
Solution: setup remarks
 Webcache and NAT are between BCrouter
and Internet
 BCrouter needs to ‘see’ the user IP address
 Otherwise not possible to make user and IP distinction
 Advantage:
• Transparent web caching is possible
 Disadvantage:
• Cached contents are also accounted and speed limited
30
Solution: integration
Suitable for each network?
• Ethernet based networks
• BCrouter does not support any routing protocols
(RIP,EIGRP…)
• BCrouter can also play a Cisco Netflow probe
• High performance
– Gigabit speeds with dual CPU system
• Redundancy (still in development)
– Possible to have backup BCrouter in hot standby
31
Solution: integration
 Scalability
 BCrouter server
• Supports virtual unlimited users
– Tested up to 50 000 users (1 Gigabyte RAM)
– Handles up to 60 000 login/logout operations per second
• Supports virtual unlimited IP addresses
– Tested up to 200 000 IP’s (1 Gigabyte RAM)
• Supports up to 300 000 packets/sec (1.5 Gigabit)
– Dual Xeon 3.6Ghz
 Clustering (Not yet implemented)
• Possible to use multiple BCrouter servers
– Each server handles a part of the given network segments
– Inter-BCrouter communication to exchange quota changes
32
Solution: integration
Quota/bandwidth exceptions?
 Yes… very powerful exception capabilities
 Exception flags
•
•
•
•
•
IP speed limit
User speed limit
IP accounting
User accounting
No login required
 Exceptions can be made for hosts or even entire
networks (both local and/or internet)
33
Solution: integration
 Quota/bandwidth exceptions examples:
 Default:
• Login required
• Accounting to both user and local IP
• Obey both user and local IP speed limits
 Local host A does not have to login to access the Internet, but still
uses IP quota and speed settings
• E.g. Embedded device that can’t login and needs network access
 Traffic from Internet host B is always possible from any local host
and is never accounted, but local host IP speed limits are obeyed
• E.g. Website with security patches
 Any combination of exception flags is possible in
either direction for any host/network
34
Solution: security considerations
 Account abuse
 Example
• User A powers off his PC without logging out
• Malicious user X takes IP of user A
• X continues to work with credentials of user A
 Solution: Auto logout
• Possibility 1: BCrouter performs logout after X minutes of inactivity
• Possibility 2: Ping probes
• Possibility 3: DHCP server
– Login server checks if IP that wants to login has been issued by the
DHCP server. Refuse login with static IP
– Use very short DHCP lease times (e.g. 15 minutes)
– Run script every few minutes that logs out inactive DHCP leases
 DHCP based auto-logout is preferred
35
BCrouter: introduction
Let’s take a look at the core element:
BCrouter




Components of BCrouter
Commands and logging
Routing and Netfilter setup
Quota/Bandwidth exceptions
36
BCrouter: introduction
Internal net
Internal net
Internal net
User
database
Log/History
server
Internal
management
network
Internal net
BCrouter
Login server
Redirect
server
Web cache
NAT
Firewall
DHCP
Internet
DNS
37
BCrouter: components
‘open’ black box
Linux operating system
 User space
•
•
•
•
DHCP forwarder
Syslog daemon
BCrouter daemon
Network configuration script
 Kernel space
• BCpolicer module
38
BCrouter: components
User space
DHCP
forwarder
BCrouter
daemon
Kernel space
Syslog
daemon
Management
interface
Kernel logging
Netfilter
framework
Input
interfaces
BCpolicer
Output
interfaces
39
BCrouter: components
 DHCP forwarder
 Forward broadcast DHCP DISCOVER to a central
DHCP server
 Dhcp-fwd
• http://www.nongnu.org/dhcp-fwd/
 Very simple application
 User space application running in chroot jail
 Listens in ‘promiscuous mode’ on specified interfaces
40
BCrouter: components
Syslog daemon
 Send logs to a remote log server for remote
processing
 Syslog-ng
• http://freshmeat.net/redir/syslog-ng/10178/url_homepage/syslog-ng
 Very powerful options (filtering, multi
logserver…)
 Logs both user space as kernel logs
41
BCrouter: components
BCrouter daemon
 Provides a network-based console to the
BCpolicer kernel module




Simple Perl script (Forking TCP server)
Allows simultaneous management access
Listens on a network socket (telnet port 23)
Communicates with the kernel module
42
BCrouter: components
Network configuration script
 Provides entire interface, routing and Netfilter
configuration and setup
 Shell script
 Executed at boot time
43
BCrouter: components
BCpolicer kernel module
 Receives login/logout commands and performs
accounting and routing decisions
 Core element of BCrouter (ipt_bcpolicer)
 Works entirely in kernel space
 Loadable module which implements an iptables
target
44
BCrouter: commands & logging
 Commands
 Login/logout
• login [username] ip [x.x.x.x] reason [text]
• logout [username] reason [text]
• logout [ip] reason [text]
 Query information
•
•
•
•
show user ip [x.x.x.x]
show ip user [username]
show quota ip [x.x.x.x]
show quota user [username]
 Configuration
• conf ip …
• conf user …
• export all
 Miscellaneous
• show uptime
45
BCrouter: commands & logging
 Commands example
bcrouter1# export all
bcrouter1# login user kuleuven/u0022948 ip 10.91.91.1 reason login demo
200 OK: login - 1142031358930300 - kuleuven/u0022948 (1) on 10.91.91.1 (login demo)
bcrouter1# show ip user kuleuven/u0022948
204 OK: show ip user - 10.91.91.1
bcrouter1# show user ip 10.91.91.1
203 OK: show user ip - kuleuven/u0022948
bcrouter1# login user kuleuven/u0022948 ip 10.91.91.2 reason 2nd login
200 OK: login - 1142031429848045 - kuleuven/u0022948 (1) on 10.91.91.2 (2nd login)
bcrouter1# show ip user kuleuven/u0022948
204 OK: show ip user - 10.91.91.2,10.91.91.1
bcrouter1# export all
conf user kuleuven/u0022948 ….
conf ip 10.91.91.1 …
login user kuleuven/u0022948 ip 10.91.91.1 reason recovering statefull info
conf ip 10.91.91.2 …
46
BCrouter: commands & logging
Logging
 Log commands and responses
 Log network/host statistics
network segment




Time of log
Log sequence number
Name of segment
Traffic counters
• Bytes and packets
• Download and upload
• Accounted and not
accounted
• Dropped and accepted
 Number of active IP’s
host




Time of log
Log sequence number
IP address
Username (–none- if no
login)
 Traffic counters
• Bytes and packets
• Download and upload
• Accounted and not
accounted
• Dropped and accepted
47
BCrouter: routing & Netfilter
Routing with BCrouter is done by a BCPOLICER
target in the PREROUTING mangle table that
alters the fwmark value of the packet and uses
this value as selector for policy based routing.
48
BCrouter: routing & Netfilter
Use Linux networking capabilities
 IEEE 802.1Q support (VLAN technology)
• Used to limit the number of physical interfaces
 Policy based routing (Routing rules)
• Used for implementing user groups
 Netfilter/Iptables framework
• Used for host exception lists
49
BCrouter: routing & Netfilter
 IEEE 802.1Q support
 Virtual LAN technology (VLAN)
 Operates on the data link layer (OSI layer 2)
• Adds 4 extra bytes to existing ethernet header
 Allows multiple LAN’s over 1 physical wire (trunking)
VLAN 1
VLAN 2
VLAN 3
VLAN
enabled
Device
dot1Q ‘trunk’
containing
VLAN 1,2,3
VLAN
enabled
Device
 Each VLAN id has its own interface device
• E.g. eth0.5 indicates VLAN id 5 on physical interface eth0
 ‘vconfig’ tool
50
BCrouter: routing & Netfilter
 Policy based routing (Routing rules)
 Multiple routing tables
• Routing policy database (RPDB)
• Each entry in the database:
– contains a routing table
– has a priority number
– has a selector (src addr, dst addr, incoming iface, tos, fwmark)
• RPDB entries are scanned according priority
• If the selector of RPDB entry applies, that routing table is used
 ‘ip rule’ tool
51
BCrouter: routing & Netfilter
 Netfilter/Iptables framework
 Extensive IP packet filtering rules
 There are 3 ‘tables’
• Each table has its own purpose
 Each table contains several ‘chains’
• Each chain operates on a different location in the network
packet flow
• Chains can also be custom-defined
 Each chain can contain several ‘rules’
• Each rule contains a selector and a target
• Rules are examined in the order they are listed in the chain
• If a selector matches, the action is determined by the target
 ‘iptables’ tool
52
BCrouter: routing & Netfilter
 Filter table: Firewall type rules
 INPUT: for packets destined for the box itself
 FORWARD: for packets being routed through the box
 OUTPUT: for locally generated packets
 NAT: Native Address Translation type rules
 PREROUTING: for altering packets as soon as they come in
 OUTPUT: for altering locally generated packets before routing
 POSTROUTING: for altering packets as they are about to go out
 Mangle: Packet alteration type rules





PREROUTING: for altering packets before routing
OUTPUT: for altering locally generated packets before routing
INPUT: for altering packets destined for the box itself
FORWARD: for altering packets being routed through the box
POSTROUTING: for altering packets as they are about to go out
53
BCrouter: routing & Netfilter
Netfilter framework
I
F
A
C
E
PREROUTING
Input
Routing
FORWARD
INPUT
Output
Routing
POSTROUTING
OUTPUT
User space
54
I
F
A
C
E
BCrouter: routing & Netfilter
 Packet filtering and routing are normally 2
different things
 However:
 There is a packet ‘property’ that can be changed by an
iptables target and can be used as a routing rule
selector: fwmark
 Fwmark




Netfilter MARK value
Integer value
Associated within the kernel with the packet
Does not alter the packet itself
55
BCrouter: routing & Netfilter
 Routing with BCrouter is done by a
BCPOLICER target in the PREROUTING
mangle table that alters the fwmark value of
the packet and uses this value as selector for
policy based routing.
 Fwmark value after BCPOLICER selects
routing table:
 1 … 97 : routing table belonging to the user group of
the identified user
 98: routing table for non-identified users
 99: routing table for redirecting users to the login site
 100: routing table that drops all packets
56
BCrouter: routing & Netfilter
I
F
A
C
E
PREROUTING mangle table
Selector
Target
.
Exception1
MARK ex.flag1
Exception2
MARK ex.flag2
…
Everything
BCPOLICER
Fwmark changed by
BCPOLICER to select
correct routing table
Quota/bandwidth
exception list changes
fwmark before entering
BCPOLICER
Input routing
Priority Selector
1
fwmark=1
2
fwmark=2
3
fwmark=98
4
fwmark=99
5
fwmark=100
Routing table
Group1
Group2
NoGroup
Redirect
Drop
I
F
A
C
E
57
BCrouter: exceptions




Exceptions passed to BCPOLICER via fwmark value
Possible to use almost all Iptables selection features
List before BCPOLICER target in PREROUTING
Fwmark value
•
•
•
•
•
IP speed limit (src: 0x0001 dst: 0x0002)
User speed limit (src: 0x0004 dst: 0x0008)
IP accounting (src: 0x0010 dst: 0x0020)
User accounting (src: 0x0040 dst: 0x0080)
No login required (src: 0x0100 dst:0x0200)
 Final value: logical OR of the wanted flags
 Example
 Traffic from Internet host B is always possible from any local host
and is never accounted, but local host IP speed limits are obeyed
 Src: localnet Dst: hostB mark: 0x0001||0x0100 = 0x0101
 Src: hostB Dst: localnet mark: 0x0002||0x0200 = 0x0202
58
BCpolicer: introduction
Everything up until now was on
macroscopic level
Let’s dig (a little bit) into the real core:
ipt_bcpolicer
The next slides are greatly simplified
 Would take couple of hours to completely
explain the entire design
59
BCpolicer: introduction
TCP/IP
 TCP: reliable protocol
• Every received packet must be acknowledged to
Sender by Receiver
• Sender retransmits when no acknowledgement
received within certain time
• Exponential delay between retransmits
• When relatively few packets are dropped,
transmission speed goes down significantly
60
BCpolicer: introduction
 Policing vs. Shaping
Policing
Accept/drop packets
Don’t delay packets
No queues
No scheduling
Shaping
Don’t drop packets
Delay packets
Queuing
Scheduling
61
BCpolicer: design principles
 Design started by determining the specifications of an
‘ideal’ bandwidth/quota system (with no user login)
 Different settings for each individual network segment
 For each network segment:
• Independent up/download settings
• Distribute available bandwidth across active hosts, favoring lowtraffic volume hosts a little
• Dynamically regulate maximum bandwidth per host to allow
maximum bandwidth usage without overloading the segment or
uplink
 For each host:
• Distribute host bandwidth equally over active connections
• Tendency to favor interactive traffic above bulk traffic
• Enforce traffic quota by dynamic bandwidth speed regulation
62
BCpolicer: design principles
Result
 General principle: leaky bucket system
 Lots of modifications and tweaks required
MeanFillRate
TokenBucket
TokenBucketSize
TokenBucketMaxSize
CurrentRate
(0…BurstRate)
POLICER
63
BCpolicer: design principles
 Example:
 Download quota: 4GB/month
 Maximum download speed: 50KByte/sec
 DownloadTokenBucketMaxSize
• = 4GTokens
 DownloadBurstRate
• = 50KTokens/sec
 DownloadMeanFillRate
• Must fill up an empty TokenBucket in 30 days
• =4GTokens/(30days*24hours*60minutes*60seconds)
• =1657 Tokens/sec
64
BCpolicer: design principles
Policer summary
 Multi exponential stream policer
• ‘smooth’ policing
• Distribute individual host bandwidth equally over its
active connections
• Tendency to favor interactive traffic above bulk
traffic
65
BCpolicer: design principles
Specifications list
 For each host:
• Distribute host bandwidth equally over active
connections
– OK : Multi logarithmic stream policer algorithm
• Tendency to favor interactive traffic above bulk
traffic
– OK : Multi logarithmic stream policer algorithm
• Enforce traffic quota by dynamic bandwidth speed
regulation
– OK : Accurate quota control because there are no Tokens
created or destroyed in the bucket system
66
BCpolicer: design principles
 Specifications list
 Different settings for each individual network segment
• OK : Each IP has its own buckets with its own settings
 For each segment:
• Independent up/download settings
– OK : Use separate buckets
• Distribute available bandwidth across active hosts, favoring
low-traffic volume hosts a little
– To be done
• Dynamically regulate maximum bandwidth per host to allow
maximum bandwidth usage without overloading the local
segment or uplink
– To be done
67
BCpolicer: design principles
 Prevent network congestion:
 Implement a ‘valve’ between TokenBucket and the
Policer
• If the ‘valve’ is fully open:
– Maximum host speed is BurstRate
• If the ‘valve’ closes:
– Slow down maximum host speed
• ‘valve’ control
– If no network segment congestion: fully open
– Network segment congestion: close valves for the local IP’s that
cause the congestion until no congestion anymore
• ‘valve’ is named ‘RateFactor Correction’
• The state of the valve is called RateFactor
68
BCpolicer: design principles
Host
BCrouter
BCpolicer
Feedback
algorithm
Internet
69
BCpolicer: design principles
MeanFillRate
TokenBucket
TokenBucketSize
TokenBucketMaxSize
CurrentRate
(0…BurstRate)
RateFactor Correction
RateFactor
(e.g. <1)
POLICER
70
BCpolicer: policing alternatives
 Advantage of policing
 Simple to implement
• No queuing
• No scheduling
 Very fast
 Drawbacks of policing
 Policing overhead problem
• Relative problem:
– Suppose unlimited stream would be 100Kbyte/sec
– Limit to 10Kbyte/sec
– 10 to 15% overhead -> receiving from the Internet 12KByte/sec
• More overhead on low latency, high bandwidth connections
– E.g. Limit to 10KByte/sec; receiving 20KByte/sec
71
BCpolicer: policing alternatives
 ‘TCP Window Size’ alteration
 TCP Window Size
• Field in TCP packet header
• Message from Receiver to Sender
• Indicates to Sender how much data can be sent to Receiver
 Should decrease retransmission overhead
• Avoid fast retransmission on fast connections
 Alters packets passing BCrouter
• Possibly bad and unforeseen consequences?
 No need for queuing
 Integration in the BCpolicer scheme
• Use the DelayTime value to decrease the window size
72
BCpolicer: policing alternatives
Shaping
 Alter the Round Trip Time of a connection
• ‘simulate’ slow connection
• TCP will try to adjust by slowing down
 Extra load and complexity due to delay queue
management
 Integration in the BCpolicer scheme
• BCpolicer already works with delay times
• Instead of accepting or dropping, put packet into a
delay queue for the delaytime
73
BCpolicer: complete design
Up until now, we only focused on the
bucket design
Complete system
 Addition of user buckets
• Same principle as IP buckets
 Addition of exception handling
• Makes program flow complex
74
BCpolicer: main flow
No
Src IP kotnet?
Start
Dst IP kotnet?
Yes
Yes
Clear src
streamupdate flag
No
srcnlr=
1
No (lr)
Src
logge
d
in?
Yes
srcnua=
1
and
srcnus=
1 No
Src IP Bucket
calculation
Clear dst
streamupdate flag
Yes (nlr)
No
Yes
srciac!=0
or
srcisl!=0
Src userbucket
Verdict (UV1)
Yes (nlr)
dstnlr=
1
No (lr)
Dst
logge
d
in?
Yes
dstnua=
1
and
dstnus=
No 1
No
Yes
Src ipbucket
Verdict (IV1)
REDIRECT
Return
IPT_CONTINUE
No
Dst userbucket
Verdict (UV2)
Yes
dstiac!=0
or
dstisl!=0
Yes
Dst ipbucket
Verdict (IV2)
DROP
Mark
translation
Final Verdict
&
Accounting
&
Logging
75
No
BCpolicer: bucket verdict
Bucket
calculations
Entrypoint
No
Respect
Speedlimit?
Yes
Yes
PBS>PTT+PktSize
No
No
PBS>PktSize
Yes
No
Verdict=DROP
TBSPktSize>0
Calculate StreamII and get the
correct SAPS and SPLAT for this
packet
Yes
QSAPS=32- ((32*SAPS) / MPS)
QDelayF=64- ((64*PBS) / PTT)
PDT=DelayMatrix(QSAPS,QDelayF)
Verdict=ACCEPT
No
SPLAT+PDT<PacketTS
Yes
Store StreamII and new SAPS (SAPSnew)
Set streamupdate flag for the current ip
(used for accounting)
Verdict=DROP
Return Verdict
Verdict=ACCEPT
Verdict=ACCEPT
76
BCpolicer: bucket calculation
Entrypoint
TB_ADD_INTERVAL
passed since the
last call?
No
Yes
Calculate new TBS
TBS=max( TBS+TBMFR*TBLAT,TBMS)
Calc max CurrentR
EffectiveR=max CurrentR
PBS+EffectiveR<PBMS
No
Yes
Calc max EffectiveR
CurrentR = max EffectiveR
(CurrentR = = EffectiveR)
TBS=TBS-CurrentR
FactorR=0
PBS=PBS+EffectiveR
Endpoint
77
BCpolicer: f.verdict&acc&log
DROP
verdict in
UV1 || IV1 ||
UV2 || IV2?
Entrypoint
No
REDIR
verdict in
UV1 || IV1 ||
UV2 || IV2?
Yes
FinalVerdict=DROP
Yes
Src IP kotnet?
No
No
Yes
FinalVerdict=REDIR
If
PktTime>SITLL+LOGSI
If
PktTime>SSTLL+LOGSS
Print Src IP logline
&
clear printed counters
Print Src Segment logline
&
clear printed counters
FinalVerdict=ACCEPT
SOURCE IP
Accounting Calculation
No
Yes
Dst IP kotnet?
No
If
PktTime>SITLL+LOGSI
If
PktTime>SSTLL+LOGSS
Print Dst IP logline
&
clear printed counters
Print Dst Segment logline
&
clear printed counters
FinalVerdict
=
REDIR?
Yes
DESTINATION IP
Accounting Calculation
Return FinalVerdict
78
BCpolicer: accounting
Entrypoint
FinalVerdict
=
DROP
No
Yes
SSTAB + = PktSize
SSTAP++
SSTAI++ if needed
streamupdate=1
SAPS at SII=SAPSnew
No
Yes
Yes (no uac)
nua=1
No (uac)
SSTDB + = PktSize
SSTDP++
SSTDI++ if needed
SSTAI++ if needed
Yes (no uac)
SINFAB + = PktSize
SINFAP++
SIFAB + = PktSize
SIFAP++
No (uac)
Yes (no usl)
nua=1
nus=1
UsrTBS - = PktSize
SIFDB + = PktSize
SIFDP++
UsrTBS += PktSize
No (usl)
UsrPBS - = PktSize
SINFDB + = PktSize
SINFDP++
No (no iac)
iac=1
Yes (iac)
IpTBS += PktSize
No (no isl)
IpTBS - = PktSize
Return
isl=1
Yes (isl)
IpPBS - = PktSize
79
Case study: KotNet
Goals
 Connect K.U.Leuven association students and
personnel to the campus network and Internet
from their homes
 Enhance possibility of study and research in an
academic environment
 Low entrance fee and costs
KotNet is an enabler for the E-university
80
Case study: KotNet
 History
 1994: Started by students in their student home
 1996-97: pilot & test projects
 1997-98 en 1998-99
• 1°, 2° operational phase K.U.Leuven residences
• access via cable TV network
– test project in 1997-98
– operational since Sept. 1998 (2 cable TV operators)
 2002 : Wireless KotNet at public places
• Test project in student restaurants
• Gradually extended to libraries, meeting rooms, classrooms
 2004 : Agreements with associated partners
81
Case study: KotNet
Providers
 K.U.Leuven – direct fiber to university
residences
 K.U.Leuven – wifi hotspots in student
restaurants, auditoria, ...
 UPC Belgium – via cable modem
 Telenet/Iverlek – via cable modem
82
Case study: KotNet
 Technology
 Layer 2
• VLAN's are used to implement city-wide networks
– connect remote buildings to centralized KotNet routers
• spanning-tree for automatic redundancy
 Layer 3
• use of private address space (10.*.*.*)
• split data streams
– traffic to K.U.Leuven networks is routed through the central
firewall
– HTTP traffic is redirected to a web cache cluster
– other traffic is handled by a Network Address Translator
cluster
83
Case study: KotNet
 Policies
 Users will consume mega-amounts of bandwidth if you
let them. Not all of their traffic is of an educational
nature...
• implementation of policies and enforcement of these policies is
absolutely essential
 p2p traffic is not allowed (Gnutella, Kazaa, eDonkey, ...)
• enforced by iptables filters on the NAT servers
 individual servers are not allowed
• enforced by a 200 Mbyte/day upload limit
• exceptions are possible for registered servers
 Internet bandwidth is not free
• enforced by a 4 Gbyte/month/USER download limit
84
Case study: KotNet
 Residences
 Approx 50 buildings
 Approx 4000 connections
 Spread geographically across the city of Leuven
• K.U.Leuven owned fiber network
 Each building is a different VLAN
 Ethernet network
• Old :
– 10Mbit HUB
– 10Mbit Half-duplex fiber uplink to KotNet backbone
• New :
– 100Mbit Switch (Cisco 2950)
– 100Mbit Full-duplex fiber uplink to KotNet backbone
85
Case study: KotNet (residences)
Old Network
10Mbit HUB
New Network
100Mbit Switch
10Mbit FOT
10Mbit FOT
Aggregation Switch
Gigabit 802.1Q VLAN trunk
KotNet
Backbone
Network
86
Case study: KotNet
Cable provider Telenet




Coverage across the region of Leuven
Approx 1500 cable modems
Approx 5000 users
3 Cable segments
• 13 Mbit downstream capacity each
• 3 Mbit upstream capacity each
• Single VLAN (flat network)
87
Case study: KotNet (Telenet)
Cable segment
Cable Router (L3)
100Mbit FOT
Telenet location
100Mbit Flat network
100Mbit FOT
KotNet
Backbon
e
88
Case study: KotNet
Cable provider UPC
 Coverage across the region of Leuven and
Brussels
 Approx 5100 cable modems
 Approx 15000 users
 52 Cable segments
• 50 Mbit downstream capacity each
• 3 * 10 Mbit upstream capacity each
• Each cable segment is a different VLAN
89
Case study: KotNet (UPC)
52 Cable segments
Cable Head end (L2)
UPC location
Gigabit 802.1Q VLAN trunk
KotNet
Backbon
e
90
Case study: KotNet
KotNet Backbone
 Gigabit Ethernet based
 Redundant where possible
• spanning-tree protocol
 Separated from K.U.Leuven backbone
 Spread geographically across the city of Leuven
91
Case study: KotNet
Gigabit 802.1Q VLAN trunk
containing 105 KotNet VLAN’s
KotNet Backbone
-Residences
-Telenet
-UPC
Monitor switch
BCrouter
Flow logger
Gigabit mirror port
Redirect
Cisco WCCP
KUL-KotNet
Firewall
KULeuven
network
Web cache
KUL-INET
Firewall
NAT
NAT
K.U.Leuven Association
Internet
92
Case study: KotNet
Pre-BCrouter
 2 High-end Cisco 7500 routers
• 30.000 EUR each
• Couldn’t handle large network segments
– Long user access lists requiring lots of CPU power
• Policy based routing not really flexible
• Logout operation very expensive (ACL reload)
• Inflexible shaping
– Not possible per ip/user/stream/usergroup and certainly
not everything together
 Need for more scalable solution!!
93
Case study: KotNet
 BCrouter
 Replaces both Cisco routers
 Hardware:
• Dell Poweredge 2650
• Dual Intel(R) Xeon(TM) CPU 3.20GHz with Hyperthreading
• ServerWorks GC-LE chipset, 400MHz front side bus, 2:1
memory interleaving, 5 PCI buses (three of which are are PCIX capable)
• 1 Gig ECC ram
• Two Intel E1000 PCI-X 133Mhz Network interfaces
• 3000 EUR
 Extremely powerful and flexible configuration
94
Case study: KotNet
 BCrouter settings
 Only 1 simultaneous login allowed by login system
 Normally: login required, obey user/IP quota/speed
 User buckets
• Download quota of 4Gbyte/month
– Max speed (BurstRate): none
• Upload quota of 200Mbyte/day
– Max speed (BurstRate): none
 IP buckets
• Download quota of 8Gbyte/month
– Max speed (BurstRate): 250Kbyte/sec
• Upload quota of 8Gbyte/month
– Max speed (BurstRate): 250Kbyte/sec
95
Case study: KotNet
 BCrouter exceptions
 DHCP, DNS server
• No login required, no user acc/bw, ip acc/bw
 Login website reachable
• No login required, no user acc/bw, ip acc/bw
 Local antivirus download website reachable
• No login required, no user acc/bw, ip acc/bw
 Local FTP server reachable
• No login required, no user acc/bw, ip acc/bw
 Unblock website reachable
• No login required, no user acc/bw, ip acc/bw
 E-learning website reachable
• Login required, no user acc, user bw, ip acc/bw
96
Case study: KotNet
User blocking system (admin view)
 Receives information from number of probes
• Email system
• Flowlogger
• Local honeypots
 Gets an IP address
•
•
•
•
Looks up the corresponding user
Logout user
Put user in a blacklist to prevent user login
Send email to the user
97
Case study: KotNet
 User blocking system (user view)
 User is logged out
 Receives email containing instructions
• Webmail is always reachable
• Local antivirus download site also reachable
 If the user tries to login:
• Message his account is blocked
• More detailed instructions about the blocking reason
• Link to unblock website
 User unblocks himself
• Only allowed 1 time each day
98
Case study: KotNet
 Flow Logger
 Emits Cisco Netflow data to logging system
• Very detailed traffic information (for each stream):
–
–
–
–
–
Start and stop time
Source and destination IP
Source and destination port
Number of bytes transmitted/received
Number of packets transmitted/received
 High performance Netflow probe:
• nProbe (http://www.ntop.org)
 Used to detect network anomalies
• Host scans and port scans
• Coupled to automatic user blocking system
99
Case study: KotNet
Email
 No direct SMTP connections to the internet
 All emails must pass the Central Anti Virus
cluster
• Scans for viruses (Mcafee Antivirus)
• Performs spam classification (Spamassassin)
• If a virus is detected, user is blocked
– User blocking system looks at sending IP
100
Case study: KotNet
Transparent caching
 No user settings required to use caching
 Reduce the traffic on the Internet link
 Cisco Content Engine
• Squid based
• WCCP
– ‘transparent’ redundancy
• Needed a quick solution
101
Case study: KotNet
Firewalling
 Done on BCrouter
• Beginning of PREROUTING mangle table
 TCP port 135,139,445 are blocked for anyone
• Prevent virus outbreaks
102
Case study: KotNet
P2P blocking
 Done on NAT cluster
 Do a pattern search in every packet
• Look for certain P2P strings
– Kazaa, Bittorrent, Edonkey
• Iptables string module
• Rather CPU expensive
• Planning to make a better version someday 
103
Case study: KotNet
 Transition to BCrouter
 Phased transition
•
•
•
•
•
•
•
•
Created 2 testing KotNet segments on BCrouter
Test connectivity and performance
Modify login scripts and backend for BCrouter
Convert 2 real KotNet segments to BCrouter
Convert all residences to BCrouter
Convert all UPC to BCrouter
Convert all Telenet to BCrouter
Parallel volume accounting on user webpage
– Real-time counters from BCrouter
– History information by previous KotNet accounting system
 No real problems encountered
104
Case study: KotNet
 Traffic impact (early testing)




User mode test version of BCpolicer
No HTTP traffic
Most popular P2P applications blocked
Only used IP addresses for accounting
• PolicerBucketMaxSize: 150 Mbyte
• MeanFillRate: 50 Kbyte/sec
• BurstRate: 250 Kbyte/sec
 Results:
• 50% reduction in traffic
• Only 5% of users ‘hit’ BCpolicer
 Users noted improvement in network quality
105
Case study: KotNet
Some Numbers
 Current daily numbers
• 400 Mbit/sec throughput total (18u – 23u)
– 150 Mbit Non-http
– 30 Mbit K.U.Leuven
• 80 Kpackets/sec
• Up to 20 000 simultaneous users
 BCrouter load
• 33% CPU at peak
106
Case study: KotNet
Management utilities
 The next few slides show the web-based
information interface
107
Case study: KotNet
108
Case study: KotNet
109
Case study: KotNet
110
Case study: KotNet
111
Case study: KotNet
112
Development: current status
BCrouter itself is functioning
 Design is done by specifications
 Implementation of ipt_bcpolicer is not yet
100% ready
• Dynamic bandwidth regulation by congestion is not
yet implemented (Fixed RateFactor of 1)
Login system is not self-contained
 Currently uses the previous login system from
KotNet
113
Development: future
 Redundancy
 Timeframe: June 2006
 Hot-standby backup system which takes over if primary fails
 Further development needed
 Build a stand-alone login/log/history system






Timeframe: September 2006
Easy custom layout
Minimal dependencies
Already have some proof of concept code
Can be done independently of further development
Login system without history information can be done very
quickly
 Planning: hire a job student for this during the summer
114
Development: future
 Black Box solution
 Timeframe: Jan 2007?
 Depends on stand-alone login system
 Packaging and generalizing scripts
• Packaging of BCrouter partly done
 TCP window size policing
 Timeframe: unknown
 Still requires research
 Planning: make a thesis out of this
 FactorRate Correction
 Timeframe: unknown
• Not an issue anymore on KotNet
 Implementation and testing
115
Development: wish list
Transform the policer into a shaper
 Probably more work than expected
 Planning: make a thesis out of this someday
Add the automatic blocking features
 Currently uses the previous KotNet solution
116
Conclusion
BCrouter concept offers great
possibilities
 User authentication
 Advanced and flexible quota and bandwidth
management
 High performance and scalable solution
BCrouter still requires a lot of
implementation
 Concepts are already tested in the field
117
THE END
Contact information:
Dirk Janssens
ICTS K.U.Leuven
[email protected]
118