Transcript Optimising ASP/ISP Interconnections, Panos Gevros

Optimising ASP/ISP Interconnections
Panos Gevros
University of Cambridge
Computer Laboratory
[email protected]
TAPAS project meeting - Dortmund, Germany
10 February 2003
outline
Application & Internet Service Providers issues
•
similarities and differences in their operations, optimisation goals
The network environment
•
topology, domains, routing, interconnection types, performance
implications
Interconnection choice : a common ISP & ASP problem:
•
Selecting which ISP(s) to interconnect with (involves facility location
and ISP assessment aspects)
Network design : an ISP-specific problem:
•
delay analysis and capacity provisioning
background & basic facts
•
•
•
•
•
For ASP/ISP network performance is crucial for delivering high quality
services (offer outsourcing, move bits around).
In network engineering there are two basic philosophies for
introducing high performance (low delay, high bandwidth) Internet
services, namely :
•
QoS / service differentiation i.e. giving preferential treatment
to selected traffic flows using router mechanisms and/or new
service models (Intserv, Diffserv).
•
bandwidth provisioning i.e. provide enough bandwidth so that
the most stringent delay requirements are being met without
differentiation between users.
It is unlikely that a new QoS service model will replace the existing
best-effort one
For some time now the major players in the area base their
operations on intelligent decisions about provisioning and
interconnection of their networks
However, the landscape in the area is far from clear…
Comparing ASP and ISP operations
•
An ASP should
• connect to ISPs so that application service performance, fulfils the terms
specified in the SLAs it has with its customers (in the absence of SLAs the
perceived performance should be considered satisfactory).
• establish presence in network places which are “close” to target customers
(same metropolitan area, exchange etc.)
•
The difference between an ISP and an ASP is that the ISP operation is
generally more broad,
• ISP customers are not only end-users, but also other network providers,
• the ISP may act as a customer w.r.t some ISPs and as a provider w.r.t to others
•
•
•
An ISP and an ASP are similar in that their operations depends on the
relationships they establish with other ISPs.
An ISP in many cases operates as an ASP by bundling with the network
connectivity offering, infrastructure services like DNS, Mail Relay, firewall, webcaching etc.
The ISP operation goals will become clear after we discuss the interconnection
environment.
The interconnection environment
• Macroscopic view of the network
Abstract view as 3-tier hierarchy of providers
Backbone (Tier1) (e.g UUNET, C&W, Sprint)
Regional (Tier2) (national backbones, part leased from Tier1’s)
Access (Tier3) (e.g. AOL)
• Brief description of BGP routing
• Types of relationships between ISPs
transit (“customer – provider”)
peering
• The role of Internet Exchanges (IX-es)
Basic definitions
• Internet : as an interconnection of Autonomous Systems (ASes):
• Autonomous System (AS): a collection of networks under a single
administrative authority distinguished by AS number (e.g an ISP is
associated with one AS number).
• Backbone: a network of geographically dispersed POPs (Points of
Presence) interconnected with high capacity links, operated by a single
administrative authority.
• Point of Presence (POP) : a physical location where a collection of
•
•
routers belonging to a single provider are used for the interconnection of
its customers to the wide-area backbone.
Route : an association between a destination network (network prefix)
in the Internet and a next hop router towards that destination. (~112,000
IP subnets in the global Internet routing table).
BGP : Border Gateway Protocol, routing protocol operating at the
borders of the providers’ networks (between neighbor AS-es) its
operation determines the routes (paths taken by the traffic flows).
A (very) brief description of BGP Operation (1/)
• BGP (Border Gateway Protocol) is a path vector protocol.
De-facto standard inter-domain routing protocol
• Operation: in-bound exchange of Route Advertisements between
border routers
• Route advertisement (rtadv) consists of
• Destination network prefix (e.g. 192.168.0.0/16),
• next hop router IP address,
• AS path info (list of AS#)
• If a border router of AS-x sends a route advertisement for
network N to a border router of AS-y this implies that AS-x
accepts to forward packets with destination address an address
in N on behalf of AS-y.
A (very) brief description of BGP Operation (2/)
• AS-path info has dual functionality,
• Loop detection
• Route selection (shorter paths are preferred)
• Inter-domain traffic engineering with BGP filters
• Input filters select which rtadv will be accepted (example policy: accept a
route advertisements if and only if the AS#s in the AS path info belong to
“trusted” AS-es)
• Output filters selects which routes are being advertised to its neighbors
• Decision process: applies (configurable) route selection criteria
• AS-path length, med, eBGP>iBGP, for tie-break keep oldest route)
• Selected route is installed in forwarding table (for each packet
does longest match prefix of its destination address and
indicates outgoing interface).
Transit and Peering : definitions (1/)
• Route advertisements are advertisements of value being offered
and they are at the heart of the contracts between ISPs.
Depending on the routes exchanged between two neighbor ISPs
there are two types of (bilateral) relationship:
• Transit is a “customer-provider” relationship between two ISPs
in which one ISP (the provider) sells to the other ISP (the
customer) connectivity to all its destinations that are available in
its routing table.
• Peering is the relationship in which the two ISPs mutually
provide connectivity to their customers (only) without any money
exchange.
Transit and Peering: example (2/)
Suppose ISP - A is buying transit from ISP-D
(denoted as A ->D) then ISP-D has the
obligation of carrying A’s traffic to/from any
Internet destination/source.
Let also ISP-A has a peering relationship
with ISP-B (A <p> B) and ISP-B in turn peers
with ISP-C (B <p> C).
Peering is a non-transitive relationship (i.e
the routes of C’s customers are visible to B
but they do not get propagated further to A,
so traffic originating from a customer of A
destined to a customer of C will not be
routed via B instead it will be routed via ISPD from which ISP-A buys transit. In the same
manner ISP-A’s routes do not get
propagated to ISP-C.
ISP C
ISP 1
ISP B
ISP A
Customers
Transit and Peering: economic facts (1/)
• Value and money flow in opposite directions
• All ISPs (except the Tier-1s) have to buy transit
• Transit fees constitute the main operational costs.
• Selling transit is necessary and desirable
– (increases the number of your customers and they are the ones who
pay)
• However, ISP cannot survive by re-selling transit alone,
– (the ISP becomes the unnecessary middleman which the customers
bypass and connect directly to its upstream)
• Can be a non-starter, depending on selling transit for value
growth, while value is a pre-requisite for attracting customers and
selling transit …
Transit and Peering: economic facts (2/)
• Reason why peering becomes important:
• it is an economic optimisation over buying transit, reduces
average per-bit costs.
• creates value growth (increases the amount of bandwidth the
ISP has to sell to its customers)
• Parties in a peering relationship perceive that the cost of peering
(fixed costs plus carrying peers traffic) is offset
• Tier-1s tend to peer only between themselves,
• do not want to commoditise IP – and prefer to compete on the
basis of “better service”
• do not have incentives to improve the performance of smaller
ISPs enabling them to compete against them in the market.
Peering (+)s
• Reduces transit costs
• buying transit costs 10ths or 100ths thousands $$/month
• Peering is low cost, provides direct traffic exchange, reduces the
load on expensive transit links.
• Reduces latency (end-to-end delay)
• Direct interconnection to a competitor ISP nearby provides lower
delay compared to traversing several transit providers.
• Increases demand for bandwidth
• High packet loss rates and high delays reduce bandwidth
consumption (adaptive TCP!) and therefore revenues. This can
be a problem (reducing revenues) with usage based charging.
Peering (-)s
• Peering is not free
– Router ports, circuits (to the exchange), man hours, cost money, lengthy
negotiations
• Assymetric traffic
– When Traffic ratio across peering boundary (outbound/inbound) exceeds
certain limits then the peering may no longer be seen as beneficial to one of
the parties involved (in practice ratios up to 4:1 can be found and still be
considered acceptable).
• Lack of SLAs
– Peering relationships unlike “customer” relationships are not followed by SLA
“guarantees” for reliability, rapid repair of problems, or penalty when a
connection performs poorly.
– Side-effect: determining the performance of an end-to-end path becomes
unpredictable when the path traverses several peering boundaries (public
internet exchanges) which tend to be congested.
Internet Exchanges (IXes) (1/2)
• Exchange is essentially the “marketplace” where ISPs peer or
buy/sell transit (first IXes were only for peering).
• IX-es offer cost savings by aggregating traffic from many
neighbours on one link (the link into the exchange). (downside
increased risk due to a single point of failure)
• Today IXes tend to differentiate towards transit-only or peering
only.
Internet Exchanges
(2/2)
• An IX it should satisfy certain criteria for ISP participation: access
issues, operational, cost, reputation, viability, neutral operation,
opportunities (for selling transit or for peering)
– Neutrally operated
• not aligned with specific carriers or ISPs which may impose restrictions on the
participants as to where they should be buying access circuits from etc.
– The cost of using the exchange
• Rack fees, port fees,cross connect fees, installation fees
– Reputation / Credibility / financial viability
• Will the exchange continue to exist in the future
• ISPs tend to be “risk averse” ; prefer established, well-populated
exchanges (and are willing to pay an “insurance” premium,
instead of landing in an untested IX)
• ISP min-conf :presence two transit and one peering IX-es
Service Provider Interconnection decisions
•
ASP should access the network so that its customers perceive a
service that is fast and reliable.
•
ISPs (Tier-2,3) face a somewhat similar problem.
•
Commonality : the importance of location it determines the quality
of
So the questions faced by both service providers (ISPs/ASPs ) are
I.
to which locations, POPs, should we buy links into (“where
should we land capacity “),
II.
which other ISPs should we interconnect with,
III.
under which terms we should be interconnecting with a certain
ISP.
No–single correct answer,
–
depend on provider; customer base, growth plans, budget limitations, (regulatory
framework in which it operates) etc.
General principles apply (usually)
The ASP interconnection
•
Depending on the services offered by the ASP there are
different strategies,
1.
2.
3.
•
Before choosing an ISP the ASP should evaluate the
candidate, criteria involve
•
•
•
•
Co-locate close to the customers
Select the same access ISPs as the target customer group
Act as an advisor, assess performance/connectivity offered by
different ISPs to the ASP site -- advise customers as to which ISPs
provide “better” access the service.
Network capacity
number of peering arrangements
information about peering policies (link capacities, upgrade
procedures, presence of SLAs)
Improve connectivity, reduce risk
•
•
multihoming (use >1 upstream ISPs)
use redundancy (have multiple links to the same ISP, possibly
using BGP policies for load balancing).
ASP interconnection
ASP
ISP
ISP
ISP
customer
ISP
ISP
ISP
customer
ISP
ISP
ISP
solution 1: strategic co-location
(possibly) VPN
ASP
ISP
ISP
ASP
ISP
ISP
ASP
ISP
ISP
ISP
customer
solution 2 :common access ISPs
ASP
ISP
ISP
ISP
ISP
ISP
ISP
ISP
customer
solution 3 : issue ISP recommendations
ASP
ISP
ISP
ISP
ISP
ISP
ISP
ISP
customers
ISP interconnection
•
•
•
•
•
(1)
Interconnection decisions based on intuition and higher-level business
considerations (strategic partners, alliances), often corporate ties often
override technical justification.
On the technical side the real AS path is selected by BGP using shortest
path (myopic - ignores amount of traffic and cost – there is space for
optimisation there).
Decisions at this point important for the viability of the business
Pattern : ISP min-conf :presence two transit and one peering IX-es
transit usually straightforward
– (cash issue, small number of players)
•
peering more complicated (reduce traffic on expensive transit links
–
–
–
–
identify targets (IX-es, ISPs)
assesment, (monitoring (i/o) traffic flows, Netflow data from routers)
negotiation, (peering policy of the identified “target” determine the outcome)
Key Question : “How much traffic are we sending to or through the AS’s that
participate in the exchange?”
ISP interconnection
•
(2)
Criterion: Exchanged Traffic Volume between AS-es, distinguish
between
– Transit traffic passes through an AS
– Terminating traffic sinks in an AS
•
•
•
create AS rankings based on “transit/terminating traffic scores”.
Assign weights to the AS-es based on their distance from the
destination subnets.
Find those AS that appears to be central in most popular paths and try
to peer with those AS!
ISP interconnection
•
(3)
More tangible evaluation criteria
– The ownership of the links ; determines the quality of the offered service.
When the ISP owns the links it has full control over potential problems.
– The physical and link layer technologies deployed – compatibility of the
technologies and their intrinsic value (i.e. state-of-the art versus outdated
transmission technologies) can determine the reliability of the service, how
easy/fast it will be to schedule future capacity upgrades (e.g light-up more
fiber is easy) etc.
– The diversity of the physical routes – physical path separation inside the
core provider’s network is important so that customers’ mission critical traffic
is not vulnerable to outages or single points of failure. Sometimes the logical
topology maps can be deceptive as they may show route diversity but the
underlying physical links are not diverse. Also in the metropolitan area where
the links terminate there should be POPs in different physical locations
(buildings).
Decision-making framework (1)
• The minimal ISP reference model
– buy Transit from two other ISPs
– Peer with as many ISPs as possible (meaningful)
– sell transit to customers
• Create an optimal “portfolio” of peering and transit
interconnections.
Decision-making framework (2)
Inputs:
• transit traffic vector
• terminating traffic vector
• cost of peering with an AS (infinite if not available)
• cost of buying transit from an AS
• cost of peering with an exchange
Algorithm for determining the threshold for participating in an
exchange
Peering provides redundancy reduces connectivity risk
• Prob[ dest-i unreachable ]
• weight dest-i based on traffic transitting through or
terminating to it
Decision-making framework (3)
Criteria for the selection of an upstream provider:
•
•
•
•
•
geographic proximity (access circuit cost)
Network reliability (uptime)
Pricing (unit bandwidth cost)
Customer support
Value added services (etc..)
Optimisation Algorithm (most likely heuristic)
Network design
Two problems:
– capacity assignment for delay minimisation
– assumes delay analysis
Delay Analysis (1/)
• Model: N-nodes, M-links, capacities Ci , i=1,..,M measured in
packets/sec (assumption all packets have the same length 1/)
• Poisson process with mean rate i,j (packets/sec) external traffic
entering (and leaving – i.e. no termination inside the network)
node -j
N
M
    jk
j 1 k 1
network
M
   i
i 1
node -k
i    jk
j
k
Delay Analysis (2/)
• Link in the network is modelled as an M/M/1 “server with a
queue” the arrival process of the packets at link-i is I
• Let Zj,k is the expected delay for a packet entering the providers
network at node j and exiting at node k,
• Let T be the average (expected) delay a packet will experience
using the network
 i, j
T   Z j , k
j 1 k 1 
N
N
Decompose T in sum of link delays Ti be the average delay a
packet incurs at link-i (waiting time in the queue plus
transmission time).
Delay Analysis (3/)
i
T   Ti
i 1 
M
Where link delays Ti are given by standard M/M/1 results
(arrival rate i and service times 1/Ci )
Ti 
1
Ci  i
Capacity Assignment (1/)
•
•
•
•
The provider knows
• the traffic (rate bit/s) on each of the links { i },
• the topology and the routing,
The goal is to minimise the mean delay T incurred by a packet crossing
the network where the minimisation happens with respect to the link
capacities { Ci } subject to total cost constraints.
The cost incurred by the network provider to install a link of capacity Ci
is given by a function d(Ci) (linear, logarithmic, follow a power law etc.
depending on the physical length of the link, the technology etc).
Obviously Ci > i /  and the optimisation problem becomes
min T (Ci )
M
s.t. d (Ci )  D
i 1
Capacity Assignment (2/)
• The problem is solved for linear cost function d( Ci ) = di Ci and
the yields
n M i d i 2
T
[
]
Dx i 1 
Where Dx = D - i di/ is the excess cost (i /  is the
minimum required amount) and n is the average path length
defined as n =  / 
Summary
•
•
•
•
•
ISPs and ASPs face similar interconnection problems,
new economic models for the ISP industry focus on user control on
the edges and less concern about extracting rent from content
hosting.
There has been no systematic way to approach these interconnection
problems
There is need for new tools and models to assist decision making in
the peering/transit arena, facility location
These will help the ISP to manage both their interconnection costs
and the performance of their network then they will be able to
“standardise” offered services and possibly provide meaningful SLAs
to the customers.
http://www.cl.cam.ac.uk/~pg281/tapas/