Transcript How to Multi-Home

How to Multi-Home
Avi Freedman
VP Engineering
AboveNet Communications
What is Multi-Homing?
• Multi-homing is the process of selecting,
provisioning, and installing a redundant
connection to the Internet.
• Could be the same provider, or a different
provider.
Why Multi-Home?
• Slow is 1,000,000% better than dead.
• You may be out of bandwidth.
• And
–
–
–
–
Telco circuits die.
Routers die.
Providers’ networks fail.
Different networks have better performance to
different sites.
A Multi-Homed Architecture
• Ideally, take advantage of the opportunity to
multi-home to remove all single points of
failure in your network.
• Use – Multiple providers, unless your current
provider will let you have cheap backup
– Multiple routers
– Multiple telco vendors
Multi-Homed Architecture
• Two routers, each with a different WAN
connection from a different telco vendor.
• Use HSRP or VRRP internally to make both
routers look like one “virtual” router.
• Eventually, multiple providers.
• Upcoming Boardwatch article with configs.
How the Internet Works
• Well, it breaks more than it works but when
it does work • The Internet is a network of networks.
• Each network (called Autonomous System)
on the Internet announces “routes”, which
are lists of the IP addresses of the boxes on
their network.
• You need to be able to send packets *to*,
and get packets *from*, everywhere.
Inbound Traffic - Routes
• Routes are announced via BGP4 (the
Border Gateway Protocol)
• Routers are announced to BGP peers.
• Each “BGP peer” can be a “network peer”
or a “transit peer”.
• Network peers exchange just lists of
customer routes.
• Each route is tagged by the ASNs it passes
through.
Inbound Traffic - Routes
• So when AboveNet and UUNET peer, only
AboveNet and UUNET routes are
exchanged. No Sprint, PSI, etc...
• Transit peers – Announce to their customers all of the routes
on the ‘net (AboveNet, UUNET, Sprint, PSI,
and the 60,000+ routes on the ‘net).
– Announce to their peers all routes heard via
transit.
Inbound Traffic - Routes
• So if you advertise 207.106.96.0/19 to
AboveNet, – If you’re a network peer, they only re-announce
207.106.96.0/19 to customers (and use it
internally);
– If you’re a transit peer/customer, they announce
207.106.96.0/19 to all of their network peers.
• That’s how you get global *inbound*
reachability.
Address Space Issues
• Noone wants to hear a route for you unless – You are multi-homed (even then, some people
don’t want to hear routers), or
– You have your own direct IP space allocation
from ARIN, RIPE, or APNIC.
• So, when you’re single-homed without your
own space, your IPs are reachable because
they’re part of your provider’s “aggregate”
block.
Address Space Issues
• For example, your provider has
207.8.128.0/17.
• You have 27.8.197.0/24 from them.
• You’re single-homed.
• The only route on the ‘net for you is the
207.8.128.0/17 route, “originated” by your
provider’s ASN (and you don’t have to do
anything special).
Address Space Issues
• If you have your own CIDR block and are
single-homed, your provider will originate
it.
• So, if you have 219.190.64.0/19, it’ll be
visible as an announcement by your
provider, originated into the BGP mesh with
your provider’s ASN as the “origin”.
Address Space Issues
• If you have your own IP space and want to
multi-home, addressing issues are simple.
• Your other provider will start also
originating your IP blocks.
• Or you’ll start speaking BGP, originate your
IP blocks, and your providers will readvertise them to the world.
Address Space Issues
• If you don’t have your own IP space, it’s a
bit more complicated.
• So, normally your ISP will only be
advertising 207.8.128.0/17 if you have
207.8.200.0/23.
• If you’re multi-homed, your other provider
will have to advertise 207.8.200.0/23.
• But *so will your first provider*.
• Why?
Address Space Issues
• Routes are chosen first by specificity.
• That is, to how many IP addresses they
refer.
• The route “covering” the fewest IP is the
most specific, and wins.
• (Otherwise default would always win and
nothing would work.)
Address Space Issues
• So, if ISP 1 advertises only 207.8.128.0/17
and ISP 2 advertises only 207.8.200.0/23,
all inbound traffic from the ‘net will come
in on ISP2.
• So, ISP 1 needs to “blow a hole in their
filters” to “leak” the more specific
207.8.200.0/23 route.
Address Space: Filtering
• Some ISPs do or did filter on routes smaller
than (more specific than) /19s in > 205.0.0.0
space.
• But it doesn’t matter as long as your two
upstreams have good connectivity.
• Why?
Address Space: Filtering
• If Sprint doesn’t see 207.8.200.0/23 from
ISP1 or ISP2, they’ll still see your
provider’s 207.8.128.0/17 route.
• So if your connectivity to ISP1 (the owner
of 207.8.128.0/17) goes down, all will be
well as long as ISP1 still sees
207.8.200.0/23 from ISP2.
• Sprint -> ISP1 -> ISP2
• This is why people don’t let you take IPs...
Load-Balancing Outbound
• You can use static default routes to control
outbound packets.
– ip route 0.0.0.0 0.0.0.0 serial0/0
– ip route 0.0.0.0 0.0.0.0 serial1/0
• If they’re equal-cost (no metric at the end),
it’ll load-balance based on *destination*, by
default.
Load-Balancing Outbound
• Why load-balance based on destination?
• For internal networking, sometimes perpacket-load balancing makes sense.
• But if you’re trying to talk to England and
one provider has a 60ms path and the other
has a 150ms path, packets will arrive out of
order and TCP and UDP apps get unhappy
and slow.
How it works, Single-Homed
• Outbound (easy):
– Use a default route to your provider.
• Inbound:
– Your provider originates a large (aggregate)
BGP route, and gives you some space from
inside it; and/or
– Your provider originates BGP routes for your
ARIN/RIPE/APNIC CIDR blocks as well.
How it Works, Multi-Homed, Static
• Outbound (easy):
– Load-balance default routes to deal with
outbound packets.
• Inbound:
– Your providers both originate BGP routes for
just the address space you’re using, even if it’s
out of one provider’s space; and/or
– Your providers both originate BGP routes for
your ARIN/RIPE/APNIC CIDR blocks as well.
How it Works, Multi-Homed, Static
• Special note:
– When providers configure BGP for singlehomed customers, they will generally “nail up”
your routes (even your directly-issued) CIDR
blocks, so that if your connection goes down
and up and down and ..., they don’t have to flap
that route out to the whole Internet. This is a
good thing.
How it Works, Multi-Homed, Static
• Special note (ctd):
– But you NEED to make sure, when you’re
multi-homed, that the providers are NOT
nailing your routes up.
– Why?
– Because if they do, when one T1 goes down,
that provider will still advertise you to the
world, thus “blackholing” you.
How it Works, Multi-Homed, BGP
• Topic of next talk.
• You either load-balance outbound with
statics, or take full routes from your
providers (if you can).
• You originate advertisements under your
ASN for your directly-issued CIDR blocks,
AND for the parts of your providers’ space
that you’re using (with their permission).
The Transition: Static Routing
• To transition:
–
–
–
–
–
Turn up the other T1/T3/Ethernet.
Put IPs on the interface.
Run tests end-end.
Start load-balancing default to the new T1.
Then, in the middle of the night, have the new
provider start advertising your IP space. Make
sure you have reachability to every other ISP
you can think of afterwards.
The Transition: Static Routing
• To transition (ctd):
– After testing it live, turn off your other transit
pipes and make sure that, after a few minutes,
you still have connectivity.
The Transition: BGP Routing
• To transition:
–
–
–
–
–
Turn up the other T1/T3/Ethernet.
Put IPs on the interface.
Run tests end-end.
Start load-balancing default to the new T1.
Then, undo that and bring up a BGP session
that permits no routes either way.
– Then start taking routes, and watch outbound
traffic.
The Transition: BGP Routing
• To transition (ctd):
– Then, start announcing your routes.
– Then, in the middle of the night, have your ISP
take out the static route and BGP announcement
they were making.
– Make sure your route is propagating.
– Test reachability.
– Turn off your other pipes.
– Test reachability.
BGP or no?
• Advantages of doing static – Cheaper/smaller routers (less true nowadays)
– Simpler to configure
• Advantages of doing BGP – More control of your destiny (have providers
stop announcing you)
– Faster/more intelligent selection of where to
send outbound packets.
– Better debugging of net problems (you can see
the Internet topology now)
Same Provider or Multiple?
• If your provider is reliable and fast, and
affordably, and offers good tech-support,
you may want to multi-home initially to
them via Frame, SMDS, or some backup
path (slow is 1,000,000% better than dead).
• Eventually you’ll want t multi-home to
different providers, to avoid failure modes
due to one provider’s architecture decisions.
Questions?
• [email protected]
• inet-access mailing list
• Nailing routes