Transcript HSN Data

HiSeasNet Data Layer
What happens after the modems
are talking through the satellite
Overview
 Big picture
 The subnet collection
 Tunnels (GRE and IPsec)
 Cisco router use in HiSeasNet
(not an IOS primer)
 Troubleshooting
 Advanced topics
The Big Picture
Legend:
Purple is Satellite RF
Teal is Synchronous Serial
Orange is “foreign” IP space
Blue is local IP space
Internet Protocol
Fundamentals
Overview/review of concepts and
how they apply to HiSeasNet
Fundamental concepts
 The Internet Protocol (IP) is a set of rules that are
followed when computers talk on the Internet.
 IP is a layer of networking above modems where
packets are relayed from one host to another until
they get to their destination.
 The IP layer is in “Layer 3” from the OSI model
 The act of accepting a packet, looking at it, and
sending it closer to its destination is called routing
and is performed by devices called routers.
 Routers are like people:
 Have many interfaces that handle information
 Must think a little to determine where that info goes
 Switches do brainless transactions on lower level
packets (Layer 2), not routing.
Typical network layout
 A packet must always go from ship to earth station to
institution’s campus, therefore…
 At least 3 hops for a packet to take to get to the
Internet on at least 3 routers (not switches, so we
have to route in layer 3)
 3 IP subnets with routes in between those hops
 HiSeasNet subnet on the ship (“DMZ”)
 Ship/shore point-to-point (“Sat P2P”)
 Earth station to home institution P2P (“Tunnel P2P”)
 Not just a block of address assignments, but fullfledged, subnetted IP blocks that everyone in the
subnet can agree on
IP subnet review
 All IP networks are broken into “subnets” that define
what addresses are considered local (who the
neighbors are that can be reached with just a
broadcast packet) to an address
 Subnets have a “network” part and a “host” part.
 The network part is defined by the subnet mask,
notated by the number of bits (ie “/24”) from left to
right that indicate the network (bits must be 1s).
 The remaining bits are the host part or decimal value
of octets (the 0s on the right).
 For example:
 Binary: 11111111 11111111 11111111 00000000
 Decimal:
255
255
255
0
IP subnet review cont’d
 Each subnet must have a network address (the
bottom most address in the block) and a broadcast
address (the top most address in the block).
 Remaining addresses in the block can be used for the
hosts are in the network.
 If those hosts want to talk outside that subnet (usually the
case), there must be a router/gateway that ferries traffic to
another network via another interface (serial, ethernet, etc.)
 Subnets can occur only at certain places in an
address range…where network addresses can be on
bit divisions (depends on the size of the subnet)
 RFC 1918 defines “private” subnet ranges to be
192.168.0.0/16, 172.16.0.0/12, and 10.0.0.0/8
Class C IP subnet example
 The typical “Class C” address block
 Subnet mask of 255.255.255.0 (aka “/24”)
 11111111 11111111 11111111 00000000
 The last 8 host bits can vary giving 255 host addresses, but
2 (network and broadcast) are used in definition of subnet
 Network address could be, for example, 192.168.0.0
 Broadcast address is then 192.168.0.255
 Usable addresses are 192.168.0.1 through 192.168.0.254
HiSeasNet DMZ subnet
example
 The typical “DMZ” block in HiSeasNet
 Subnet mask of 255.255.255.240 (aka “/28”)
 11111111 11111111 11111111 11110000
 The last 4 host bits can vary giving 16 host addresses, but 2
(network and broadcast addresses) are used in definition of
subnet
 Network address could be, for example, 172.16.1.16
 Broadcast address is then 172.16.1.31
 14 usable addresses are 172.16.1.17 through 172.16.1.30
 One address gets used by the router/gateway (usually the
first one, 172.16.1.17 in this case), so really only 13 free
addresses if you want to actually use HiSeasNet
172.16.1.x/28
network possibilities
Confused about valid
network address
locations?
Use a table. Generate
one by hand, or cheat
and use an IP
calculator.
I like the one at
www.subnetmask.info
Network
172.16.1.0
172.16.1.16
172.16.1.32
172.16.1.48
172.16.1.64
172.16.1.80
172.16.1.96
172.16.1.112
172.16.1.128
172.16.1.144
172.16.1.160
172.16.1.176
172.16.1.192
172.16.1.208
172.16.1.224
172.16.1.240
Hosts
from
172.16.1.1
172.16.1.17
172.16.1.33
172.16.1.49
172.16.1.65
172.16.1.81
172.16.1.97
172.16.1.113
172.16.1.129
172.16.1.145
172.16.1.161
172.16.1.177
172.16.1.193
172.16.1.209
172.16.1.225
172.16.1.241
to
172.16.1.14
172.16.1.30
172.16.1.46
172.16.1.62
172.16.1.78
172.16.1.94
172.16.1.110
172.16.1.126
172.16.1.142
172.16.1.158
172.16.1.174
172.16.1.190
172.16.1.206
172.16.1.222
172.16.1.238
172.16.1.254
Broadcast
Address
172.16.1.15
172.16.1.31
172.16.1.47
172.16.1.63
172.16.1.79
172.16.1.95
172.16.1.111
172.16.1.127
172.16.1.143
172.16.1.159
172.16.1.175
172.16.1.191
172.16.1.207
172.16.1.223
172.16.1.239
172.16.1.255
HiSeasNet P2P subnet
example
 The point-to-point “transit network”
 Subnet mask of 255.255.255.252 (aka “/30”)
 11111111 11111111 11111111 11111100
 The last 2 host bits can vary giving 4 host addresses, but 2
(network and broadcast addresses) are used in definition of subnet
 Network address could be, for example, 192.168.68.4
 Broadcast address is then 192.168.68.7
 Usable addresses are 192.168.68.5 and 192.168.68.6
 Allows us to route between two routers. They each get an
address in the transit network and have a very small place
where just they can talk to each other. Only room for two in this
network. Each is the other’s gateway.
 Used between ship and earth station
 Used between earth station and home institution (wrapped in a
tunnel)
HiSeasNet subnet collection
 DMZ (subnet before firewall) is usually a /29 (5
usable addresses + router/gateway) or, better yet, a
/28 subnet (13 usable addresses + router/gateway)
 Sat P2P (between ship and earth station) is a /30
subnet
 Tunnel P2P (between earth station and home
institution) is a /30 subnet. Physical end points are
internet hosts, logical addresses in tunnel are /30.
 All three of these subnets come from home
institution address space be it public or private.
This is the key to making the ship appear as though
it is part of the home institution network!
IP Tunnels
Overview of GRE and IPsec and
how those protocols apply to
HiSeasNet
What is an IP tunnel?
 A way of designating a collection of packets as part
of a higher level virtual link (think wormhole)
 Done through additional identifying headers on
packets
 The effect might be to:




Shorten a long path logically
Use foreign IP space in a network
Encrypt a link
Use other protocols (ie IPX) through IP
 Examples include GRE, IPsec, L2TP, SSL, PPTP
GRE tunnels
 Generic Routing Encapsulation (GRE) standard
defined in RFC 1701
 Very simple, efficient, easy to configure, well
supported in most firewalls and routers
 Has no authentication or encryption…just for
packaging strange packets in normal links
 Configured as a simple tunnel interface on a Cisco
 24 byte header is added, so the new payload in a
standard 1500 byte packet is now 1476 bytes.
 Data coming into tunnel is 24 bytes too big and needs to be
repackaged into smaller packets on shore.
 Sometimes repackaging is done on the ship.
 Either way, it is less efficient, adds to router CPU load, but
works fine for slow links.
IPsec tunnels
 “IP Security” standard defined in RFC 2401
 Can act like a tunnel, but is not necessarily. They are
still valid IP packets that contain a complete IP
packet in their payload.
 Look like any other IP packet to routers
 Implemented in a host networking stack or in a
router or switch
 Work may be offloaded to a hardware accelerator
 Encrypts and authenticates between ends
 Bridges a network, so no need for tunnel space
 Trickier to configure on end points (ACL-based)
 Very standard and supported by many
routers/firewalls now
Tunnels in HiSeasNet
 Primarily used for shortening routes between earth
station and institution and carrying that institutions
IP addresses to the ship.
 We prefer to use GRE tunnels for efficiency and ease
of setup. They also separate traffic a little better with
clear endpoints and no bridging.
 Some institutions want security over the shore links
(may be internal networks they are passing to the
ship or no GRE support), so we have run IPsec links
for them.
 We can run IPsec over GRE
Why bother with tunnels?
 So why even bother with tunnels and subnets when a
single IP address could be offered to a ship and set
up NAT like a cable/DSL modem?
 Not all ships want to NAT (especially bigger ones)
 Would require a VPN on the ship to see private networks at
the home institution
 Doesn’t allow incoming connections very easily
 UCSD networking will not offer permanent IP space to folks
that don’t sign up for UCSD networking policies
Cisco Routers
Not an IOS review, but a larger
picture of how Ciscos are used in
HiSeasNet
Why use Cisco routers?
 There is a lot going on at the IP layer
 IP packets on the ships need to come off via synchronous
serial satellite modems
 IP packets at the earth station need to go to the home
institution through the internet
 We need IP routing to be done in a simple, efficient,
reliable, hands-off way.
 Cisco Systems, Inc. makes routers that support
synchronous serial interfaces that work with Comtech
modems. These routers can reliably handle all the
strange routing that HiSeasNet does with minimal
interaction.
 Doesn’t have to be a Cisco box, but it is a known,
solid solution that we have experience with.
Cisco gear on shore
 Ship and earth station require synchronous serial
interfaces (WIC-1T or WIC-2T module board) and
accompanying EIA-530 cable (CAB-530MT or CAB-SS530MT depending on the serial board)
 Earth station uses a pair of 2821 routers to handle
more serial connections and more routing capacity.
Should handle at least 5Mbit okay, but may need to
upgrade for larger data rate events.
 Earth station routers split for Ku-band vs. C-band
 Home institutions terminate GRE or IPsec links on
campus Cisco routers, switches, PIX firewalls, or even
Netscreen firewalls…anything that supports tunnel
protocol and is in the right network place.
Cisco out at sea
 Doesn’t take much at the ship…usually a fairly low
end router (2600 series or newer 2800
series…usually a 2811) with a switch attached for the
DMZ.
 Basic ship Cisco config is a serial-to-ethernet box
with minor routing and BGP announcement
 Some ships handle more in their router configs
(firewalling, failover interfaces for shore connections,
links to SWAP, etc.). May require additional hardware
modules.
 Do you need a backup router? Possibly. Had a small
Cisco die in the first month of HiSeasNet.
 No one is terminating a HiSeasNet link at a Linux
box, but it is theoretically possible.
Cisco configuration
 Routers keep a configuration file in NVRAM.
 The initial loading of this file can be tricky
 The configuration can be modified on the router, then
saved to the startup file
 Routers have user accounts and two layers of
passwords
 Routers are usually named “rv-shipname-gw”
 Generally a configuration gets set and is left alone
 Any fiddling you want to do is at your own risk
 Firmware should be updated from time to time when
security problems are announced.
interface Serial0/0/0
description "Serial line to satellite modem"
ip address 131.128.19.225 255.255.255.252
ip access-group 131 in
no ip redirects
hostname rv-endeavor-gw
no ip unreachables
!
no ip proxy-arp
boot-start-marker
ip accounting output-packets
boot system flash:c2800nm-entservicesk9-mz.123-14.T7.bin ip mtu 1400
boot system flash:c2800nm-entservicesk9-mz.123-8.T9.bin no ip mroute-cache
boot-end-marker
no keepalive
!
fair-queue
enable secret 5 $1$/nBM$yIEQ7v/blahblahblah
no cdp enable
!
!
no ip dhcp use vrf connected
router bgp 64521
ip dhcp excluded-address 131.128.217.225 131.128.217.239 no synchronization
!
bgp log-neighbor-changes
ip dhcp pool dmz
network 131.128.0.0
network 131.128.217.224 255.255.255.224
redistribute connected
default-router 131.128.217.225
neighbor 131.128.19.226 remote-as 64521
!
neighbor 131.128.19.226 description ucsd-sdsc-roadnet-2611
interface FastEthernet0/0
neighbor 131.128.19.226 next-hop-self
description Ships Network
neighbor 131.128.19.226 weight 30000
ip address 131.128.217.225 255.255.255.224
neighbor 131.128.19.226 distribute-list 28 out
no ip mroute-cache
no auto-summary
duplex auto
!
speed auto
ip route 0.0.0.0 0.0.0.0 Serial0/0/0 205
no cdp enable
access-list 28 permit 131.128.217.224 0.0.0.31
access-list 131 permit ip any 131.128.217.224 0.0.0.31
access-list 131 permit ip any 131.128.19.224 0.0.0.3
access-list 131 deny ip any any log
Basic Ship Config
More Cisco Resources
 Internetwork Operating System (IOS) is the operating
system Cisco routers run. It has a steep learning
curve and lots of ways to get into trouble.
 If you feel the need to fiddle with your router:
 You are on your own.
 Get a good reference
 Cisco IOS Cookbook (ISBN: 0596527225)
 Cisco: The Complete Reference (ISBN: 0072192801)
 Anything else that seems fit to your interest/skill level
 Really handy to know how to ping from router
 It can be handy to know how to update
firmware…but you will probably mess it up the first
time or two.
Troubleshooting
Ways to go about troubleshooting
the data layer of HiSeasNet
Problem solved!
 The vast majority of (all?) routing problems are on
shore
 If you think you found a routing problem:
 Contact HiSeasNet tech team and ask us to look into our
routing on shore.
 We probably know about it already, but sometimes our routers
spontaneously hang or otherwise go quiet.
 Sometimes the problem is an outage at the home institution
somewhere between the internet and the tunnel to the earth
station (often both)
 Those problems on the ship may be:
 Problems getting to the HiSeasNet router
 Due to a router change after installation (accidental,
intentional mistake, unplugged cable, reboot, etc.)
Troubleshooting overview
 This is only an issue if:
 The antenna is successfully tracking the bird
 The modem is locked up happily and is transmitting and
receiving
 The router is powered up and all cable connections are
correct (must have a link light on the ethernet port!)
 …but no packets are going across the link
 Important to note that packets must go in both
directions for the data path to be established
 Tools to use are “ping”, “traceroute”, sometimes
“telnet”, and a packet sniffer if you know how
 Clueless? Follow the packet path on the next slide,
and tell us what you find (substitute your own
addresses, though)
Warning!
Not all ships have the same router
setup. HiSeasNet flexibility causes
some things to be different for
how institutions connect in.
Know how you connect for best
troubleshooting results.
Example packet path from the
ship
 Packet leaving workstation in Oceanus’s DMZ
 Workstation in DMZ [/28] (128.128.252.34)
 Into DMZ side of router (128.128.252.33)
 Out serial side of ship router in Sat P2P [/30]
(128.128.252.213)
 In shore side of Sat P2P (128.128.252.214)
 Out earth station side of tunnel [/30] (128.128.252.218)
 Ignore Internet hops from 137.110.255.81 to 128.128.252.194
here…its all in a tunnel.
 In institution side of tunnel (128.128.252.217)
 Inside WHOI network…can stay here, go to Internet, or be
ignored.
 If packets go to Internet, they must return through
reverse path (ie via WHOI)
Ping
 The DMZ side of the router on the ship should
respond to pings from the Internet if the network is
public. That is the end-to-end test.
 Pings should be about 800ms round-trip on a clean
link. When the link is congested, it could be many
thousands of ms.
 Since congested links give lots of delay, use a ping
command that handles a large delay. Windows ping
command needs a -w (?) option, Cisco needs a
“timeout” option.
 If you ping from the router, be sure to set your
source correctly if you have many interfaces.
 Far ends of tunnels don’t ping so well.
Ping testing
 If end-to-end ping doesn’t work, try a closer
destination, possibly from the router (if you are ok
with IOS): “ping <dest> timeout 10 source
serial0/0/0”
 <dest> can be the other address in the Sat P2P network
 Dest can be other hosts, possibly along the way, possibly on
the Internet.
 Looks like:
Cisco ->
ucsd-sdsc-roadnet-gw#ping knorr timeout 4 source serial0/0/0
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 128.128.252.209, timeout is 4 seconds:
Packet sent with a source address of 137.110.255.93
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 620/620/624 ms
Unix ->
foley@epicenter 4> ping -s 128.128.252.17
PING 128.128.252.17: 56 data bytes
64 bytes from hsnkr.whoi.edu (128.128.252.17): icmp_seq=0. time=871. ms
64 bytes from hsnkr.whoi.edu (128.128.252.17): icmp_seq=1. time=795. ms
64 bytes from hsnkr.whoi.edu (128.128.252.17): icmp_seq=2. time=795. ms
^C
Traceroute
 Is supposed to show all hops between two
hosts
 Does not work well through NAT connections
 May not work due to firewalling at campus
 Source is important…sourcing from router’s
ethernet interface is always helpful
 Will take a very long time from the ship due
to satellite delay
Traceroute output from router
 Traceroute to generic
host on UCSD campus
from Pelican’s router
 Looks like:
 1 is shore side of Sat
P2P
 2 is institution side of
tunnel
 3 is inside institution
 4 to15 is Internet
 16 is UCSD front door
 17 is UCSD campus
 18 is end node
rv-pelican-gw#traceroute 132.239.4.66 source 204.196.250.129
Type escape sequence to abort.
Tracing the route to 132.239.4.66
1 192.168.2.6 572 msec 572 msec 576 msec
2 192.168.2.1 628 msec 628 msec 628 msec
3 162.75.221.73 632 msec 632 msec 636 msec
4 64.200.121.22 640 msec 640 msec 644 msec
5 64.200.121.21 648 msec 648 msec 648 msec
6 64.200.210.53 644 msec 648 msec 648 msec
7 64.200.210.65 648 msec 648 msec 648 msec
8 64.200.249.130 656 msec 648 msec 648 msec
9 4.68.110.13 648 msec 648 msec 648 msec
10 4.68.19.126 656 msec 660 msec 648 msec
11 4.69.136.157 656 msec 656 msec 652 msec
12 4.69.132.77 684 msec 680 msec 684 msec
13 4.69.137.18 692 msec
4.69.137.22 740 msec
4.69.137.26 692 msec
14 4.68.20.4 680 msec
4.68.20.68 680 msec
4.68.20.132 716 msec
15 4.78.194.82 680 msec 680 msec 684 msec
16 137.164.24.210 684 msec 684 msec 688 msec
17 132.239.255.145 712 msec 684 msec 684 msec
18 132.239.4.66 684 msec 684 msec 684 msec
Packet sniffing
 Probably overkill for simple problems…most
HiSeasNet issues are just about connectivity
 Looks at what packets are flowing across an interface
 Great tool for seeing what is really going on, but may
not be easy to use
 Often more than you want to know
 Not standard on all computers
 Check out snoop, tcpdump, WireShark (formerly
Ethereal), snort, or something else
 Traces can be very helpful for subtle problems (like
MTU issues, worms, link saturation, etc.)
Advanced topics
A few extra things to think about
after it is all working
Starts with answers, ends with
questions
Common carrier routing
 We automatically share the shore-to-ship link via
Cisco fair queuing
 Allows for bursting when other ships are quiet
 All shore-to-ship routes for a satellite go through one
transmitting modem
 Ship routers receive all traffic for all ships in that
footprint on the serial port, but filter out just theirs
 Modems on shore are largely receive-only
 Shore routing is a bit tricky to split traffic in on one
modem, but sent on another
The MTU problem
 Maximum Transmission Unit (MTU) is the largest
amount of data that can be sent in one bundle
through an interface
 Default for Ethernet is 1500
 Smaller if tunnel overhead is added
 Bigger MTU means fewer headers and more
efficiency. Smaller MTU means less efficient.
 All routers along the way must agree on this or be
willing to adjust to meet something else
 Hard to agree since many routers block agreement
protocols for security reasons.
 In HiSeasNet, we want smaller packets for better
VoIP, but shared outroute causes problems. Every
institution chokes on different values.
Quality of Service (QoS)
 Is it a delay problem or a jitter problem?
 Wouldn’t it be great to increase the priority of certain
packets?
 Yes: VoIP connections might be smoother, important traffic
could be faster, etc.
 No: It really doesn’t do much good if the packets are small
 We do this for VoIP boxes, and have found that such
small packets get stuck behind large ones if they are
just a little too late.
 (If you setup VoIP and want an IP included here, let us
know and we will add it to the list)
 Solution: Smaller MTUs…but not so good in
HiSeasNet as discussed earlier
Services offered
 What services are you willing to offer your users?
 VoIP (Skype, Vonage, campus PBX, etc.), instant messaging,
video conferencing
 Web, FTP, mail, rsync
 SSH, VPNs
 Streaming data
 Software updates, license key servers, etc.
 How will these services be maintained and advertised
to the user?
 Will they fit into your network layout?
Security
 This comes in many flavors and may include:
 Privacy of communications
 Data can be encrypted (ship-to-shore? shore-to-shore?)
 Permission to use network services
 Are you in a public or private network?
 Access to resources can be restricted
 Viruses and spam getting onto the ship
 Wasting bandwidth (denial of service)
 What threats are you afraid of?
 What risks are you willing to take?
 HiSeasNet can be flexible, but define your policy first,
then look for the technical solution
Policy
Who gets access to HiSeasNet?
What priorities are there for HiSeasNet use?
Where can HiSeasNet be used on a ship?
What sort of guest access is available?
How is bandwidth used efficiently?
Can shore staff change configurations on the ship?
Can the ship be contacted from shore, or should
connections only be one-way?
 What VoIP services are offered and to who?







 Discussion about who is doing what in the fleet?
Earth station network
Monitoring
 How does one monitor a network where links are
expected to go out?
 How does one monitor a network with status that is
not under his control or knowledge?
 How does one monitor through firewalls into private
IP space?
 Links go down, stations go offline, and sirens do not
go off at the earth station. It is hard to tell what is a
problem and what is normal behavior.
The Big Picture
Legend:
Purple is Satellite RF
Teal is Synchronous Serial
Orange is “foreign” IP space
Blue is local IP space