Transcript DREN IPv6 Implementation

DREN IPv6 Implementation
Update
Techs in Paradise, 2008
Jan 21, 2008
Honolulu, HI
Ron Broersma
DREN Chief Engineer
High Performance Computing Modernization Program
[email protected]
21-Jan-2008
DREN IPv6 Update
1
Background
• WAN provider for the DoD R&D community
• Also serving as DoD’s IPv6 pilot
implementation of the DoD CIO June ’08
mandate
• Deploying IPv6 where possible in a
production environment
– See what works and what’s broken
– See what’s missing
– Share lessons learned
21-Jan-2008
DREN IPv6 Update
2
Previously
• Reported to you previously:
– Most serious problem is lack of IPv6 support in many
security products (firewall, IDS, IPS, VPNs, web proxy, etc).
• Major inhibitor to deployment of IPv6 in protected enclaves
– Numerous bugs
• hurts deployment and require workarounds
– Increased complexities with running dual-stack
– Vista missing some v6 support
– Some things getting better
• Important fixes in key software components
• Some incentives from Asian demands
– The scare over RH0, now deprecated
– Router meltdown
– Proponents and providers aren’t eating their own dogfood
21-Jan-2008
DREN IPv6 Update
3
What’s new
• Still finding bugs, but they are getting harder to
diagnose
• Increased campus deployment efforts
• Expanded peering
• Testbed downsizing
• Tunnel brokers updated
• www.v6.dren.net web site restored and updated
– Moved to new server (virtual)
– Fedora 7
– Some cleanup and restructuring of content
• DHPCv6 interoperability testing
• Vendors still aren’t eating their own dogfood
21-Jan-2008
DREN IPv6 Update
4
DREN IPv6 Testbed
• Starting to shut down DRENv6 testbed
– 3 (of 9) Core nodes shut down
• ERDC
• NAVO
• AFRL Kirtland
– Moving Peering from testbed to production
network
•
•
•
•
21-Jan-2008
SPRINT
UUNET
NTT/Verio
… and others in the works
DREN IPv6 Update
5
DRENv6 “testbed”
Logical Topology
Cisco
AIX-v6
C&W
Global
Crossing
LAVAnet
6TAP
Abilene
FIX-West
Hurricane
Electric
Abilene
TIC
NTTCom
Verio
WPAFB
Dayton
ARL
JITC
HP
San Diego
WCISD
SD-NAP
SDSC
SSC San Diego
Aberdeen
Tunnel broker
AOL
Wash D.C.
HICv
6
NRL
Vicksburg
(Hawaii)
SSAPAC
SPRINT
Albuquerque
AFRL
Kirtland AFB
ATM PVC (OC-3)
tunnel
21-Jan-2008
SSC Charleston
ERDC
Stennis
NAVO
DREN IPv6 Update
vBNS+
IXP
Core Router
ISP or
BGP Neighbor
“site”
6
Peering
• Upgraded all NIDS at peering locations to insure visibility of
IPv6 traffic (security).
• Renewed effort to improve peering
– Production network only had 8 operational as of 2 months ago.
– Move peers from testbed to production
• Production network used testbed for transit to most peers.
– Add new peers
•
•
•
•
•
•
•
•
•
•
UUNET (WAE and HAY)
TWAREN (Seattle, working on Starlight)
SPRINT (MAEwest)
NTT/Verio (vastly improved performance to Japan sites0
NLR (Seattle, MAEwest)
Hurricane Electric (WAE tunnel 1/22/08, MAE west cross connect being worked)
UH (last week)
HIX and LavaNet in the works
USGS (this week)
Qwest (working final draft of agreement)
• Had to adjust policy
– Waive requirement to peer at 3 locations
– Waive requirement for high bandwidth native peering
21-Jan-2008
DREN IPv6 Update
7
Path from UH to DREN
Before…
dchp-166-122-119-217:~ ron$ traceroute6 www.v6.dren.net
traceroute6 to narf.v6.dren.net (2001:480:10:1050::5)
from 2607:f278:5:3:217:f2ff:fee8:66db, 30 hops max, 12 byte packets
1 2607:f278:5:3::1 2.873 ms 3.163 ms 5.251 ms
2 2607:f278:0:5::2 19.21 ms 6.222 ms 12.63 ms
3 so-2-1-0.bb1.a.syd.aarnet.net.au 95.954 ms 103.687 ms 102.948 ms
4 ge-0-0-0.bb1.b.syd.aarnet.net.au 107.493 ms 95.859 ms 100.499 ms
5 so-2-1-0.bb1.b.sea.aarnet.net.au 264.709 ms 259.346 ms 259.667 ms
6 dren-1-lo-jmb-706.sttlwa.pacificwave.net 157.101 ms 160.606 ms 170.337 ms
7 so12-0-0-0.sandiego.dren.net 189.137 ms 189.147 ms 186.446 ms
8 narf.v6.dren.net 186.873 ms 190.175 ms 207.715 ms
After…
dchp-166-122-119-217:~ ron$ traceroute6 www.v6.dren.net
traceroute6 to narf.v6.dren.net (2001:480:10:1050::5)
from 2607:f278:5:3:217:f2ff:fee8:66db, 30 hops max, 12 byte packets
1 2607:f278:5:3::1 2.927 ms 2.413 ms 4.273 ms
2 2607:f278:0:1::1 9.538 ms 18.885 ms 2.41 ms
3 2001:480:0:c81::1 17.07 ms 2.736 ms 3.02 ms
4 2001:480:0:c2f::1 54.571 ms 57.151 ms 54.373 ms
5 so12-0-0-0.sandiego.dren.net 88.173 ms 90.586 ms 84.086 ms
6 narf.v6.dren.net 84.489 ms 84.001 ms 84.304 ms
21-Jan-2008
DREN IPv6 Update
8
Recent Frustrations
• Products still not doing IPv6
– Juniper NSM - slipped 18 months
– Tipping Point - slipped 18 months
– Bluecoat web/proxy - still no beta code, although promised a
year ago
• Bugs, bugs, bugs
– Netscreen ND is broken
– BIND sometimes stops responding when using IPv6
transport
– Losing first packet breaks NTP
– ping6 annoyance
– … and more
21-Jan-2008
DREN IPv6 Update
9
Netscreen ND bug
No.
Time
1 0.000000
2 0.001721
Source
Destination
Protocol Info
fe80::211:92ff:fe10:ca90 ff02::1:ff00:1
ICMPv6
Neighbor solicitation
fe80::212:1eff:feaf:9906 fe80::211:92ff:fe10:ca90 ICMPv6
Neighbor advertisement
Internet Control Message Protocol v6
Type: 136 (Neighbor advertisement)
Code: 0
Checksum: 0x5e5b [correct]
Flags: 0x00000060
0... .... .... .... .... .... .... .... = Not Router
.0.. .... .... .... .... .... .... .... = Not Solicited
..0. .... .... .... .... .... .... .... = Not Override
Target: 2001:480:822:1f01::1
ICMPv6 Option (Target link-layer address)
Type: Target link-layer address (2)
Length: 8
Link-layer address: 00:12:1e:af:99:06
21-Jan-2008
DREN IPv6 Update
10
Neighbor Advertisement
0
1
2
3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
Type
|
Code
|
Checksum
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|R|S|O|
Reserved
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+
Target Address
+
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
Options ...
+-+-+-+-+-+-+-+-+-+-+-+-
S
21-Jan-2008
Solicited flag. When set, the S-bit indicates that
the advertisement was sent in response to a
Neighbor Solicitation from the Destination address.
The S-bit is used as a reachability confirmation
for Neighbor Unreachability Detection. It MUST NOT
be set in multicast advertisements or in
unsolicited unicast advertisements.
DREN IPv6 Update
11
Switch loses first packet
• Foundry BigIron will lose first packet after
period of no activity
When the mac address of ins1.sd is not in the mac table of the BigIron....
[[email protected] ~]$ ping6 ins1.sd
PING ins1.sd(ins1.sd.spawar.navy.mil) 56 data bytes
64 bytes from ins1.sd.spawar.navy.mil: icmp_seq=1 ttl=64 time=0.778 ms
64 bytes from ins1.sd.spawar.navy.mil: icmp_seq=2 ttl=64 time=1.87 ms
When the mac address is in the table....
[[email protected] ~]$ ping6 ins1.sd
PING ins1.sd(ins1.sd.spawar.navy.mil) 56 data bytes
64 bytes from ins1.sd.spawar.navy.mil: icmp_seq=0 ttl=64 time=0.789 ms
64 bytes from ins1.sd.spawar.navy.mil: icmp_seq=1 ttl=64 time=4.85 ms
Notice the first time that it loses sequence 0.
21-Jan-2008
DREN IPv6 Update
12
Losing first packet
First there is the IPv6 neighbor discovery, which makes it through and is answered because it is sent to a ethernet multicast address..
17:57:26.913312 00:0b:db:93:b5:73 > 33:33:ff:00:00:02, 2001:480:10:4::3 > ff02::1:ff00:2:
icmp6: neighbor sol: who has 2001:480:10:4::2
17:57:26.915433 00:0b:db:93:b5:85 > 00:0b:db:93:b5:73, 2001:480:10:4::2 > 2001:480:10:4::3:
icmp6: neighbor adv: tgt is 2001:480:10:4::2
By now the mac address (00:0b:db:93:b5:85) should be in the mac table (but I think it isn't yet, due to some internal delay).
The next packet is the one that never makes it through the BigIron...
17:57:26.915457 00:0b:db:93:b5:73 > 00:0b:db:93:b5:85, 2001:480:10:4::3 > 2001:480:10:4::2: icmp6: echo request seq 0
That echo request is never replied to. But, the next one makes it OK...
17:57:27.912144 00:0b:db:93:b5:73 > 00:0b:db:93:b5:85, 2001:480:10:4::3 > 2001:480:10:4::2: icmp6: echo request seq 1
17:57:27.912901 00:0b:db:93:b5:85 > 00:0b:db:93:b5:73, 2001:480:10:4::2 > 2001:480:10:4::3: icmp6: echo reply seq 1
Is it possible that 24 microseconds (between the NA and the echo request packet) isn't enough time for the BigIron to get the mac table updated?
21-Jan-2008
DREN IPv6 Update
13
Losing first packet: impact
• NTP fails
– Normally backs off to 1024 second updates
– Single UDP packet sent on update
– Client never sees packet, declares server down,
and restart at 64 seconds, backs off, then fails
again
21-Jan-2008
DREN IPv6 Update
14
ping6 annoyance
• ping6 does a DNS
lookup on EVERY packet
– Bug in addrconf patch
– The value was never
copied to the cache
--- iputils/ping.c.addrcache 2002-09-20 17:08:11.000000000 +0200
+++ iputils/ping.c 2003-05-15 16:41:19.000000000 +0200
@@ -1124,6 +1124,12 @@
{
struct hostent *hp;
static char buf[4096];
+ static __u32 addr_cache = 0;
+
+ if ( addr == addr_cache )
+
return buf;
+
+ addr_cache = addr;
if ((options & F_NUMERIC) ||
!(hp = gethostbyaddr((char *)&addr, 4, AF_INET)))
--- iputils/ping6.c.addrcache 2002-09-20 17:08:11.000000000 +0200
+++ iputils/ping6.c 2003-05-15 16:41:19.000000000 +0200
@@ -893,7 +893,14 @@
*/
char * pr_addr(struct in6_addr *addr)
{
- struct hostent *hp = NULL;
+ static struct hostent *hp = NULL;
+ static struct in6_addr addr_cache = {{{0,0,0,0}}};
+
+ if ( addr->s6_addr32[0] == addr_cache.s6_addr32[0] &&
+
addr->s6_addr32[1] == addr_cache.s6_addr32[1] &&
+
addr->s6_addr32[2] == addr_cache.s6_addr32[2] &&
+
addr->s6_addr32[3] == addr_cache.s6_addr32[3] )
+
return hp ? hp->h_name : pr_addr_n(addr);
if (!(options&F_NUMERIC))
hp = gethostbyaddr((__u8*)addr, sizeof(struct in6_addr), AF_INET6);
21-Jan-2008
DREN IPv6 Update
15
Expanded deployment
• Many campus LANs now 100% IPv6 enabled
• One site has achieved dual-stacking ALL desktops
and servers (except 2)
– Others have active programs to do the same, along with
getting help desk trained and tooled for supporting it.
• Remove “A” record from DNS for servers
– Possible when all clients for that server are dual-stack
• Servers running v6-only (no IPv4 address on any
interface, nor on network it is connected to)
– Fedora 7 - some manual fiddling of the config is required,
but works
– DNS issues (BIND sometimes stops responding when doing
v6 transport queries)
21-Jan-2008
DREN IPv6 Update
16
Fedora 7 IPv6-only
• Connect to IPv6-only LAN, so no cheating.
• Configure from GUI, like any point-and-click
sysadmin would do.
• Getting it to actually work…
– Manual hacking of ifcfg-eth0
• Delete IPV6ADDR and IPV6PREFIX
• Set ONBOOT = yes
– Fix broken /etc/hosts
– Configure sendmail to listen on ::1
– Fix /etc/yum.repos.d/* files to point to an IPv6capable mirror
21-Jan-2008
DREN IPv6 Update
17
IPv6-enablement milestones
and scoring (proposed)
1.
2.
IPv6 address allocation and address/routing plan developed
LAN (wired and wireless) is fully v6-enabled (all routers do v6 on all interfaces) and is connected to the
IPv6 Internet (WAN).
–
–
3.
4.
5.
6.
The implication is that any v6-enabled device can connect anywhere in the LAN and get IPv6 Internet connectivity.
(worry about routing implementation, make sure address plan is right, think about security and performance, play with
multicast, make sure network staff is trained to support it, etc)
Internal infrastructure services (DNS, NTP, DHCP, SMTP, etc) are IPv6-enabled
Public facing services (public DNS, MXs, public web site) are IPv6-enabled
Network management infrastructure (management LAN, SNMP, SYSLOG, MIBs, management access to
switches/routers/etc) is IPv6-enabled
Most workstations and servers are v6-enabled
–
(behind this is the support infrastructure, i.e. help desk and other IT support, and a message to sys admins to turn on
IPv6 where possible, and servers have AAAA records in DNS, and your help desk tools/scripts for things like host
registration and update are upgraded to handle IPv6 addresses
7.
Once critical mass is reached on the client side, remove "A" records for servers (resulting in final incentive
and some pain for those that didn't dual-stack their workstations).
8.
Migrate some network segments to IPv6-only, with no IPv4 addresses anywhere
–
You don't need to do them all at once, just one at a time as their clients all become dual-stack
–
(this forces one to figure out how to make hosts operate in a v6-only environment, helps the organization figure out
what's missing, forces the implementation of some kind of transition mechanism to bridge to the IPv4-only world, plus
adds continued incentive to get more stragglers upgraded since they can't even get there by typing the IP address
9.
Deploy advanced features (mobile-ip, v6 multicast, etc.), provide remote IPv6 access (travellers,
telecommuters, home, etc) from v4-only environments, cleanup and reduce complexity (readdressing
network if necessary), ....
10. Declare victory
•
you claim a perfect score in public, and are willing to stand up to scrutiny
Scoring: Scale of 0 to 10. 1 point for any milestone that is 95% complete.
21-Jan-2008
DREN IPv6 Update
18
Commitment to IPv6?
• Are network product vendors really committed to
IPv6 support?
• Are they using it in their production networks?
• Do they have an IPv6 presence on the Internet?
• Do they follow the “eat your own dogfood”
principle?
• A survey…
21-Jan-2008
DREN IPv6 Update
19
June 2007
Vendor
scorecard
Jan 2008
• Looked in DNS to see if
there were AAAA
records for www, MX,
and DNS.
• Quick sampling of major
computer and network
companies showed no
public facing IPv6.
• 6 months ago, and then
today.
21-Jan-2008
DREN IPv6 Update
20
Scorecard – IPv6 Summit
Sponsors (March ’07)
• Grand and Gold
Sponsors of 2007 IPv6
Summit.
• Only one has an IPv6
presence at their
corporate “front door”
21-Jan-2008
DREN IPv6 Update
21
Summary: Situation Today
• DREN has been successfully using IPv6 in a
production environment, with many dualstack systems and services, for years
– Modern operating systems just work, out of the box
(MacOSX, Linux, Vista, Solaris 10, etc)
• Most urgent needs from our perspective:
– Need parity with IPv4 in all implementations
– Enabling IPv6 must NOT break things
– Need to make security stacks fully IPv6 capable
• Firewalls, IDS, proxies, IDP/IPS, ACLs
– Eat your own dogfood
• The rest of DoD will not make the June ’08 deadline.
• The US is falling far behind Asia and other regions.
– Prevalent attitude: IPv6 just another GOSIP, or ADA, or ….
– A call to action
21-Jan-2008
DREN IPv6 Update
22
END
21-Jan-2008
DREN IPv6 Update
23