Transcript Mapping the Internet and Intranets

Mapping the
Internet and
Intranets
Steve Branigan, Hal Burch, Bill
Cheswick
[email protected]
120 slides
Motivations
• Intranets are out of
control
– Always have been
• Highlands “day after”
scenario
• Internet tomography
• Curiosity about size
and growth of the
Internet
• Same tools are useful
• Panix DOS attacks
– a way to trace
anonymous packets
back!
for understanding any
large network,
including intranets
Mapping the Internet and intranets
slide 2 of 120
The Original Project
• Long term reliable
collection of Internet
and Lucent
connectivity
information
– without annoying
too many people
– movie of Internet
growth!
• Develop tools to probe
intranets
• Extended database for
researchers
• Attempt some simple
visualizations of the
data
Mapping the Internet and intranets
slide 3 of 120
Uses for the Internet data
• topography studies
• long-term routing studies
• publicly available database
– (“open source”) for spooks
• interesting database for graph theorists
• combine with other mappers to make an
actual map of the Internet
Mapping the Internet and intranets
slide 4 of 120
History of the Project
• Started in August 1998 at Bell Labs
• April-June 1999: Yugoslavia mapping
• July 2000: first customer intranet scanned
• Sept. 2000: spun off Lumeta from
Lucent/Bell Labs
• June 2002: “B” round funding completed
Mapping the Internet and intranets
slide 5 of 120
Related Work
• See Martin Dodge’s cyber geography page
• MIDS - John Quarterman
• CAIDA - kc claffy
• Mercator
• Enter “internet map” in your search engine
Mapping the Internet and intranets
slide 6 of 120
Mapping the Internet and intranets
slide 7 of 120
Mapping the Internet and intranets
slide 8 of 120
Methods - data collection
• Single reliable host connected at the
company perimeter
• Daily full scan of Lucent
• Daily partial scan of Internet, monthly full
scan
• One line of text per network scanned
– Unix tools
Mapping the Internet and intranets
slide 9 of 120
Methods - network
scanning
• Obtain master network list
– network lists from Merit, RIPE, APNIC, etc.
– BGP data or routing data from customers
– hand-assembled list of Yugoslavia/Bosnia
• Run a traceroute-style scan towards each
network
• Stop on error, completion, no data
– Keep the natives happy
Mapping the Internet and intranets
slide 10 of 120
Daily database
• 100-200MB of text
• compresses to 5-10MB
• daily Internet results available from mapping
web page
– have not checked to see who gets it!
• Saved to different partition, and offloaded to
other secure computer
Mapping the Internet and intranets
slide 11 of 120
Traceroute
• Probes toward each target network with
increasing TTL
• Probes are ICMP, UDP, TCP to port 80, 25,
139, etc.
• Some people block UDP, others ICMP
Mapping the Internet and intranets
slide 12 of 120
Traceroute
Hop 1
Hop 2
Hop 3
Router
Router
Router
IP
IP
IP
IP
Hardware
Hardware
Hardware
Hardware
Client
Application level
TCP/UDP
Hop 3
Hop 4
Server
Application level
Router
Router
IP
IP
IP
Hardware
Hardware
Hardware
Mapping the Internet and intranets
TCP/UDP
slide 13 of 120
Send a packet with a TTL
of 1…
Hop 1
Hop 2
Hop 3
Router
Router
Router
IP
IP
IP
IP
Hardware
Hardware
Hardware
Hardware
Client
Application level
TCP/UDP
Hop 3
Hop 4
Server
Application level
Router
Router
IP
IP
IP
Hardware
Hardware
Hardware
Mapping the Internet and intranets
TCP/UDP
slide 14 of 120
…and we get the death
notice from the first hop
Hop 1
Hop 2
Hop 3
Router
Router
Router
IP
IP
IP
IP
Hardware
Hardware
Hardware
Hardware
Client
Application level
TCP/UDP
Hop 3
Hop 4
Server
Application level
Router
Router
IP
IP
IP
Hardware
Hardware
Hardware
Mapping the Internet and intranets
TCP/UDP
slide 15 of 120
Send a packet with a TTL
of 2…
Hop 1
Hop 2
Hop 3
Router
Router
Router
IP
IP
IP
IP
Hardware
Hardware
Hardware
Hardware
Client
Application level
TCP/UDP
Hop 3
Hop 4
Server
Application level
Router
Router
IP
IP
IP
Hardware
Hardware
Hardware
Mapping the Internet and intranets
TCP/UDP
slide 16 of 120
… and so on …
Hop 1
Hop 2
Hop 3
Router
Router
Router
IP
IP
IP
IP
Hardware
Hardware
Hardware
Hardware
Client
Application level
TCP/UDP
Hop 3
Hop 4
Server
Application level
Router
Router
IP
IP
IP
Hardware
Hardware
Hardware
Mapping the Internet and intranets
TCP/UDP
slide 17 of 120
Advantages
• We don’t need access (I.e. SNMP) to the
routers
• It’s very fast
• Standard Internet tool: it doesn’t break
things
• Insignificant load on the routers
• Not likely to show up on IDS reports
• We can probe with many packet types
Mapping the Internet and intranets
slide 18 of 120
Limitations
• Outgoing paths only
• View is from scanning host only
• Takes a while to collect alternating paths
• Gentle mapping means missed endpoints
• Imputes non-existent links
Mapping the Internet and intranets
slide 19 of 120
The data can go either
way
B
C
D
A
E
F
Mapping the Internet and intranets
slide 20 of 120
The data can go either
way
B
C
D
A
E
F
Mapping the Internet and intranets
slide 21 of 120
But our test packets only
go part of the way
B
C
D
A
E
F
Mapping the Internet and intranets
slide 22 of 120
We record the hop…
B
C
D
A
E
F
Mapping the Internet and intranets
slide 23 of 120
The next probe happens
to go the other way
B
C
D
A
E
F
Mapping the Internet and intranets
slide 24 of 120
…and we record the other
hop…
B
C
D
A
E
F
Mapping the Internet and intranets
slide 25 of 120
We’ve imputed a link that
doesn’t exist
B
C
D
A
E
F
Mapping the Internet and intranets
slide 26 of 120
Remediations
• Alternate routes not a factor on intranets
• Scan from several sources
• “stitching” needed
– Traceroute in different directions gives
different interface IP addresses
– Techniques needed to link multiple IP
addresses to a single host machine
Mapping the Internet and intranets
slide 27 of 120
Network scanning
• Custom program
• Concurrently scans towards 500 nets at
once
• Throttled to 400 packets/sec
– 100 p/s over dialup modems!
• Slow daily scan for host on destination
network
Mapping the Internet and intranets
slide 28 of 120
Data collection
complaints
• Australian parliament was the first to
complain
• List of whiners (25 nets)
• Military noticed immediately
– Steve Northcutt
– arrangements/warnings to DISA and CERT
Mapping the Internet and intranets
slide 29 of 120
Visualization goals
• make a map
– show interesting features
– debug our database and collection
methods
– hard to fold up
• geography doesn’t matter
• use colors to show further meaning
Mapping the Internet and intranets
slide 30 of 120
Mapping the Internet and intranets
slide 31 of 120
Colored by
AS number
Mapping the Internet and intranets
slide 33 of 120
Mapping the Internet and intranets
slide 34 of 120
Mapping the Internet and intranets
slide 35 of 120
Mapping the Internet and intranets
slide 36 of 120
Mapping the Internet and intranets
slide 37 of 120
Mapping the Internet and intranets
slide 38 of 120
Mapping the Internet and intranets
slide 39 of 120
Mapping the Internet and intranets
slide 40 of 120
Mapping the Internet and intranets
slide 41 of 120
Mapping the Internet and intranets
slide 42 of 120
Mapping the Internet and intranets
slide 43 of 120
Mapping the Internet and intranets
slide 44 of 120
Mapping the Internet and intranets
slide 45 of 120
Mapping the Internet and intranets
slide 46 of 120
Mapping the Internet and intranets
slide 47 of 120
Mapping the Internet and intranets
slide 48 of 120
Mapping the Internet and intranets
slide 49 of 120
Mapping the Internet and intranets
slide 50 of 120
Map Coloring
• distance from test host
• IP address
– shows communities
• Geographical (by TLD)
• ISPs
• future
– timing, firewalls, LSRR blocks
Mapping the Internet and intranets
slide 51 of 120
Colored by IP address!
Mapping the Internet and intranets
slide 52 of 120
Colored by geography
Mapping the Internet and intranets
slide 53 of 120
Colored by ISP
Mapping the Internet and intranets
slide 54 of 120
Colored by distance
from scanning host
Mapping the Internet and intranets
slide 55 of 120
US military
reached by ICMP ping
Mapping the Internet and intranets
slide 56 of 120
US military networks
reached by UDP
Mapping the Internet and intranets
slide 57 of 120
Mapping the Internet and intranets
slide 58 of 120
Mapping the Internet and intranets
slide 59 of 120
Yugoslavia
An unclassified peek at a new
battlefield
120 slides
Mapping the Internet and intranets
slide 61 of 120
Un film par Steve
“Hollywood”
Branigan...
120 slides
Mapping the Internet and intranets
slide 63 of 120
fin
120 slides
NYC after 9/11
120 slides
CIDR and IP Counts
180K
175K
Count
170K
# Edges
# CIDRs
# IPs
165K
160K
155K
150K
145K
9/11 9/12 9/13 9/14 9/15 9/16 9/17 9/18 9/19 9/20 9/21 9/22
Dateintranets
Mapping the Internet and
slide 66 of 120
Routers in New York City
1400
# Routers
1300
1200
1100
1000
9/11 9/12 9/13 9/14 9/15 9/16 9/17 9/18 9/19 9/20 9/21 9/22
Date
Mapping the Internet and intranets
slide 67 of 120
Internet before 9/11/2001
Mapping the Internet and intranets
slide 68 of 120
Internet after 9/11/2001
Mapping the Internet and intranets
slide 69 of 120
Let’s look at some
intranets
120 slides
Mapping the Internet and intranets
slide 71 of 120
Mapping the Internet and intranets
slide 72 of 120
Mapping the Internet and intranets
slide 73 of 120
Mapping the Internet and intranets
slide 74 of 120
Mapping the Internet and intranets
slide 75 of 120
Mapping the Internet and intranets
slide 76 of 120
Anything large
enough to be
called an
“intranet” is
out of control
120 slides
This is not the fault of
network administrators!
• Robust internet design frustrates central
control
– Ad hoc growth
– Mergers and acquisitions frustrate
long-term network planning and
policies
• CIOs and auditors already know this
Mapping the Internet and intranets
slide 78 of 120
Mapping the Internet and intranets
slide 79 of 120
Mapping the Internet and intranets
slide 80 of 120
Mapping the Internet and intranets
slide 81 of 120
Mapping the Internet and intranets
slide 82 of 120
We call these
“routing leaks”
• Easily-found holes in the intranet perimeter
• Show up nicely on the maps
• Leaking hosts or routers announce routes to
other networks or the Internet
• Sometimes left over from an old corporate
split
• Non-functional VPNs can show up
Mapping the Internet and intranets
slide 83 of 120
This was
Supposed
To be a
VPN
Mapping the Internet and intranets
slide 84 of 120
The maps are useful, but
not the main data
• We collect tens of megabytes of network
data
• There were unexpected subtleties to this
process
• How do you display all this information,
given that different clients want different
data?
Mapping the Internet and intranets
slide 85 of 120
The second technology:
host leak detection
• Developed to find hosts that have access to
both intranet and Internet
• Or across any privilege boundary
• Leaking hosts do not route between the
networks
• May be a dual-homed host
• Not always a bad thing
• Technology didn’t exist to find these
Mapping the Internet and intranets
slide 86 of 120
Possible host leaks
• Miss-configured telecommuters connecting
remotely
• VPNs that are broken
• DMZ hosts with too much access
• Business partner networks
• Internet connections by rogue managers
• Modem links to ISPs
Mapping the Internet and intranets
slide 87 of 120
Leak results
• Found home web businesses
• At least two clients have tapped leaks
– One made front page news
Mapping the Internet and intranets
slide 88 of 120
Leak Detection
Prerequisites
• List of potential leakers: obtained by census
• Access to intranet
• Simultaneous availability of a “mitt”
Mapping the Internet and intranets
slide 89 of 120
Leak Detection Layout
mitt
D
Mapping host
A
Internet
intranet
• Mapping host with
address A is
connected to the
intranet
• Mitt with address D
has Internet access
• Mapping host and
C
B
mitt are currently the
same host, with two
interfaces
Test host
Mapping the Internet and intranets
slide 90 of 120
Leak Detection
mitt
D
Mapping host
A
• Test host has known
address B on the
intranet
• It was found via
Internet
intranet
census
• We are testing for
C
B
unauthorized access
to the Internet,
possibly through a
different address, C
Test host
Mapping the Internet and intranets
slide 91 of 120
Leak Detection
mitt
D
Mapping host
A
• A sends packet to B,
with spoofed return
address of D
• If B can, it will reply
Internet
intranet
C
to D with a
response, possibly
through a different
interface
B
Test host
Mapping the Internet and intranets
slide 92 of 120
Leak Detection
mitt
D
Mapping host
A
• Packet must be crafted
so the response won’t
be permitted through the
firewall
• A variety of packet types
Internet
intranet
and responses are used
• Either inside or outside
address may be
discovered
• Packet is labeled so we
C
B
know where it came from
Test host
Mapping the Internet and intranets
slide 93 of 120
Leak Detection
mitt
D
Mapping host
A
• This describes
outbound leaks
• Inbound leaks are
Internet
intranet
C
usually much more
serious
B
Test host
Mapping the Internet and intranets
slide 94 of 120
Possible problems
• NAT
• egress filtering
• transit of sensitive data over the public
networks
Mapping the Internet and intranets
slide 95 of 120
Our new tools give new
views of intranets
• The pictures are mostly for management
• Maps can show progress
– red is bad, blue is good
– we can color the maps in many ways
• The real value in the reports is the list of
anomalies
– network leaks, routing loops, open
routers, etc.
Mapping the Internet and intranets
slide 96 of 120
How we scan
• Via dialup, using RAS servers
• Secure tunnel, if you prefer
– IP/SEC
– PPTP
– others?
Mapping the Internet and intranets
slide 97 of 120
What we do
• Probe the network for things not in the
official list
• Run a host enumeration
• Run leak tests on each host found
Mapping the Internet and intranets
slide 98 of 120
Technology used
• Traceroute
• SNMP queries
– Router type
– Routing tables
• Pings
• Special leak detection probes
– ICMP
– UDP
– Other possible if requested
Mapping the Internet and intranets
slide 99 of 120
Report
• HTML-based
• Delivered on CDROM or DVD
• Maps
• Executive summary shows highpoints
• Interactive map viewer tool for Windows
Mapping the Internet and intranets
120
slide 100 of
Competitors?
• Not yet, not quite
– Many use the same terms, but offer
different services
• Some components are pretty easy and free
– Host enumeration
– But we do it better (!)
• A bit like HP OpenView
– OpenView doesn’t scale
– Much slower
Mapping the Internet and intranets
120
slide 101 of
Value
• Discovers unknown parts of the network
• Data feeds into existing tools, enhancing their value
• You can’t secure what you don’t know about
• Due diligence for intranets
– Insurance?
• M&A activity
• Personnel turnover leaves legacy connections
• Business partners
Mapping the Internet and intranets
120
slide 102 of
Getting a report
• Web-based
• We can send you a CD-ROM
• You can access a web server
– FreeBSD-based
– One-time password authentication
– Very paranoid server
Mapping the Internet and intranets
120
slide 103 of
Sample report
120 slides
Mapping the Internet and intranets
120
slide 105 of
Mapping the Internet and intranets
120
slide 106 of
Mapping the Internet and intranets
120
slide 107 of
Mapping the Internet and intranets
120
slide 108 of
Internet report
120 slides
Mapping the Internet and intranets
slide 110 of 120
Mapping the Internet and intranets
slide 111 of 120
Mapping the Internet and intranets
slide 112 of 120
Mapping the Internet and intranets
slide 113 of 120
Mapping the Internet and intranets
slide 114 of 120
Mapping the Internet and intranets
slide 115 of 120
Mapping the Internet and intranets
slide 116 of 120
Intranet “Best
current practices”
We are acquiring the data to
produce a paper: statistics over a
variety of large intranets
120 slides
Some intranet statistics
from Lumeta clients
Intranet sizes (devices)
Corporate address space
Address space usage efficiency
% devices in unknown address space
7,900
365,000
81,000 745,000,000
0.01%
20.86%
% routers responding to "public"
% routers responding to other
0.14%
0.00%
75.50%
52.00%
Outbound host leaks on network
% devices with outbound ICMP leaks
% devices with outbound UDP leaks
0
0%
0%
176,000
79%
82%
Inbound UDP host leaks
% devices with inbound ICMP leaks
% devices with inbound UDP leaks
0
0%
0%
5,800
11%
12%
Mapping the Internet and intranets
% hosts running Windows
36%
slide 118 of 120
84%
Mapping the Internet and intranets
slide 119 of 120
Mapping the
Internet and
Intranets
Steve Branigan, Hal Burch, Bill
Cheswick
[email protected]
120 slides