Transcript Using Source Models to Evaluate Worm Defenses.

Using Source Models for Evaluating Worm Defenses
Weaver, Paxson, Crosby
Using Source Models to
Evaluate Worm Defenses
Nicholas Weaver
Vern Paxson
Scott Crosby
The Problem of
Evaluating Worm Defenses
Using Source Models for Evaluating Worm Defenses
Weaver, Paxson, Crosby
• How can we learn if our proposed defenses are
effective?
– Can they detect and block a propagating worm?
– Will they disrupt our normal network traffic?
• Worm defenses must disrupt the network when they detect a
worm
• But we don’t want Network Autoimmune Disease
– Can an attacker cause our defenses to disrupt normal
traffic?
• We don’t want Induced Network Autoimmune Disease either
Page 2
Outline:
Using Source Models for Evaluating Worm Defenses
Weaver, Paxson, Crosby
• General principles of testing
– How much detail to emulate?
– Types of worm experimentation
– Types of background traffic emulation
• Trace replay
• Source models
• Live networks
• The tested system: Approximate TRW
– Distributed emulation for background
– Layer 4 empirical source models
• Testing Approximate TRW on DETER
Page 3
How Much
Detail?
Using Source Models for Evaluating Worm Defenses
Weaver, Paxson, Crosby
• Want to accurately model all details which affect the
system
– But don’t need extra detail
• So what does the system do?
– Does it act as a Layer 2 bridge looking at Layer 4 packets?
• Need very high fidelity for Layer 4 and below
– Does it look at connections or payloads?
• How does it look at payloads?
– Does it only detect & respond to attacks on an end system or in the
network?
• Where does it live in the world?
• How will this interact with the real world?
– Will it cause false positives on normal traffic?
– Will normal traffic hide attacks?
• Thus can’t just model attacks, have to model the background
Page 4
Several Ways for
Testing Worms
Using Source Models for Evaluating Worm Defenses
Weaver, Paxson, Crosby
• Simulation: Write a simulator
– Small, efficient, safe
– Can only test simulated defenses
• Emulation: Write a program which propagates messages
– Small, efficient, safe, can be run on live networks
– Only works where the defense doesn’t key in on emulation
artifacts
• Existing attacks: Launch an exploit
– Safe, can be run on live networks
– Only works when detecting attacks, not self propagation
Page 5
Ways of Testing
(Continued)
Using Source Models for Evaluating Worm Defenses
Weaver, Paxson, Crosby
• Synthetic Host: Create a deliberate host for a worm
– Safe and highly realistic
– Legal risks
• Existing Malcode: Use an existing worm and vulnerable
service
– Reasonably safe, highly realistic
– Legal risks, can’t defend against the next worm
• Modify existing malcode: Hack an existing worm
– OK safety, highly realistic, can defend against the next worm
– Major legal risks if the worm escapes
• Novel malcode: Use a new worm
– Highly risky, where do you get the worm from?
Page 6
But The Problem is
The Background Traffic
Using Source Models for Evaluating Worm Defenses
Weaver, Paxson, Crosby
• It’s easy to model the worm
– Just pick the appropriate technique for your defense
– Often only need to use the attack:
• EG, for scan detection, you don’t need to test with a worm,
you can test with nmap
– Othertimes, emulation is the best:
• Emulation is highly realistic on many levels
• But how do you know about false positives? About how
normal traffic creates false negatives by masking attacks?
– Need to have background traffic sufficient to test how the defense
will work on a real network
• Traffic needs to be suitably faithful for what the defense is looking at
– Real networks are very diverse
Page 7
Background Traffic:
Traces
Using Source Models for Evaluating Worm Defenses
Weaver, Paxson, Crosby
• Traces: Use a recording of existing traffic
– Simply replay through the device
• Advantages:
– Easy to use
– Reproducible
• Disadvantages:
– Loses some realism (Do you get the ARPs?)
– Traces are shaped by the environment where they are recorded
– Simplistic replay is insufficient
• Permute the network -> permutes how TCP flows behave
– Can’t do “What IF” experiments easily
• Add more devices than you have traces
– May be difficult to export traces
Page 8
Background Traffic:
Source Models
Using Source Models for Evaluating Worm Defenses
Weaver, Paxson, Crosby
• Create an (analytic or empirical) model of how systems behave
– Capture what the host wants to do
• On some level of abstraction: What a connection, application, host, or user
wants to do
– Then write programs which follow scripted behavior
• Advantages:
– Easy to do reproducible, what-if experiments
• Change topologies etc
– Can get some higher fidelity in some areas
• EG, network behavior (ARPs, TCP response to lost and delayed packets, etc)
– May be easier to export models
• Disadvantages:
– Can be unrealistic in other areas
• EG, no actual data
– Can be very difficult to create the proper abstraction
Page 9
Background Traffic:
The Live Network
Using Source Models for Evaluating Worm Defenses
Weaver, Paxson, Crosby
• “Just Do It”: Run the system on your
own LAN
• Advantages:
– VERY high realism
– The “Eat Your Dogfood” effect
• Disadvantages:
– Nonreproducible
– Limited vantage-points
– Limited what-if opportunities
• Highly limited by the peculiarities of the deployment
– Can be horribly disruptive when testing active defenses
• Can be mitigated by using alarms instead of actual blocks
Page 10
The Defense Under Test:
Approximate TRW
Using Source Models for Evaluating Worm Defenses
Weaver, Paxson, Crosby
• Approximate-TRW (AC-TRW) attempts to detect and
block scanning in a local-area-network
– It sits as a bump in the wire and attempts to discover scans
• A failed connection is counted as a scan (+1)
• A successful connection is counted as normal (-1)
– A connection is assumed to be a failure until the response is received
• If a system’s count gets above a threshold (+5 for high-sensitivity,
+10 for normal operation), it is blocked
• Can contain scanning worms
– EG, Code Red, Blaster, Welchia, Slammer, Witty…
• By preventing the worm from finding new victims
– But requires universal deployment in the network being protected
• Works fine in a well structured LAN
• Can’t work in the Internet core
Page 11
How Approximate TRW
Works in Click
Using Source Models for Evaluating Worm Defenses
Weaver, Paxson, Crosby
• Implemented AC-TRW as a Click element
– Click is a framework for writing packet processing programs
• Two Ethernets are brought up in promiscuous mode
– Packets are passed across the bridge unchanged, including the
MAC address
• Thus it is a layer-2 bridge
– Manipulates traffic based on Layer-4 (TCP/UDP headers) but is a
Layer-2 device on a Layer-2 network
• Detects scanning by tracking IP addresses and active connections
• If a system is scanning, its packets are dropped rather than passed
• Click is a wonderful tool for writing both packet
processors and monitors
– In-line Click can sample links without switch support
• Tested using source models and emulated worms
– Also a single-point live test on ICSI’s network
Page 12
Testing Approximate TRW:
Worm Emulation & Source Models
Using Source Models for Evaluating Worm Defenses
Weaver, Paxson, Crosby
• A large database of connection summaries from LBNL traces of
internal Enterprise traffic
– Database schemas to (ad-hoc) classify the source and destination types of
all systems
• Based on human classification of what behavior defines what type of system,
such as IDLE::Windows, Backup Software service, file server, etc
– Classification process continued until all hosts had a classification
– Each connection includes success or failure status, duration, volume,
source and destination ports, source and destination system types
• Script generator:
– For each system in the script:
• Randomly assign a system type based on the empirical distribution
• Then randomly sample the connection database for this source type
– Set the destination to be an appropriate system of the destination’s type
• Discover all ports where this node’s type ever receives valid connections
Page 13
Testing Approximate TRW:
Worm Emulation & Source Models
Using Source Models for Evaluating Worm Defenses
Weaver, Paxson, Crosby
• Model program runs on a system and replays a script
– Opens up listener ports
– For each scripted communication, contact the destination
• Currently, just TCP
– Uses real system calls, so generates real traffic (including ARP
requests)
• How connections work is perfectly modeled
• Distribution/behavior of connections is based on empirical models
• Model programs can also emulate a worm
– If it receives the message “WORM” on an open port
• Spawn 100 threads (1/second) to spread “WORM” messages on that
port to random destinations in the experiment
– From layer-4 and below, this looks perfectly like a worm
• Generates the proper failures and successes with real packets
Page 14
The Experimental
Topology
Using Source Models for Evaluating Worm Defenses
Weaver, Paxson, Crosby
• Each group of end-nodes is on
its own Ethernet network
• Each group is connected to the
core through an AC-TRW device
– Device is layer-2 transparent:
Unless packets are blocked, the entire
network is a single Ethernet network
– All systems are on the same IP subnet
• Systems are clustered based on source type
• Two special nodes: sinkall and dropall
– Sinkall acknowledges all traffic (respond-all modeler)
– Dropall is used to manage experimental runs
• Management is done through the experimental network:
ensures that the experimental network is operating correctly
Page 15
Experimenting with
Approximate TRW
Using Source Models for Evaluating Worm Defenses
Weaver, Paxson, Crosby
• The “Small” experiment:
– 100 virtual nodes running the
source-model software
• 3 nodes failed to come up properly:
Call it a “feature” and move on
– 5 AC-TRW devices which pass packets
• One false-positive when AC-TRW
is set to high sensitivity
– Script contains several scripted failures
– Represented a highly anomalous client
– Script replay is far faster than real time:
amplifies false-positives
• A worm on port 515 infects all hosts
most times
– Worm is detected and blocked, just this is above the epidemic threshold
• The large clusters, when infected, are expected to find the other clusters
– Need more devices and/or a sparser address space
Page 16
Problem Discovery:
The “MAC Cache”
Using Source Models for Evaluating Worm Defenses
Weaver, Paxson, Crosby
• The experiments are very high fidelity at Layer-4
and below
– Normal Ethernet packets, real ARP and TCP, real
switches
• A bug discovered in AC-TRW: One time, the
initiator couldn’t contact the clients
– It contacted the first 20 normally, but the 21st and
beyond blocked
• The TRW devices were blocking this traffic
• The TRW devices not on the path were counting valid TCP
SYNs as scan attempts, but not seeing the SYN/ACKs
– Why was there this false positive?
Page 17
What Happened?
Using Source Models for Evaluating Worm Defenses
Weaver, Paxson, Crosby
• The initiator’s ARP cache was still valid
– Had a proper IP->MAC binding for all recipients
• The switch’s MAC cache was invalid
– There was no longer a MAC->Port binding
• So SYN from initiator to node1:
– SYN got broadcast through the Ethernet, as the switch didn’t know where
to send it
• The SYN/ACK from node1 to initiator:
– SYN/ACK was sent point-to-point
• So devices NOT on the path saw the SYN but not the response
– So called it a scan attempt
• The Layer 2 fidelity is VERY GOOD
– Actually, its “perfect”, all are real devices with real behavior
– But because it’s the testbed, its easier to monitor and generate behavior
Page 18
Other Notes:
Know The Testbed
Using Source Models for Evaluating Worm Defenses
Weaver, Paxson, Crosby
• The testbed is really four
small testbeds
– Connected by relatively narrow pipes
• Each GigE link can only
support 10 VLANs
QuickTime™ and a
TIFF (LZW) decompressor
are needed to see this picture.
• Need to partition designs to
account for these limitations
– Partitioning is straightforward (each “mini” testbed uses different
node types) but needs to be done
• There may be opportunities to increase the inter-switch
bandwidth
– Feedback between experimenters and testbed operation
• I didn’t discover the significance until a week ago when I tried to
swap in the 100/20 experiment
Page 19
Future Work
Using Source Models for Evaluating Worm Defenses
Weaver, Paxson, Crosby
• More experiments:
– Have a 100 node with 20 defensive device experiment
• Should contain a worm better for the same topology
– Also a 200/40 node experiment
• Scalability can go even higher with some minor testbed changes
• Improve the source models
–
–
–
–
Include UDP and ICMP behavior in the script
Add more flow-based information
Better selection of end-host roles
Develop analytic models of end-hosts
• Modify AC-TRW
–
–
–
–
Fix “MAC Cache” problem
Track ARP scanning
Do something with UDP broadcast packets
Make more LAN centric
Page 20
Conclusions
Using Source Models for Evaluating Worm Defenses
Weaver, Paxson, Crosby
• Types of worm experiments
• Types of background traffic
– Choose the one which suits the experiment
• A start on source models:
– Initial experience with an empirical source-model for
testing a live defense
• Testing a defense: Approximate TRW
– It works
– False positives are reasonable
– BUG DISCOVERED: The MAC Cache problem
Page 21
Using Source Models for Evaluating Worm Defenses
Weaver, Paxson, Crosby
Page 22
Using Source Models for Evaluating Worm Defenses
Weaver, Paxson, Crosby
Page 23
Parallel
Simulation
Using Source Models for Evaluating Worm Defenses
Weaver, Paxson, Crosby
• Sometimes simulation is all you need
– Worm simulators may be deterministic, but they should specify a seed for
the pRNG
• As the actual evolution of a worm is a probabilistic process
• Run tens or hundreds of
simultaneous experiments
• Special risks:
– Effectively none
• Limitations:
300000
Number of infections
– Using DETER as a cluster
resource rather than as a
Testbed
– Allows high throughput
The simulated propagation of a codered-like worm without scaledown
Run 1
Run 2
Run 3
Run 4
Run 5
Run 6
Run 7
Run 8
Run 9
Run 10
250000
200000
150000
100000
50000
0
0
1
2
3
4
5
6
7
8
9
10
Time (h)
– Only works with simulated defenses, can’t actually test with real defenses
Page 24
11
Distributed
Simulation
Using Source Models for Evaluating Worm Defenses
Weaver, Paxson, Crosby
• Worm simulation is often time-consuming:
– Attempting to simulate a large portion of the Internet
• Distributed simulation attempts to improve this
performance
– Each system models some subset of the network
– Communications between
systems to only express
The Heterogeneous Cluster Model
Compared with Empirical Results
known-successful infection
attempts
– Allows lower latency for
single experiments
– Worse multi-experiment throughput
1
0.8
0.6
Scaled Scans/s
Scaled Infection Count
Scaled Scans/S/Worm
Simulated Scanrate
Simulated Infection Count
Simulated Scans/S/Worm
0.4
• Special risks:
– Effectively none
0.2
0
0
1
2
3
4
Minutes After Release
Page 25
5
Distributed
Emulation
Using Source Models for Evaluating Worm Defenses
Weaver, Paxson, Crosby
• Nodes on the simulator contain a small daemon program
– This daemon waits for requests and then responds by sending out
messages like a worm would (an "honor system" worm)
• Looks to the network exactly like a worm
– Allows testing of network-based defenses
• Doesn't look like a worm on the end host
• Can't self propagate: No self propagating code, just self-propagating
messages
• Special risks:
– low:
only real risk would
be net load if the
traffic is inadvertently routed
• Testing:
– As long as the emulated components are what the defense looks for, the
generated traffic is perfect for testing
– If the defense will key in on the emulation structure’s artifacts (eg, XML
headers) rather than the emulated behavior, it isn’t effective
• Modify the defense to ignore these artifacts
Page 26
Synthetic
Hosts
Using Source Models for Evaluating Worm Defenses
Weaver, Paxson, Crosby
• Design MacroSloth® FubarWare® server with a known
security flaw
– Eg, a deliberately engineered stack overflow
• Modify/build a worm to exploit the synthetic vulnerability
– A worm can't survive without a host program
• Looks to both the network AND the end hosts like a worm
– Can be used to test both host based and network based defenses
• Special risks:
– Network risk: low
Equivalent to distributed emulation, escaped traffic may be a load
problem and information leak, but no infection risk
– Legal risks: Moderate to High
Call your lawyer first: you are writing/modifying malcode.
What if the source code is used by an attacker in a later worm?
Page 27
Existing Attacks
Using Source Models for Evaluating Worm Defenses
Weaver, Paxson, Crosby
• Just launch some non-propagating scripts at the target
– You can modify them to your heart’s content
• Looks like an attack to both the end-host and the network
– So if your system relies on detecting the attacks, not a worm, this
works just fine
• Many proposed worm-detection systems are detecting worm-related
attacks
• Network risks: Very low
– Targeting is under your control: If you attack someone, you meant
to or you screwed up, not the program
• Legal risks: Very low
– Has anyone been prosecuted for providing scripts to the 31337
SKR1PT K1DD13Z?
• Especially if you didn’t provide the script, but it somehow leaked out?
Page 28
Existing Malcode
Using Source Models for Evaluating Worm Defenses
Weaver, Paxson, Crosby
• Install an old & vulnerable software
on the nodes
– Release an old, known worm
– Or simply expose vulnerable services
to the Internet
• Looks like a worm to both the network and the
end hosts
– Although existing defenses may need to have specific
knowledge removed if testing 0-day defenses
• Special risks:
– Network risk: relatively low
Anyone who can get infected will be infected
by somebody soon anyway
– Legal risk: High
They may be going to be infected anyway, but
it way YOUR escaped experiment which did it!
Page 29
Modified Existing Malcode
Using Source Models for Evaluating Worm Defenses
Weaver, Paxson, Crosby
• Install an old & vulnerable software
on the nodes
– Release a modified version of an old worm
– Or a new worm for the old vulnerability
• Looks like a worm to both the network
and the end hosts
– Again, defenses need knowledge removed
• Special risks:
– Network risk: Moderate
Anyone who can get infected will be infected
by somebody soon anyway, but the damage
may be greater depending on payload
– Legal risk: Very High
The FBI would like to talk to you about the
worm that got released
++
Page 30
Novel Malcode
Using Source Models for Evaluating Worm Defenses
Weaver, Paxson, Crosby
• Something new and evil is in existence…
– What the @#)$(* does it do?
– It's not out there (yet)
• Special risks:
– Network risk: Potentially catastrophic:
This worm might not exist in the
wild
– Legal risks: Potentially catastrophic:
"Ooops, I just destroyed the Internet…"
• DETER is currently not set up to handle
such experiments yet
Page 31