Transcript 15-744: Computer Networking

15-744: Computer Networking
L-23 Network Intrusion Detection Systems
(NIDS)
NIDS
• Background
• Bro: A NIDS
• NIDS Traffic Normalization
• Honeycomb: NIDS signature generation
2
Recall: Network “101” vs. Reality
Traditional view: “Dumb”
network
Reality: Lots of in-network processing
Appliances or Middleboxes:
IDS, Firewall, Proxies, Load balancers….
3
Recall: Middleboxes Galore!
Data from a large enterprise
Type of appliance
Survey across 57 network operators
Number
Firewalls
166
NIDS
127
Media gateways
110
Load balancers
67
Proxies
66
VPN gateways
45
WAN Optimizers
44
Voice gateways
11
Total
Middleboxes
636
Total routers
~900
APLOMB (SIGCOMM’13)
4
Firewalls
• Network-level firewalls:
• Limit access to the network
• Installed at perimeter of the network
• Allows traffic specified in the policy
• Drops everything else
Internal Network
Firewall
Internet
5
Typical Firewall Configuration
• Internal hosts can access
DMZ and Internet
Internet
• External hosts can access
DMZ only, not Intranet
• DMZ hosts can access
Internet only
DMZ
X
X
• Advantages?
• If a service gets
compromised in DMZ it
cannot affect internal hosts
Intranet
6
Intrusion Detection Systems (IDS)
• Firewalls allow traffic only to legitimate hosts and
services
• Traffic to the legitimate hosts/services may
contain attacks
• Solution?
• Intrusion Detection Systems
• Monitor data and behavior
• Report when identify attacks
7
Firewall vs. NIDS
• Firewall
• Active filtering
• Fail-close
Internet
NIDS
• Network IDS
• Passive monitoring
• Fail-open
FW
Internet
• Advantages and
disadvantages?
8
IDS justification
• Comments on IDS
• “IDS easy to circumvent”
• “Installing IDS is useless”
• “Impossible to design bulletproof IDS”
• Responses
• IDS is one component of a security system:
Prevention, Detection & Recovery, Redundancy
• Increases difficulty of successful attack: “Raising the bar”
• If system defends against 95% of attackers, we can
concentrate on the remaining 5%
9
Evolution of the IDS
• Notion that audit/log data might help to discover
abuse/misuse dates to early 1970s
• Framework for analysis established by Jim Anderson in
1980 in “Computer Security Threat Monitoring and
Surveillance”
http://seclab.cs.ucdavis.edu/projects/history/CD/ande80.pdf
• Computers can generate lots of log/audit data
• Early analysis techniques were based on analysis of this
audit data, and used a mix of statistical and AI based
techniques
10
Evolution of the IDS
• By the late 1980s, networked computers were increasingly
common and many intrusions involved external access via
a network
• The Network Security Monitor developed at the University
of California Davis was the first IDS to work directly with
network data as opposed to log data
• Essentially a packet sniffer feeding data to an analysis engine
11
Evolution of the IDS
• Today, the vast majority of IDS systems are
based on the analysis of network traffic.
• There are numerous commercial offerings as well
as a number of research systems and systems
offered by the open source community.
• Increasingly, the administration of IDS systems is
integrated into system and network management
frameworks.
12
Types of IDS
Signaturebased
Anomalybased
Host-based
Network-based
13
Signature-based IDS
• Characteristics:
• Uses known pattern
matching to signify
attack
• Can identify intrusions from packet header/data
• May use Boolean operators in rule set
• ‘this_string’
• ‘this_variable’ AND ‘that_number’
• ‘this_string’ AND ‘that_variable’ NOT ‘that_tcp_flag’
14
Signature-based IDS
• Advantages
• Widely available
• Easy to implement
• Easy to update
• Example?
15
Signature-based IDS
• Disadvantages
• Cannot detect attacks for which it has no signature
• Must be updated for each new attack and attack variant
• Lag time from new exploit to update can be dangerous
• ‘New’ attack variant can be created by changing a single
string
• May be resource intensive
16
Anomaly-based IDS
•Characteristics
• Uses statistical models or a
machine learning engine to
characterize normal usage
behaviors
• Recognizes departures from
normal as potential intrusions
17
Anomaly-based IDS
• Advantages
• Can detect attempts to exploit new and unforeseen
vulnerabilities
• Can recognize unusual traffic based on a number of
characteristics:
• Payload
• Source address
• Time
• Can recognize authorized usage that falls outside the
normal pattern
18
Anomaly-based IDS
• Disadvantages
• Generally slower, more resource intensive
compared to signature-based IDS
• Greater complexity, difficult to configure
• Higher percentages of false alerts
• Link between abnormal and intrusive may be weak
19
Anomaly-based IDS
• Typical deployment environments
• Currently few typical deployments
• Anomaly-based IDS considered immature, too
error-prone for widespread use
• Example:
• DDoS area
20
Host-based IDS
• Characteristics:
• Runs on a single host
• Can analyze audit-trails, logs, integrity of files
and directories, etc.
• Tripwire, Fcheck
• RealSecure® Server Sensor
• Research systems such as Emerald
• May report to central
administrative console
21
Host-based IDS
• Advantages:
• Relatively easy to deploy and to manage
• Only one machine is involved
• May require only one administrator
• Creates single source of log and alert information
• Generally not resource intensive - in most
cases
• Often will not require CPU, memory, etc. beyond
what is needed for OS and applications
22
Host-based IDS
• Disadvantages:
• Works well for a single machine; extremely
labor-intensive to monitor multiple machines
each running a host-based IDS
• If the host is compromised, the IDS may cease
to function and thus no more alerts will be
generated
23
Host-based IDS
• Typical deployment environments:
• A single mission-critical machine
• User’s desktop machine
24
Network-based IDS
• Characteristics:
• Network monitor
• Passively captures traffic and inspects it
• Can also function in a client-server model
• Sensors are located on multiple machines across the network
• All sensors feed data to console
• Console machine handles logging and alerting
25
Network-based IDS
• Advantages
•Positioned properly, can test effectiveness of
firewalls, router access lists, etc.
•Can monitor multiple machines from one physical
and logical location
•Console can generate an alert if a monitored
machine/network has ceased to send information
•Operator can see patterns in traffic
• Amount
• Type
26
Network-based IDS
• Disadvantages:
• Since it is capturing all network packets, can produce
large log/alert files
• Can be difficult to cull through vast amount of information
• Console machine generally must be quite powerful,
similar to a workgroup server
• If console machine goes down then multiple machines
may be left unmonitored
• Communication from sensors to console may increase
overall network traffic levels
27
NIDS
• Background
• Bro: A NIDS
• NIDS Traffic Normalization
• Honeycomb: NIDS signature generation
28
Bro: Detecting intruders in real-time
• Bro is a standalone NIDS developed by Vern Paxson
• Designed to keep LBL an open environment (to resist the
need to install a firewall)
• Goals
•
•
•
•
•
•
High-speed monitoring, no packet drops
Real-time notification
Separate mechanism from policy
Extensibility
Simple to use, guard against mistakes
Tolerate attacks on NIDS
• More powerful than Snort, but less popular. Why?
• Misuse detection (signature-based) or anomaly detection
(specification-based or statistical-based)?
Paxon, 1998
29
Bro system architecture
Policy script
Alerts/notifications
Policy Script Interpreter
Event control
Complexity
of operations
increases
Event stream
Event Engine
tcpdump filters
Volume of data
decreases
Filtered packet stream
libpcap
Packet stream
Network
30
libpcap Layer
• Only passes relevant packets to Event Engine
• Uses BSD Packet Filter (BPF) to efficiently filter
packets
• Filter rules
• tcp port finger or tcp port ftp or tcp port telnet or
port 111 or tcp[13] & 7 != 0
31
Reminder: TCP header format
32
Event engine
• State for each connection, based on <SrcIP, SrcPort, DstIP,
DstPort>
• If state not present, allocate fresh state
• TCP processing
• Update state based on SYN/FIN/RST flags
• Process acknowledgment
• SYN generates a timer event, if nothing happens after 5 min,
generate connection_attempt event
• UDP processing
• Initial packets generate udp_request and udp_reply
events
33
Policy script interpreter
• Clear separation of event generation and
response to achieve clear separation between
mechanism and policy
• Advantage: extensibility (adding a new protocol
analyzer and new event handler usually separate
from other components)
• Event are stored in a FIFO queue and processed
sequentially
• Policy script interpreter executes event handlers
34
Bro policy scripts
• Goal: A clear and error-free language
• Written in a specialized language:
•
•
•
•
•
Network types (IP addresses, connections, protocol, etc.)
Typed constants, variables
Network operators (comparison, ranges, etc.)
Control statements (IF/THEN, etc.)
Regular expressions
• It can
• Generate alerts
• Call exterior programs
35
Bro policy scripts:
variables and operators
if ([H, S] in allowed_services)
… it’s okay …
36
Bro policy scripts:
Statements, functions, and events
37
Offline Analysis
• Checkpointing in order to
• Reclaim memory of dormant connections
• Offline-analysis
• Logs maintained for a long time for
• Forensics on past break-ins
• Complex analysis that would be too expensive to be
real-time
38
Attacks on Bro
• Overload attack
• Send packets that match filters
• Send packet streams that generate events
• Try to generate events that lead to recording to disk
• Defense strategy 1: Assume policy script is secret.
• Good assumption?
• Defense strategy 2: Lower the load (e.g., stop
capturing HTTP traffic)
• Effective if attacker does not know how Bro lowers load
39
Attacks on Bro monitor
• Crash attack
• Find packet sequence that crashes monitor
• Exhaust memory, disk resources
• Defense strategy 1: Careful code analysis
• Defense strategy 2: OS-level watchdog timer with
subsequent packet capturing (sacrificing real-time
detection)
40
Attacks on Bro monitor
• Subterfuge attack
• Mislead Bro: find traffic patterns that Bro and the end system
interpret differently
• Example: Carefully setting TTL field
• Defense strategy: Traffic normalization
41
Application-specific processing
• Case studies:
•
•
•
•
Finger
FTP
Portmapper
Telnet
• FTP Example:
• Match username against sensitive IDs
• Check access to sensitive files
42
NIDS
• Background
• Bro: A NIDS
• NIDS Traffic Normalization
• Honeycomb: NIDS signature generation
43
NIDS: Evasion and Normalization
• Problems
• NIDS only has partial knowledge of what traffic
the host sees (e.g., TTL expires, MTU)
• Ambiguities in TCP/IP (e.g., Overlapping IP &
TCP fragments)
• Different OS implement standard differently
• Approach: traffic normalization
Handely et al., 2001
44
Small TTL attack
NIDS sees:
A
T
T
I
A
C
K
NIDS
Internet
Host
Attacker’s data stream
A
T
T
I
A
C
K
End-host sees:
A
T
T
A
C
K
same TCP seq #, “I” has short TTL
45
Fragmentation overlap attack
NIDS sees:
A
T
T
A
I
C
K
NIDS
Internet
Host
Attacker’s data stream
A
T
T
A
I
C
End-host sees:
K
A
T
T
A
I
C
K
same TCP seq #
or same IP frag offset
46
Approach: traffic normalizer
• Introduce “bump in the wire”: traffic normalizer
to evade protocol ambiguities
NIDS
Internet
Normalizer
Host
47
Alternative approaches
• Host-based IDS
• Loses the advantages of monitoring the entire site
cheaply
• Major deployment and management efforts
• Detailed Intranet map
• Required knowledge of every OS and app
• Bifurcating analysis
• If the NIDS does not know which of the two
interpretations the end system may apply to an
input packet, split the analysis context
• State explosion?
• Aside: New opportunities given SDN/NFV?
48
Normalization tradeoffs
• Normalization vs. protection
• Off-load firewall/NIDS functionality
• End-to-end semantics
• Drop overlapping IP/TCP fragments
• Increase TTL in packets with low TTL
• What applications does this break?
• Keeping state
• “Fail closed” possible, given sufficient state:
• Terminate dormant connections first
• DoS through state exhaustion is a challenge
• Inbound vs. outbound traffic
• Assumes one side of the connection is trusted
49
Real-world considerations
• Cold start
• Normalizer will reboot periodically
• Normalizer lacks knowledge of ongoing connections
• Need to carefully design normalizer not to disturb established
connections
50
Real-world considerations
• Attacks against normalizer
• State exhaustion attacks
• Note: occasionally dropping a packet is ok, IP is unreliable
• SYN flooding, ACK flooding (allocate state to deal with cold
start), unacknowledged data flooding
• Defense strategy:
• Monitor system resources to determine whether normalizer
is under attack
• Instantiate state only if traffic seen from an internal host
51
Identifying normalizations
• Methodology: Walk through each header field
1. Identify fields that may cause packet to be dropped/interpreted
differently by router or end-host
2. Find ways to normalize it
3. Analyze impact on protocols
52
Identifying normalizations
Example: TTL (Time-to-live)
•Solution 1: NIDS could measure the number of hops to
every end-host
• Con: Large number of states, change of routing, …
•Solution 2: NIDS may see ICMP time-exceeded-in-transit
packets
• Con: ICMP messages may be rate limited or filtered
53
Identifying normalizations
Example: TTL (Time-to-live)
• Solution 3:
• Configure the normalizer with the longest path across the
internal site.
• Rewrite the TTL field of incoming packets to the above value
• Cons:
• What if a routing loop passes through the normalizer?
• Breaking semantics of traceroute
54
Identifying normalizations
• IP identification field is used for uniquely identifying
the group of fragments of a single IP datagram.
55
Identifying normalizations: IP Identification
• IP ID field used for
stealth port scanning
• Permute IP ID
deterministically
prevents internal
hosts from being
misused as patsies
• Reliable RST
protects internal
hosts from being
victims
56
NIDS
• Background
• Bro: A NIDS
• NIDS Traffic Normalization
• Honeycomb: NIDS signature generation
60
Honeycomb: Motivation
• NIDS work based on signatures
• How to generate signatures to begin with?
• Common practice is manual and expertisebased
• Can we do better?
Kreibich and Corwcroft, 2004
61
Honeycomb: Background
• Good NIDS signatures should be
• Narrow enough; otherwise, high false positives
• Flexible enough; otherwise, high false negatives
• Honeypots: decoy computer resources to detect or
counteract computer resources
• Examples: dummy database items, dummy web
servers, …
• Key Idea behind Honeycomb:
• Traffic sent to a honeypot should be malicious
• We can extract its pattern and use it as a NIDS signature
62
Honeycomb: Architecture
• Good NIDS signatures:
63
Honeycomb: Per-packet workflow
64
Honeycomb - Signature Creation
• If there is any existing connection state for
the new packet, that state is updated,
otherwise new state is created.
• If the packet is outbound, processing stops
here.
• Honeycomb performs protocol analysis at
the network and transport layer.
65
Honeycomb – Connection Tracking
• Signature creation is based on comparing new
data to old data therefore connections and
packets must be maintained for a period of
time
• Handshake and established connections are
kept separate as not to fill up the hashtables.
66
Honeycomb – Connection Tracking
• Goal: Find attack signatures by
• finding deviation from protocols
• finding inbound traffic patterns
67
Honeycomb: Pattern matching in flow content
• Use a longest common substring (LCS)
algorithm to spot similarities in packet
payloads
68
Honeycomb: Pattern matching in flow content
• Horizontal pattern detection: happens every nth
message and applies LCS algorithm
69
Honeycomb: Pattern matching in flow content
• Vertical detection: concatenates messages then
applies LCS algorithm
• Advantage?
70
Honeycomb – Signature generation
• Signatures are indefinite and can be built
upon if they are improved
• Signature aggregation
• Example: An HTTP GET substring in attack flow
• Signatures are output in Bro and Snort-like
signatures
71
Honeycomb - Results
NetBios web
MSSQL
(Slammer worm)
Distribution of TCP and UDP traffic destination ports in packets
directed at the honeypot.
72
Honeycomb - Results
Signature Honeycomb created for the Slammer Worm
73
Honeycomb - Results
Signature Honeycomb created for the CodeRed II Worm
74
Summary
• Network intrusion detection systems is a complement to a
firewall
• Goal: Finding malicious traffic both low false positives and
false negatives
• Decoupling policy from analysis is important
• Protocols are ambiguous
• Unclear how end-hosts implement the ambiguous parts
• Can be used to evade NIDS
• Possible solution: Protocol normalization (a.k.a. protocol
scrubbing)
• Opportunities for automatic signature generation
75
Next lecture
• Privacy
• Required reading:
• Balancing Accountability and Privacy in the Network
• Skim:
• Tor: The Second-Generation Onion Router
• Optional:
• Infranet: Circumventing Web Censorship and
Surveillance
76