Transcript Project Overview - Complex event processing

Defeating Large Scale Attacks:
Technology and Strategies for
Global Network Monitoring
The NetViewer Experiment
PAVG in collaboration with
Networking Systems
R. Kamath, E. Jang, D. Luckham
Project Goals
Detect system misuse on a global
level
 User re-configurable and flexible
 Hierarchical organization of monitors
 Correlation of distributed monitors
 Monitor activity from diverse sources
 Monitor at multiple levels of
abstraction

2
Stanford NetViewer Experiment

Uses Stanford Rapide Toolset
 Uses Complex Event Processing
technology
 Uses Talarian’s SmartSocketsTM
middleware for distributed processing
FOR MORE INFO...
Http://pavg.stanford.edu/rapide
Http://pavg.stanford.edu/cep
3
NetViewer Experiment setup
NetFlow FlowCollector
Intrusion
Monitor
Map
CEP
Logger
Flow
Efficiency
Monitor
Filter
Map
PassThrough
Monitor
Log Files
Cisco
NetFlow
FlowCollector
Complex Event
Processing
Monitor Views
4
SUNet Campus Network
Internet
Core
Gateway
Undergrad
Education
Redundancy
Gateway
Grad.
Education
Admin
Host 1
Business
School
Redundancy
Gateway
Stanford
Hospital
Computer
Center 1
Redundancy
Gateway
Admin
Host 2
To
FlowCollector
Redundancy
Gateway
Computer
Center 2
Core
Gateway
Internet
5
Complex Event Processing

Accept network ‘events’ from any source
– CISCO NetFlow FlowCollector, tcpdump

Correlates events based on content and
temporal relationship between events
 Event Processing Agents (EPAs)
connected in an Event Processing Network
(EPNs)
 Both post-mortem and real-time
processing
6
Event Processing Agents (EPAs)
-- Loggers and Filters

Loggers
– Convert external data into events
– E.g. CISCO FlowCollector logs to events

Filters
– Select a subset of events based on pattern
– E.g. Only connections from Stanford hosts
7
EPAs-- Maps and Viewers

Maps
– Search for patterns in input events
– Generate appropriate output events
– E.g. look for IP scans and generate alarms

Viewers
– Graphical display of data in events
– Tables, Bar Graphs
8
RapNet User interface

RapNet
–
–
–
–
–
Graphical Interface to NetViewer tool
Easy access to EPA and EPN library
Easy re-configuration of EPAs
Easy modification of EPNs
Construct new EPNs using EPAs
9
NetViewer running under RapNet
10
Hierarchical monitoring

Two types of hierarchy
– Abstraction hierarchy
• NetViewer monitors data at different abstraction
levels
– Topological hierarchy
• NetViewers at different locations

NetViewers at different levels
communicate using SmartSockets
middleware
 General case: arbitrary network of
monitors
11
Network Abstraction Hierarchy

Application layer
– Host-based monitoring
– Data exchanged by SMTP, TELNET, FTP, HTTP
protocols

Transport layer
– Data exchanged by TCP/IP suite of protocols

Network layer
– Router-based monitoring
– IP and UDP packets
12
Topological Hierarchy -- multiple
gateways example

Distributed processing of data
 Each NetViewer at level 1 monitors data
from a different gateway
 Results (e.g. top 10 IPs) from level 1
NetViewers sent to level2 NetViewers
 Level 2 NetViewers correlate the results of
level 1 NetViewers
– E.g. compute top 10 IPs over all gateways
13
Distributed monitoring on SUNet
Receiver running
NetViewer 3
Sender running
NetViewer 1
Core gateway
Admin host
Admin host
SmartSockets over SUNet
Sender running
NetViewer 2
Admin host
Press gateway
14
Current Status -- EPAs

Library of Event Processing Agents (EPAs)
– Traffic categories
• Web, Mail, DNS, ftp …
– Scan Detectors
• IP scan, Port scan
– Policy violation detectors
• Access to restricted hosts
• Access to restricted ports on hosts
– Traffic event filters
• Web, Mail, Hosts, Networks
15
Current Status -- EPNs

Library of Viewers
– Tables
– Bar graphs
– Pie charts

Library of Event Processing Networks
(EPNs)
– Network of EPAs
– Graphical viewers to display results
16
Research Directions

Hierarchical monitoring
– Data sources from different layers
– Correlation of results from multiple NetViewers

Accept more input formats
 Distributed processing
– Assign individual EPAs within a NetViewer to
run on different machines

Expand EPA library
– Work on mail spam detection
17
Experiment results on SUNet

NetViewer used to process router logs
– Real-time performance of about 1000 log
records/sec

Generated traffic statistics
– Top IPs by packets or bytes
– Classification of traffic into categories such as
internal/external, web/mail/DNS etc.

Intrusion detection
– Detected IP and port scans
– Well-known attack signature e.g. finger attack
18
Related projects -- CIDF

Correlates information from multiple
intrusion detectors
– Reduces false alarms
– Prioritizes network warnings

Part of the DARPA Common Intrusion
Detection Framework (CIDF)
– Multiple intrusion detectors in cyber battlefield
FOR MORE INFO...
Http://seclab.cs.ucdavis.edu/cidf
19
Overview of the CIDF project

Goal
Experiment with semantic interoperability of different
components in CIDF

Groups Involved
Group A: produces GIDOs, questions, detailed English
description of the events, and the answers to the
questions.
Group B: gets 10 scenarios and produces 10 GIDOs
describing the scenarios.
Group C: gets the questions and high level scenarios from
B and builds the code. Then, gets 10 GIDOs and
produces text answers to the questions - Stanford
belongs to group C.
20
Processing GIDOs with CEP agents
Question
Target ID
Description
Search Pattern
Input
GIDO




CIDFLogger
CMEvent
Builds events
Points to C++ GIDO tree
Question Agent
Process the C++ GIDO
tree w ith Question
Answer
to user
Make each GIDO an event
Use (and fix) our existing cidfLogger
Separate event processing agent called “Qagent”
Provides flexible way of handling GIDOs
21
Qagent



Finds an answer from a given GIDO and a query
pattern.
Qagent traverses the tree to find all the possible
paths that can lead to the answer.
The question is fed to the program as a text file
with two sections:
– The input file may contain a text description
– Patterns to be searched from the tree.
The pattern lines are preceded with “@question:”

Implemented in C++ (I.e. not map language)
– Easier tree traversal
– File input
22
Pattern Language

Lists of SID separated by comma. Answer is the
subtree after the last SID
Attack,AttackSpecifics,IPV4Address

“#true” or “#false” to get the sibling SID rather
than child SID of the last SID for the answer.
ByMeansOf,Attack#true

‘^’ to indicate that the SID is one of the base SID
that applies to all other parts of the pattern
^And,^Copy,Outcome,ReturnCode?success=FileSource,File
Name
23
Examples
Event1
Brief description:
This is an attack that began on
Monday, May 24, at 12:44.
What is the certainty of this
attack?
@question:
Attack,Certainty
( Attack
( Initiator
( IPV4Address 134.52.160.76 )
)
( Target
( IPV4Address 134.52.160.114 )
)
( AttackSpecifics
( Certainty 100 )
( Severity 50 )
( AttackID 000000020000000f )
)
( When
( BeginTime Mon May 24 12:44:17
1999 PDT )
( EndTime Mon May 24 12:44:18 1999
PDT )
)
)
24
Team Members





Rajesh Kamath (rkamath@pavg)
David Luckham (dcl@pavg)
Eunhei Jang (ejang@pavg)
John Kenney (jjk@pavg)
James Vera (vera@pavg)
25