Transcript Report on Multi-agent Data Fusion System: Design and

Report on Intrusion Detection
and Data Fusion
By Ganesh Godavari
Outline of the talk
•
•
•
•
Intrusion Detection
Data fusion
Motivation
Traditional models
Intrusion Detection & Data Fusion
• Intrusion Detection System
– Protect availability and provide confidentiality and
integrity of critical information infrastructures
• Data Fusion : task of data processing aiming at
making decisions on the basis of distributed data
sources specifying an object
• Data sources
– Different physical nature
• Electromagnetic signals, sensor data…
– Different accuracy
• Reliability?
Motivation & challenges
• Threat analysis
– Known & unknown Pattern templates, traffic
analysis, statistical-anomaly detection and
state based detection
• Provide Reliability
– Reduce false alarms, increase user
confidence
Characteristics of IDS based on
Waltz model
•
Detection performance
– Detection characteristics like false alarm rate, detection probabilities and ranges
for an intrusion characteristic
•
Spatial/temporal resolution
– Ability to distinguish between two or more intrusions in space and time
•
Spatial coverage
– Span of coverage or field of view of the sensor
•
Detection and Tracking modes
– Mode of operation of the sensor i.e. staring or scanning; single or multiple target
tracking
•
Target Revisit Rate
– Rate at which an intrusion is revisited by the sensor to perform measurements
•
Measurement Accuracy
– Statistical probability that the measurement or observation is accurate
•
Measurement dimensionality
– Number or measure of variables between target categories
Contd..
• Hard Vs. Soft Data Reporting
– Status of the sensor reports
– can a decision be made without correlation or
does the sensor require confirmation
• Detection/Tracking Reporting
– Characteristic of the sensor to report
individual events or maintain a time-sequence
of the events or events
Hierarchy of IDS Data Fusion
Inferences
Threat Analysis
Situation Assessment
Behavior of Intruder
Identity of Intruder
Rate of Intrusion
Existence of intrusion
Types of Inference
High
Medium
Low
Level of Inference
Data fusion and OODA model
• Decision support systems and data fusion
system need to be tightly coupled
• Decision support system must
– Observe
• Collection of data from sensors, network sniffers, system log
files
– Orient
• Data mining concepts of learning unknown characteristics.
– Decide
• Refinement of knowledge into threat knowledge and
determination of appropriate counter measures
– Act
• Automated and human responses to threat/vulnerability
OODA mapping
• Three levels of abstraction
– Data
• Measurement and observations
– Information
• Data placed in context, indexed and organized.
– Knowledge/intelligence
• Information explained and understood
Intrusion Detection Data Fusion
Situational knowledge
used for
Analyzing objects and groups
against existing
Intrusion detection
templates to provide
assessment
Data is correlated in time
Data is assigned weighted
Metrics based on relative importance
Alignment to a common
frame of reference
Calibration and filtering
Observation identifiers,
time of observation,
and description
Correlation between level 3 threat assessment and security
Policy and objectives determine the implications of current
Situation base. The whole process is refined via level 4 resource
Management based on situational awareness
This ID model is based on deductive process
used to detect previously known patterns in
many sources of data
notes
• Situational data is collected from sniffers and other ID sensors with
primitive observation identifiers, time of observation and
descriptions. This raw data requires calibration or filtering known as
level0 refinement. All the three measurements must be aligned to a
common frame of reference. This alignment is known as level1
object refinement. Here data is correlated in time and data is
assigned weighted metrics based in relative importance.
Observation may be associated and paired and placed in context in
an information base. Situation refinement provides situational
knowledge and awareness. Situational knowledge is used to
analyze objects and aggregated groups against existing intrusion
detection templates to provide assessment of the current situation
and suggest or identify future threat attacks. Correlation between
level3 threat assessment and security policy and objectives
determine the implications of the current situation base. The entire
process is refined via level 4 resource management based on
situational awareness.
Technical terms !!
• Data mining/knowledge discovery : search
for hidden patterns based on previously
undetected intrusions to help develop new
detection templates
• Data fusion Vs data mining
– Inference method and temporal perspective
Intrusion detection data mining
notes
Raw data from relevant network
management and intrusion detection
systems are collected and indexed in the
data warehouse. Major Technical issue is
how to reconcile the raw data from many
different formats and inconsistent data
definitions.
Process involved in intrusion
detection data mining
• Data cleansing
– check to insure the collected data is in correct ranges
and limits
– evaluate overall consistency of the data
– ensure hierarchical relationship exists
• Data selection and transformation
– Initial sets that will be used for data mining are
selected
• Data mining
– Performed on selected data sets in either manual or
automated modes
Data mining operations
characterized by waltz
•
Clustering
– Data is segmented into subsets that share common properties
•
Association
– Analysis of both the cause and effect and structure relationship between data
sets
•
Statistical Analysis
– Determine the likelihood of characteristics and association in selected data sets
•
Rule Abduction
– Development of IF-THEN-ELSE rules that describe associations, structures and
test rules
•
Link or tree abduction
– Performed to discover relationships between data sets and interesting
connecting pattern properties
•
Deviation Analysis
– Locate and analyze deviations from normal statistical behavior
•
Neural Abduction
– Process of training artificial neural networks to match data, extract node weights
and structure (similar to abducted rule sets)
Intrusion detection data mining
contd..
• Discovery modeling
– Information is mined into new ID knowledge
– Development of refined models to predict
future events based on historical data
• Visualization
– human process of pattern recognition
Questions
?
References
• Intrusion detection systems and multi
sensor data fusion: creating cyber
situational Awareness by Tim Bass
Communications of the ACM (2000)