Transcript Fast Packet Retrieval Project

IEEE VAST Challenge 2009
Presented By Grant Vandenberghe
(TEAM DRDC)
[email protected]
Defence Research and
Development Canada
Recherche et développement
pour la défense Canada
Canada
Introduction
The solutions to these challenges were produced using an application
called the Network Traffic Explorer (NTE) originally presented at
VizSec 2008. The NTE provides an application front-end for a large
library of packet analysis and graph drawing tools.
The NTE allows the user to write short scripts to produce a wide
variety of diagrams. The solutions to the VAST challenges were
produced using a series of custom scripts written specifically to solve
them.
NTE Application Front End
Packet Analysis
Library
Graph Drawing
Library
MATLAB
Defence R&D Canada – Ottawa • R & D pour la défense Canada – Ottawa
Mini-challenge #1 – Badge and Network
Traffic
Load data into MATLAB
Convert data into
meaningful data format
Sanitize proximity data
Transfer the VAST data
into NTE data structures
Associate physical space
with employee id
The following steps were
followed to process the data
Time strings (YYYY/MM/DD@hh:mm:ss) converted
to a real numeric value.
IP addresses converted to integer values
Code created to compensate for double badging,
piggybacking, double entry double exit, and end of
day events
VAST
NTE Standard Session Data Structure
:
Employee ID
Run data queries to
detect abnormal activity
Plot The Result
Defence R&D Canada – Ottawa • R & D pour la défense Canada – Ottawa
Sanitizing Notes
Although the challenge instructions indicated that
“employees are required to prox into and out of the
restricted area” - this did not prove to be true.
For example, Employees 38 and 49 entered the classified
room twice without leaving it. At several different instances
Employee 30 left the secret room without entering it.
Although employees do not badge out of the building, it is
assumed they leave the building 10 minutes after the last
activity of the day. In cases where the employee leaves for
lunch the last activity prior to lunch is used.
The following employees piggybacked into the building:
0,7,8,13,27,36,37,38,39,48,49,50,51,54,55,58, and 59.
There is a small amount of time skewing between the
proximity and session traffic. It is assumed that sessions
starting a minute after entering the secret room are
associated with time skewing.
Defence R&D Canada – Ottawa • R & D pour la défense Canada – Ottawa
Hypothesis –Employees Should Only Be In
One Place At Once
After carefully reviewing the data it was noted that there
are instances where an employee’s computer was starting
outgoing sessions while the employee was in the secret
room. This event is assumed to be significant since the
employee’s computers do not transmit data after the end
of the day.
(Note: In real life the software installed on the users box will call home for a
variety of reasons both legitimate and otherwise)
Defence R&D Canada – Ottawa • R & D pour la défense Canada – Ottawa
Locations Of Abnormal Activity
The NTE freedraw
function allows the
user to overlay
vertices on top of a
gif/jpeg image.
The red dots on the
diagram indicate the
location of abnormal
activities. As can be
clearly seen the
activity does not
have an obvious
pattern.
Defence R&D Canada – Ottawa • R & D pour la défense Canada – Ottawa
Layered Timeline Plot
The layered
timeline function
allows the overlay
of multiple time
events on a
GANTT chart
Zooming in exposes the
details. The green line
indicates an active session
while the employee was
inside the classified room
(purple bar)
Defence R&D Canada – Ottawa • R & D pour la défense Canada – Ottawa
Unusual Communication Patterns
The layered timeline plot shows several events where an employee
was both in the classified room and starting new sessions at his
desk. Shown below is a list of anomalies.
User
User
User
User
User
User
User
User
User
15’s
16’s
16’s
30’s
31’s
41’s
41’s
52’s
56’s
computer
computer
computer
computer
computer
computer
computer
computer
computer
at
at
at
at
at
at
at
at
at
(2008/1/31@13:10)
(2008/1/10@16:01)
(2008/1/15@16:14)
(2008/1/24@08:06) ???? Does not look like others
(2008/1/10@14:27)
(2008/1/17@12:12)
(2008/1/29@16:08)
(2008/1/31@09:41)
(2008/1/29@15:41)
Defence R&D Canada – Ottawa • R & D pour la défense Canada – Ottawa
Using the NTE to Dig Into The Dataset
The NTE application front end takes
user input through a GUI interface and
then both displays and runs the
command on the background library.
Using the NTE reporting tools it was
found that most anomalous sessions
sent large volumes of information to 1
IP address
(NTE MAIN GUI)
print_session_summary_ev(SSN_SUM,'ALL','CLIENT_IP=37.170.100.16&SSN_START_TIME>2008/1/15@16:05:00&SSN_START_TIME<2008/1/15@16:20:00');
ID=53950 2008-01-15 16:14:34.563000 Dur=35.094373 37.170.100.16:0 > 100.59.151.133:8080 tcp Pkts=0> 1< Bytes=6773214> 24661< No_FIN_RST
By querying this IP address we found even more similar activity.
BAD_SSN_NUM=print_session_summary_ev(SSN_SUM,'ALL','SERVER_IP=100.59.151.133');
ID=26896 2008-01-08 17:01:33.001000 Dur=46.060503 37.170.100.31:0 > 100.59.151.133:8080 tcp Pkts=0> 1< Bytes=8889677> 12223< No_FIN_RST
ID=36424 2008-01-10 14:27:12.238000 Dur=33.902674 37.170.100.31:0 > 100.59.151.133:8080 tcp Pkts=0> 1< Bytes=6543216> 22315< No_FIN_RST
ID=37370 2008-01-10 16:01:53.956000 Dur=44.264896 37.170.100.16:0 > 100.59.151.133:8080 tcp Pkts=0> 1< Bytes=8543125> 12312< No_FIN_RST
ID=53950 2008-01-15 16:14:34.563000 Dur=35.094373 37.170.100.16:0 > 100.59.151.133:8080 tcp Pkts=0> 1< Bytes=6773214> 24661< No_FIN_RST
ID=54444 2008-01-15 17:03:29.342000 Dur=49.291777 37.170.100.31:0 > 100.59.151.133:8080 tcp Pkts=0> 1< Bytes=9513313> 14324< No_FIN_RST
ID=62646 2008-01-17 12:12:10.990000 Dur=19.062808 37.170.100.41:0 > 100.59.151.133:8080 tcp Pkts=0> 1< Bytes=3679122> 24423< No_FIN_RST
ID=65499 2008-01-17 17:57:19.341000 Dur=30.432881 37.170.100.18:0 > 100.59.151.133:8080 tcp Pkts=0> 1< Bytes=5873546> 25234< No_FIN_RST
ID=72065 2008-01-22 08:50:21.894000 Dur=51.732218 37.170.100.13:0 > 100.59.151.133:8080 tcp Pkts=0> 1< Bytes=9984318> 42231< No_FIN_RST
ID=76928 2008-01-22 17:41:55.862000 Dur=45.976596 37.170.100.16:0 > 100.59.151.133:8080 tcp Pkts=0> 1< Bytes=8873483> 16778< No_FIN_RST
ID=83558 2008-01-24 09:46:34.452000 Dur=40.546378 37.170.100.10:0 > 100.59.151.133:8080 tcp Pkts=0> 1< Bytes=7825451> 23783< No_FIN_RST
ID=83854 2008-01-24 10:26:31.321000 Dur=28.661523 37.170.100.32:0 > 100.59.151.133:8080 tcp Pkts=0> 1< Bytes=5531674> 22479< No_FIN_RST
ID=87501 2008-01-24 17:07:34.775000 Dur=50.427031 37.170.100.20:0 > 100.59.151.133:8080 tcp Pkts=0> 1< Bytes=9732417> 42347< No_FIN_RST
ID=103076 2008-01-29 15:41:32.763000 Dur=51.941731 37.170.100.56:0 > 100.59.151.133:8080 tcp Pkts=0> 1< Bytes=10024754> 29565< No_FIN_RST
ID=103358 2008-01-29 16:08:10.892000 Dur=34.985554 37.170.100.41:0 > 100.59.151.133:8080 tcp Pkts=0> 1< Bytes=6752212> 57865< No_FIN_RST
ID=103689 2008-01-29 16:38:06.553000 Dur=40.227446 37.170.100.20:0 > 100.59.151.133:8080 tcp Pkts=0> 1< Bytes=7763897> 54565< No_FIN_RST
ID=110381 2008-01-31 09:41:03.815000 Dur=28.908492 37.170.100.52:0 > 100.59.151.133:8080 tcp Pkts=0> 1< Bytes=5579339> 22147< No_FIN_RST
ID=112400 2008-01-31 13:10:23.841000 Dur=46.967461 37.170.100.15:0 > 100.59.151.133:8080 tcp Pkts=0> 1< Bytes=9064720> 11238< No_FIN_RST
ID=113945 2008-01-31 16:02:44.572000 Dur=70.918689 37.170.100.8:0 > 100.59.151.133:8080 tcp Pkts=0> 1< Bytes=13687307> 485421< No_FIN_RST
Defence R&D Canada – Ottawa • R & D pour la défense Canada – Ottawa
Who Has No Alibi?
Using a combination of
MATLAB “numeric-set”
filters and data queries
unavailable employees
were discovered.
The red dots on the diagram
indicate that when the
data extrusion activity
occurred the employee
was:
(1) Not in the building
(2) Inside Classified Room
(3) At their desk using the
network (within the
last 60 seconds)
(The clusters of boxes
indicate that all
employees have an alibi
for more than one
event.)
Defence R&D Canada – Ottawa • R & D pour la défense Canada – Ottawa
Root Cause of Anomaly
If the attack was triggered by a person then it should be
possible to spot any employee with the opportunity to start
the session. From the timing of the events however all the
employees have an alibi for more than one event.
This looks more like some type of malware is being used to
extrude the data from the network.
Defence R&D Canada – Ottawa • R & D pour la défense Canada – Ottawa
Answers to Mini Challenge 1
MC1.1: Identify which computer(s) the employee most likely used to send
information to his contact in a tab-delimited table which contains for
each computer identified: when the information was sent, how much
information was sent and where that information was sent.
TIME
2008-01-08
2008-01-10
2008-01-10
2008-01-15
2008-01-15
2008-01-17
2008-01-17
2008-01-22
2008-01-22
2008-01-24
2008-01-24
2008-01-24
2008-01-29
2008-01-29
2008-01-29
2008-01-31
2008-01-31
2008-01-31
17:01:33.001
14:27:12.238
16:01:53.956
16:14:34.563
17:03:29.342
12:12:10.990
17:57:19.341
08:50:21.894
17:41:55.862
09:46:34.452
10:26:31.321
17:07:34.775
15:41:32.763
16:08:10.892
16:38:06.553
09:41:03.815
13:10:23.841
16:02:44.572
Source IP
37.170.100.31
37.170.100.31
37.170.100.16
37.170.100.16
37.170.100.31
37.170.100.41
37.170.100.18
37.170.100.13
37.170.100.16
37.170.100.10
37.170.100.32
37.170.100.20
37.170.100.56
37.170.100.41
37.170.100.20
37.170.100.52
37.170.100.15
37.170.100.8
Target IP
100.59.151.133
100.59.151.133
100.59.151.133
100.59.151.133
100.59.151.133
100.59.151.133
100.59.151.133
100.59.151.133
100.59.151.133
100.59.151.133
100.59.151.133
100.59.151.133
100.59.151.133
100.59.151.133
100.59.151.133
100.59.151.133
100.59.151.133
100.59.151.133
Outbound Bytes Inbound Bytes
8889677
12223
6543216
22315
8543125
12312
6773214
24661
9513313
14324
3679122
24423
5873546
25234
9984318
42231
8873483
16778
7825451
23783
5531674
22479
9732417
42347
10024754
29565
6752212
57865
7763897
54565
5579339
22147
9064720
11238
13687307
485421
MC1.2: Characterize the patterns of behavior of suspicious computer use.
Large session are sent after an employee leaves their desk. Packets
are sent to a single external IP address.
Defence R&D Canada – Ottawa • R & D pour la défense Canada – Ottawa
Mini-Challenge 2 Social and Geospatial
The NTE has a large library of function calls which that were
leveraged to produce the social network diagrams.
In this solution the graph data query engine, the layout
algorithms and plotting routines were used to produce the
diagrams.
In this case, the tools can plot about 400 devices however
since the social network was so large the tools could only plot a
subset of the data.
Defence R&D Canada – Ottawa • R & D pour la défense Canada – Ottawa
Solution Process
Import the raw data
Store Node-to-Node Data into the NTE graph query structure
Find all potential middle men (Boris)
Check if there is a potential leader and 3 handlers on
each middle man
Check if the three handlers share a common employee
and do not talk directly to one another
Grab links related to the employee/leader/Boris/Handler
Send the selected graph data to the plotting engine
Defence R&D Canada – Ottawa • R & D pour la défense Canada – Ottawa
Social Network Diagram
Answer MC2.1:
Since vertex 194
is not directly
connected to the
fearless leader
the organization
of the criminal
network matches
situation A
Defence R&D Canada – Ottawa • R & D pour la défense Canada – Ottawa
Social Network Diagram - Annotated
Fearless leader
Employee
Boris
3
Handler
Defence R&D Canada – Ottawa • R & D pour la défense Canada – Ottawa
Social Network Diagram
Answer: MC2.3
There is a shorter
path to the
Fearless leader
Defence R&D Canada – Ottawa • R & D pour la défense Canada – Ottawa
Geospatial Diagram
Diagram created
with the NTE
freedraw_graph
function.
The fearless
leader appears to
have more
international
contacts in
Posana. Whether
that is significant
is not clear.
International Contact
Fearless Leader
Middleman
Handler
Employee
Defence R&D Canada – Ottawa • R & D pour la défense Canada – Ottawa
Answers to Mini-Challenge 2
MC2.1: Which of the two social structures, A or B, most closely match
the scenario you have identified in the data? A
MC2.2: Provide the social network structure you have identified as a
tab delimitated file. It should contain the employee, one or more
handler, any middle folks, and the localized leader with their
international contacts.
100
251
194
563
4994
92
4
282
551
589
629
1450
1630
2077
2103
3235
3946
4776
5078
5561
Employee
Handler
Handler
Handler
Middleman
Leader's International
Fearless Leader
Leader's International
Leader's International
Leader's International
Leader's International
Leader's International
Leader's International
Leader's International
Leader's International
Leader's International
Leader's International
Leader's International
Leader's International
Leader's International
Contact
Contact
Contact
Contact
Contact
Contact
Contact
Contact
Contact
Contact
Contact
Contact
Contact
Contact
@schaffter
@benassi
@reitenspies
@pettersson
@good
@tolbert
@szemeredi
@decker
@chandru
@kodama
@nakhaeizadeh
@barvinok
@heyderhoff
@streng
@wotawa
@reed
@hogstedt
@bolotov
@avouris
@wenocur
Defence R&D Canada – Ottawa • R & D pour la défense Canada – Ottawa
Answers to Mini-Challenge 2
MC2.3: Characterize the difference between your social network and the
closest social structure you selected (A or B). If you include extra
nodes please explain how they fit in to your scenario or analysis.
There is a more direct path between the fearless leader and the employee
(through 14, 22, 170, 351)
MC2.4: How is your hypothesis about the social structure in Part 1
supported by the city locations of Flovania? What part(s), if any, did
the role of geographical information play in the social network of part
one?
The handlers are located in the same city as the employee.
MC2.5: In general, how are the Flitter users dispersed throughout the
cities of this challenge? Which of the surrounding countries may have
ties to this criminal operation? Why might some be of more significant
concern than others?
The social networking group is predominantly Flovanian. There is slightly
more international contacts associated with Posana both in terms of the
Fearless Leaders Contacts and the Social network in general.
Defence R&D Canada – Ottawa • R & D pour la défense Canada – Ottawa
Mini-Challenge 3
I was not able to complete the mini-challenge 3
however I do find it suspicious that at Location 1
at 45min 27sec into the first video two people are
meeting and exchanging a document on the
street.
Defence R&D Canada – Ottawa • R & D pour la défense Canada – Ottawa