Transcript pptx - Comcast.net

Centralized Logging:
Why Would I Want to Do That?
Garrett Lanzy
Access more information security training for
campus technical staff and earn CEUs:
its.mnscu.edu/security/training/
Outline
• Problems
• Centralized Logging at Metropolitan State
– Architecture and Configuration
– Other Considerations
•
•
•
•
Real-Life Scenario – How Does Logging Help?
Future Directions
Conclusions
Q&A
Access more information security training for
campus technical staff and earn CEUs:
its.mnscu.edu/security/training/
Problems: Logging Issues
• Logs spread across servers and network
devices
• Different devices use different log formats
• Default settings often omit important
information
• Log data retention varies
• An “event” can involve numerous different
devices – hard to search/correlate
Key Questions
•
•
•
•
What should be logged?
How can this data be gathered?
How long should this data be retained?
How can searching be simplified?
An Answer:
Centralized Logging
Access more information security training for
campus technical staff and earn CEUs:
its.mnscu.edu/security/training/
Centralized Logging at
Metropolitan State University
• Single central logging server (axe)
– Linux (SUSE Linux Enterprise Server)
– syslog-ng logging daemon
– 30GB partition for logs
• Stores 65 days worth of logs for 60+ devices (including
firewall logs for 2 campuses)
– Physical server
– Same server used for network monitoring (Nagios)
What is syslog?
• Logging architecture for Unix/Linux
• Developed since 1980s
• Provides services for local & centralized
logging (single or multi-tier)
• Communication protocol standardized by IETF
– RFC 3164 - “classic” BSD Unix syslog (UDP only)
– RFC 5424 – TLS transport support
– Neither defines format of log messages
What is syslog-ng?
•
•
•
•
•
“Next generation” syslog
Open-source & commercial versions
Supports UDP, TCP, and TLS transports
Flexible configuration options
Many options for storing log data
– Simple files
– Structured files (CSV, etc)
– Database interfaces
syslog Message Facilities
• syslog defines 24 types of sources for logs (called
facilities) – common ones:
–
–
–
–
–
–
–
kernel – messages from OS kernel
user – user-level messages
mail – e-mail related messages
system – messages from system daemons/services
auth – security/authorization messages
syslog – messages generated by syslog itself
local0 – local7 – implementation-defined message
• Metro design creates separate files per facility for
each device
syslog Message Severities
• Each message assigned a severity by creator
• Can be used to help filter by importance
Severity Description
0
Emergency: system is unusable
1
Alert: action must be taken immediately
2
Critical: critical conditions
3
Error: error conditions
4
Warning: warning conditions
5
Notice: normal but significant condition
6
Informational: informational messages
7
Debug: debug-level messages
Metro Centralized Logging Design
• Based on Novell Cool Solutions article
Centralized Syslogging with Syslog-NG on
SUSE Linux by Scott Flowers
http://www.novell.com/coolsolutions/feature/18044.html
• Supports classic syslog on UDP port 514 &
syslog over TCP using TCP port 5140
• Simpler directory structure for easier searches
• Added automated compression via logrotate
Logging server syslog-ng config
• Default SuSE configuraton contains logging source
src for messages originating on local server
• Metro design adds logging source nethosts for logs
coming form other servers
• Adds logging destination byhost which writes logs for
each host to a specific directory
/var/log/HOSTS/hostname
• Log files named facilityname.log
• Local logs for central logging server not changed
syslog-ng.conf for nethosts source
# define source for messages from network hosts
Source nethosts {
#
# process log messages from network:
#
udp(ip(“0.0.0.0") port(514));
tcp(ip(“0.0.0.0") port(5140) keep-alive(yes) max_connections(100));
};
syslog-ng.conf for byhost destination
# separate out hosts into individual log directories
destination byhost {
file("/var/log/HOSTS/$HOST/$FACILITY.log"
owner(root) group(root) perm(0640) dir_perm(0750) create_dirs(yes)
);
};
log {
source(src);
source(nethosts);
destination(byhost);
};
Configuring other devices/servers
• Use TCP over UDP when possible
– Exception: Cisco ASA devices stop forwarding if
syslog over TCP configured & not working
• Check server/device logging settings to make
sure appropriate information is logged
– Example: some servers only log failed login
attempts, best to also log successful ones
• Make sure to adjust firewall configuration on
centralized logging server
Configuring Linux Server - syslog
• syslog always sends via UDP (default port 514)
• Sample config to send messages for all
facilities of severity informational or higher to
central log server – add lines to syslog.conf:
# send all logs of severity info or higher to logging server
*.info @x.x.x.x
Configuring Linux Server - syslog-ng
• Define destination loghost for TCP port 5140
on logging server, send all messages there
• Add following lines to syslog-ng configuration:
destination loghost {
tcp("x.x.x.x" port(5140));
};
log {
source(src);
destination(loghost);
};
Configuring Network Devices
• Most managed network devices support
logging to central syslog server
• Many configurable via web/GUI interfaces
• Sample Cisco IOS configuration:
Core-6500#conf term
Enter configuration commands, one per line. End with CNTL/Z.
Core-6500(config)#logging source-interface Vlan1
Core-6500(config)#logging x.x.x.x
Core-6500(config)#end
Configuring ASA Firewall Logging
• MnSCU-owned ASA firewalls can be
configured to log to campus logging servers contact MnSCU network support team:
[email protected]
• For campus-owned devices, easy to set up
with Cisco ASDM:
– Configuration -> Device Management -> Syslog Servers
– Configuration -> Device Management -> Syslog Setup
Configuring ASA Firewall Logging
• MnSCU-owned ASA firewalls can be
configured to log to campus logging servers contact MnSCU network support team:
[email protected]
• For campus-owned devices, easy to set up
with Cisco ASDM:
– Configuration -> Device Management -> Syslog Servers
– Configuration -> Device Management -> Syslog Setup
Configuring Cisco ACS Logging
• Can configure logging for each “report name”
• To start: from web admin interface, select System
Configuration -> Logging
• Select Configure link for syslog column of desired
“report”
• Click Enable Logging checkbox
• Scroll down, enter IP address of server, set port to
514, and max bytes to 1024
• Submit
Sample ACS Config (pt. 1)
Sample ACS Config (pt. 2)
Sample ACS Config (pt. 3)
Windows Servers
• Windows event logging traditionally has had
no support for centralized logging
• Several add-on products available to send
Windows events to syslog server
– Snare for Windows – open-source product
• Microsoft has added capability for centralized event
collection in Windows 2008 – for info, see:
http://www.windowsecurity.com/articles/Centralized-Auditing-here-FREE.html
Snare Agent for Windows
• Snare Agent for Windows is open-source, free
software (GNU Public License)
• Available at:
http://www.intersectalliance.com/projects/SnareWindows/
• 2 versions:
– Snare for Windows: supports NT, 2000, XP, 2003
– Snare for Windows Vista: supports Vista, 2008,
Windows 7
• Either will run on 32 or 64-bit installations
Configuring Snare Agent
Windows Event Log Settings
• Security audit settings – clean Win2008 install
A better idea?
Other Considerations
•
•
•
•
Clock synchronization
Log rotation/space utilization
Backups
Network Monitoring
Clock Synchronization
• System/device clocks need to be synchronized
in order to correlate log data
• Any state/MnSCU router can be used as an
NTP time source
• Most network devices can sync to NTP
• Windows standalone servers or domain
controllers should be synchronized to NTP
http://support.microsoft.com/kb/816042
Linux NTP Configuration
• Linux VMs: try kernel option clock=pit
• Remove any references to local
unsynchronized clock (127.0.0.1)
• Simple sample ntp.conf
driftfile /var/lib/ntp/drift/ntp.drift
# path for drift file
logfile /var/log/ntp
server 172.16.255.238
server 156.98.1.1
Log Rotation
• logrotate – Linux utility to rotate and
compress log files
• Creates new log files when run and deletes
after specified interval
• Metro configuration:
– Rotate files daily
– Delay compression for 1 day
– Keep 65 days worth of logs
Log server logrotate.conf additions
# rotate logs in /var/logs/HOSTS tree
/var/log/HOSTS/*/*.log {
daily
compress
delaycompress
dateext
missingok
notifempty
rotate 65
maxage 90
sharedscripts
create 640 root root
postrotate
/etc/init.d/syslog reload
endscript
}
Real-Life Scenario –
Have your received one of these?
> Subject: Notice ID: 22-79162500 Notice of Unauthorized Use of Paramount Pictures Corporation Property
> Date:
Fri, 23 Apr 2010 10:07:58 -0700
> From:
[email protected]
> Reply-To: [email protected]
> To: [email protected]
> -----BEGIN PGP SIGNED MESSAGE----> Hash: SHA1
> Notice ID: 22-79162500
> Notice Date: 23 Apr 2010 16:58:09 GMT
> Minnesota State University System
> Dear Sir or Madam:
>
> BayTSP, Inc. ("BayTSP") swears under penalty of perjury that Paramount
> Pictures Corporation ("Paramount") has authorized BayTSPto act as
> its non-exclusive agent for copyright infringement notification.
> BayTSP'ssearch of the protocol listed below has detected
> infringements of Paramount's copyright interests on your IP
> addresses as detailed in the below report.
… And to the Relevant Information:
> Evidentiary Information:
> Notice ID: 22-79162500
> Initial Infringement Timestamp: 23 Apr 2010 16:23:27 GMT
> Recent Infringement Timestamp: 23 Apr 2010 16:23:27 GMT
> Infringers IP Address: 199.17.228.252
> Protocol: BitTorrent
> Infringed Work: She's Out of My League
> Infringing File Name: She's Out of My League 2010 CAM-lu
> Infringing File Size: 608227413
> Bay ID: d116ad77ba844d11a350055c9227e2b0d12a29d9|608227413
> Port ID: 45739
> Infringer's DNS Name:
> Infringer's User Name:
How to proceed?
• 199.17.228.252 is a NAT address for the
student wireless network on the Midway
campus
• Search the Midway ASA firewall log to
determine the local IP address
• Then search the ACS appliance log to find the
user logged in
…Um, but wait….
• Logs older than 1 day old are compressed, so
need to decompress before they can be
searched
axe:~ # cd /var/log/HOSTS/mwpix/
axe:/var/log/HOSTS/mwpix # bzcat local4.log-20100423.bz2 >/tmp/mwpix.log
axe:/var/log/HOSTS/mwpix # cd /var/log/HOSTS/172.16.210.49/
axe:/var/log/HOSTS/172.16.210.49 # bzcat auth.log-20100423.bz2 >/tmp/acs.log
axe:/var/log/HOSTS/172.16.210.49 #
What does the ASA log look like?
Apr 23 09:00:00 mwpix %ASA-6-305011: Built dynamic TCP translation from
inside:10.15.38.69/49523 to outside:199.17.228.252/20192
Apr 23 09:00:00 mwpix %ASA-5-304001: 10.15.38.69 Accessed URL
65.55.17.27:http://www.msn.com/
Apr 23 09:00:00 mwpix %ASA-6-302020: Built inbound ICMP connection for faddr
207.171.116.193/0 gaddr 199.17.231.41/0 laddr 199.17.231.41/0
Apr 23 09:00:00 mwpix %ASA-6-302021: Teardown ICMP connection for faddr
207.171.116.193/0 gaddr 199.17.231.41/0 laddr 199.17.231.41/0
Apr 23 09:00:00 mwpix %ASA-6-305011: Built dynamic UDP translation from
inside:10.15.38.67/62560 to outside:199.17.228.252/49397
Apr 23 09:00:00 mwpix %ASA-6-305011: Built dynamic UDP translation from
inside:10.15.38.67/60379 to outside:199.17.228.252/41373
Apr 23 09:00:00 mwpix %ASA-4-313005: No matching connection for ICMP error
message: icmp src inside:10.15.38.67 dst outside:199.17.241.241 (type 3, code 3) o
n inside interface. Original IP payload: udp src 199.17.241.241/53 dst
10.15.38.67/60379.
Apr 23 09:00:00 mwpix %ASA-6-305011: Built dynamic UDP translation from
inside:10.15.35.98/50165 to outside:199.17.228.252/38489
Apr 23 09:00:00 mwpix %ASA-7-609001: Built local-host outside:74.63.145.159
We’re looking for….
• … a line that contains:
199.17.228.252/45739
• … at about 16:23:27 GMT (11:23:27 CDT)
grep is your friend!
• grep – utility to search by regular expression
• Some basic regular expression syntax:
– Most characters “match” themselves
– Exceptions: [\^$.|?*+()
– \ is an escape character; meaning varies by context, always
suppresses special meaning of [\^$.|?*+()
– . matches any single character (except the line break
character)
– * repeats the previous item 0 or more times
– …. So .* matches “anything”
http://www.regular-expressions.info/reference.html
grep in action
axe:/tmp # grep 199.17.228.252/45739 mwpix.log
Apr 23 08:49:17 mwpix %ASA-6-305011: Built dynamic TCP translation from
inside:10.15.34.45/1701 to outside:199.17.228.252/45739
Apr 23 08:49:47 mwpix %ASA-6-305012: Teardown dynamic TCP translation
from inside:10.15.34.45/1701 to outside:199.17.228.252/45739 duration
0:00:30
Apr 23 11:26:23 mwpix %ASA-6-305012: Teardown dynamic TCP
translation from inside:10.15.38.87/50372 to
outside:199.17.228.252/45739 duration 0:03:00
Looks like we have a match - a connection of
duration 3 min. which ended at 11:26:23 –
i.e., was initiated at 11:23:23 (vs. 11:23:27)
Now check the ACS log
(note: ACS timestamps are GMT, not local time)
axe:/tmp # grep 10.15.38.87 acs.log
Apr 23 16:19:06 172.16.210.49 CisACS_01_PassedAuth y2gjm36g 1 0
Message-Type=Authen OK,User-Name=vvvvvv,NAS-IPAddress=172.16.212.7,Caller-ID=10.15.38.87,NAS-Port=vvvvvv,GroupName=Default Group,Filter Information=No Filters activated.,
Apr 23 16:19:06 172.16.210.49 CisACS_03_RADIUSAcc y2gjm36h 1 0 UserName=vvvvvv,NAS-IP-Address=172.16.212.7,NAS-Port=29,GroupName=Default Group,Framed-IP-Address=10.15.38.87,Calling-StationId=10.15.38.87,Acct-Status-Type=Start,Acct-SessionId=4bd1c873/00:24:2b:ae:79:c1/38738,
Apr 23 18:05:44 172.16.210.49 CisACS_03_RADIUSAcc xjgh1n40 1 0 UserName=vvvvvv,NAS-IP-Address=172.16.212.7,NAS-Port=29,GroupName=Default Group,Framed-IP-Address=10.15.38.87,Calling-StationId=10.15.38.87,Acct-Status-Type=Stop,Acct-InputOctets=495945068,Acct-Output-Octets=1553689408,Acct-SessionId=4bd1c873/00:24:2b:ae:79:c1/38738,Acct-Session-Time=6398,AcctInput-Packets=1155133,Acct-Output-Packets=1692907,
axe:/tmp #
Determination
• User vvvvvv made the connection outlined in
the e-mail and downloaded 1.5GB – this case
will be handled as inappropriate use
• Actual time to perform the log search process
(including reading the e-mail, logging in to the
logging server, decompressing the logs, and
searching): less than 5 minutes
Future Directions
• Centralized logging for Windows workstations
• Use TLS connections instead of TCP
– Makes snooping much more difficult
– Prevents insertion of “bogus” logs
• Automated log monitoring/proactive
notification
• Log to database instead of stream files?
– Disk space is cheap, right?
Conclusions
• Centralized logging is not that difficult to set
up
• Significant benefits for both normal
operations and incident handling
• Can save large amounts of time for
network/server/security administrators
Questions?
Garrett Lanzy
Through 5/11/10:
[email protected]
5/12/10 & After:
[email protected]
Access more information security training for
campus technical staff and earn CEUs:
its.mnscu.edu/security/training/