Transcript Introduction - Northern Kentucky University

CIT 470: Advanced Network and
System Administration
Logging
CIT 470: Advanced Network and System Administration
Slide #1
Topics
1.
2.
3.
4.
5.
6.
System logs
Logging policies
Finding logs
Syslog
Syslog servers
Log monitoring
CIT 470: Advanced Network and System Administration
Slide #2
System Logs
• Logs record status and error conditions.
• Where do log messages come from?
– Kernel
– Accounting system
– System services
• Logging methods:
– Service records own logs (apache, cron).
– Service uses syslog service to manage logs.
CIT 470: Advanced Network and System Administration
Slide #3
Logging Policies
1.
2.
3.
4.
Throw away log data.
Save for a while, then throw away.
Rotate log files
Archive log files
CIT 470: Advanced Network and System Administration
Slide #4
How to choose a logging policy?
1.
2.
3.
4.
Are there any data retention requirements?
How much disk space do you have?
How quickly do you need to retrieve logs?
Could you find the source of a security
issue with the logs you keep?
CIT 470: Advanced Network and System Administration
Slide #5
Throwing Away
• Not recommended.
• Leaves you unaware of:
– Software and hardware problems
– Security incidents
• It may take time to detect an incident.
– Keep logs for at least a month or two.
CIT 470: Advanced Network and System Administration
Slide #6
Rotation
• Keep backup files for each day/week
logfile
logfile.1
logfile.2
logfile.3
• Rename files each day/week to move old
ones back in list.
– Compress rotated logs to save disk space.
– Remove/archive logs that are X days old.
CIT 470: Advanced Network and System Administration
Slide #7
Rotation
#!/bin/sh
cd /var/log
mv logfile.2 logfile.3
mv logfile.1 logfile.2
mv logfile logfile.1
cp /dev/null logfile
chmod 600 logfile
CIT 470: Advanced Network and System Administration
Slide #8
logrotate
Program to handle log rotation.
– Run via /etc/cron.daily.
– Configured via /etc/logrotate.conf.
Options
–
–
–
–
–
How often to rotate
How long to keep logs
Compression or not
Log file permissions
Pre- and post-rotate scripts
CIT 470: Advanced Network and System Administration
Slide #9
logrotate.conf
# rotate log files weekly
weekly
# keep 4 weeks worth of backlogs
rotate 4
# create new (empty) log files after rotating old
create
# uncomment if you want your log files compressed
#compress
# RPM packages drop log rotation information into
include /etc/logrotate.d
# no packages own wtmp -- we'll rotate them here
/var/log/wtmp {
monthly
create 0664 root utmp
rotate 1
}
CIT 470: Advanced Network and System Administration
Slide #10
Archiving Logs
Store logs to archival media (tape.)
– Archive after X days/weeks.
– Should be part of regular backup plan.
– May want to save logs for all hosts together.
CIT 470: Advanced Network and System Administration
Slide #11
Finding Logs
Most logs are stored under
– /var/log
– /var/adm
Check syslog's configuration
– /etc/syslog.conf
To find other logs, read startup scripts
– /etc/init.d/*
– and manuals for services started by scripts.
CIT 470: Advanced Network and System Administration
Slide #12
Finding Logs
Log file
Program
Contents
messages
syslog
Various program/kernel logs.
syslog
syslog
Various program/kernel logs.
auth.log
su, ssh, login
Authorization fail/success.
lastlog
login, xdm
Logins, commands.
wtmp
login
Login accounting data.
acct/pacct
kernel
UNIX process accounting.
Xorg.log
X-Windows
X-Windows failures/info.
CIT 470: Advanced Network and System Administration
Slide #13
Syslog
Comprehensive logging system.
Frees programmers from managing log files.
Gives sysadmins control over log management.
Sorts messages by
Sources
Importance
Routes messages to destinations
Files
Network
Terminals
CIT 470: Advanced Network and System Administration
Slide #14
Syslog Components
Syslog
Daemon that does actual logging.
Additional daemon, klog, gets kernel messages.
openlog, syslog, closelog
C library routines to submit logs to syslog.
logger
User-level program to submit logs to syslog.
Can use from shell scripts.
CIT 470: Advanced Network and System Administration
Slide #15
Example Syslog Messages
Feb 11 10:17:01 localhost /USR/SBIN/CRON[1971]: (root) CMD ( runparts --report /etc/cron.hourly)
Feb 11 10:37:22 localhost -- MARK -Feb 11 10:51:11 localhost dhclient: DHCPREQUEST on eth1 to
192.168.1.1 port 67
Feb 11 10:51:11 localhost dhclient: DHCPACK from 10.42.1.1
Feb 11 10:51:11 localhost dhclient: bound to 10.42.1.55 -- renewal in
35330 seconds.
Feb 11 14:37:22 localhost -- MARK -Feb 11 14:44:21 localhost mysqld[7340]: 060211 14:44:21
/usr/sbin/mysqld: Normal shutdown
Feb 12 04:46:42 localhost sshd[29093]: Address 218.38.30.101 maps to
ns.thundernet.co.kr, but this does not map back to the address POSSIBLE BREAKIN ATTEMPT!
Feb 12 04:46:44 localhost sshd[29097]: Invalid user matt from
::ffff:218.38.30.101
CIT 470: Advanced Network and System Administration
Slide #16
Configuring Syslog
Configured in /etc/syslog.conf
Format: selector <Tab> action
Ex: mail.info /var/log/mail.log
Selector components
Source (facility)
List of facilities separated by commas or *.
Importance (level)
Can be none or *
CIT 470: Advanced Network and System Administration
Slide #17
/etc/syslog.conf
# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none
/var/log/messages
# The authpriv file has restricted access.
authpriv.*
/var/log/secure
# Log all the mail messages in one place.
mail.*
/var/log/maillog
# Log cron stuff
cron.*
/var/log/cron
# Everybody gets emergency messages
*.emerg
*
# Save news errors of level crit and higher in a special file.
uucp,news.crit
/var/log/spooler
# Save boot messages also to boot.log
local7.*
/var/log/boot.log
CIT 470: Advanced Network and System Administration
Slide #18
Syslog Facilities
Facility
Used By
kern
The kernel
user
User processes (default)
mail
Mail servers and related software.
daemon
System daemons (except mail, cron)
auth
Security and authorization-related commands.
lpr
Print server and related commands.
cron
Cron daemon.
local0-7
Eight local levels for other programs.
CIT 470: Advanced Network and System Administration
Slide #19
Syslog Levels
Level
Meaning
emerg
Panic situations (hardware failure, crash)
alert
Urgent situations
crit
Critical situations
err
Non-critical errors.
warning
Warnings.
notice
Might merit investigation.
info
Informational messages.
debug
Debugging (typically enabled temporarily.)
CIT 470: Advanced Network and System Administration
Slide #20
Syslog Actions
Action
Meaning
filename
Write message to file on local machine.
@hostname
Send message to syslogd on hostname.
@ip
Send message to syslogd at IP address.
user1,user2
Write message to user screen if logged in.
*
Write message to all logged-in users.
CIT 470: Advanced Network and System Administration
Slide #21
Testing Syslog
stu> for i in {debug,info,notice,warning,err,crit,alert,emerg}
> do
> logger -p daemon.$i "Test message for daemon, level $i"
> done
stu> tail /var/log/daemon.log
Feb 11 15:57:00 localhost stu: Test message for daemon, level debug
Feb 11 15:57:00 localhost stu: Test message for daemon, level info
Feb 11 15:57:00 localhost stu: Test message for daemon, level notice
Feb 11 15:57:00 localhost stu: Test message for daemon, level warning
Feb 11 15:57:00 localhost stu: Test message for daemon, level err
Feb 11 15:57:00 localhost stu: Test message for daemon, level crit
Feb 11 15:57:00 localhost stu: Test message for daemon, level alert
Feb 11 15:57:00 localhost stu: Test message for daemon, level emerg
CIT 470: Advanced Network and System Administration
Slide #22
Syslog Variants
Some use m4 macros
auth.notice
ifdef(‘LOGHOST’,
‘/var/log/authlog’, ‘@loghost’)
Red Hat Linux variants
Allows spaces as separators.
New operators: = (this priority only)
Ex: mail.=info
New operators: ! (except this pri and higher)
Ex: mail.info,mail.!err
CIT 470: Advanced Network and System Administration
Slide #23
Syslog NG
Free drop-in replacement for syslog.
More configurable
Save logs to templated location (auto-rotates.)
Filter logs based on program, time, message, etc.
Message format customization.
Allows easy logging to remote database.
Improved networking
TCP support as well as UDP.
Improved security
Doesn’t trust hostnames in remote messages.
TCP transmission permits encrypted tunneling (stunnel.)
CIT 470: Advanced Network and System Administration
Slide #24
Log Servers
Collect all syslog data on one server.
Allows logging to scale to large networks.
Logs can be correlated across machines.
Security-sensitive logs not on compromised host.
Routers and diskless-hosts must log to a server.
Need two syslog.conf files
Client: sends all logs across network to server.
Server: saves logs to database or local files.
CIT 470: Advanced Network and System Administration
Slide #25
Log Monitoring
Too much data for a human to process.
Logs arrive 24x7 too.
Use an automatic monitoring program
Triggers on patterns found in log.
Examples: logwatch, swatch
# 3ware logs
watchfor /(?i)3w-xxxx.+no longer fault
tolerant/
mail=root,subject=LW warn: disk 3ware
RAID not fault tolerant
throttle 1:00:00,use=regex
CIT 470: Advanced Network and System Administration
Slide #26
References
1.
2.
3.
4.
5.
6.
7.
Michael Bower, Building Secure Servers with Linux,
O’Reilly, 2005.
Aeleen Frisch, Essential System Administration, 3rd
edition, O’Reilly, 2002.
Jeremy Mate, “Log Analysis with Swatch,”
http://sial.org/howto/logging/swatch/, 2005.
Jeremy Mate, “Logging with syslog-ng,”
http://sial.org/howto/logging/syslog-ng/, 2005.
Evi Nemeth et al, UNIX System Administration
Handbook, 3rd edition, Prentice Hall, 2001.
Shelley Powers et. al., UNIX Power Tools, 3rd edition,
O’Reilly, 2002.
Syslog-ng FAQ, http://www.campin.net/syslogng/faq.html.
CIT 470: Advanced Network and System Administration
Slide #27