Transcript GILK: A Dynamic Instrumentation Tool for the Linux Kernel

GILK: A Dynamic Instrumentation
Tool for the Linux Kernel
David J. Pearce, Paul H.J. Kelly, Tony Field and Uli Harder
Based on paper presented at 12th International
Conference on Computer Performance Evaluation
(TOOLS’02), London, April 2002 (Springer LNCS 2324)
See http://www.doc.ic.ac.uk/~djp1/gilk.html
[email protected]
Motivation
 Instrumentation: “Do programmers really know what is going on?”
» Need to visualise program behaviour
» Build, debug/validate performance models
» Understand and fix performance problems
» Exploration: users need to explore system behaviour interactively
 Auditing: “How can we monitor what is going on?”
» Relies on understanding of program behaviour
» Minimise impact on system
» Need flexible programmable tool
» Operating at lowest possible level to circumvent
countermeasures
Introduction: instrumenting the Linux kernel
 GILK is a dynamic instrumentation tool for the Linux
Kernel
»
»
»
»
Run-time patching of the OS kernel’s code
On the fly: no need to reboot, no need to restart servers
“Instruments” can be added and removed via a simple GUI
Or programmatically – adding and removing an instrument
takes milliseconds
» No need to recompile the kernel
» No need to interrupt the web server etc
Measure, audit, intercept…
 Instruments…
» We provide a simple set of instruments for performance
measurement, selectable from the GUI
 User-definable instruments
»
»
»
»
API allows instruments to be defined by the user in C
Intercept any kernel function
Log parameters, message contents etc
Instruments can also check parameters against security
policy, enforce rules, etc
GILK – the GUI
 GUI starts with
Linux kernel’s
symbol table
 Lists all kernel
functions
» System calls
» Network protocol
stack
» File system
» Device drivers
» Inter-process
communication
» Memory
management
GILK – An Introduction
» Click on a
kernel function
» Dialogue offers
to attach
instrument to
function
» Various
instruments
» Before, after or
both
» Or browse
control flow
graph to
instrument
internal blocks
GILK – An Introduction
» An instrumentation
“experiment” runs
automatically
» Each instrument has its
own launch and
splashdown time
» So that performance
impact of
instrumentation can be
staggered
» All instruments
automatically removed
on completion/exit
» Instrument data is
logged to experiment
output file
Example: validate TCPdump
 Uli Harder is researching statistical “self-similarity” properties of
network traffic
»
»
»
»
»
Using standard TCPDUMP tool to obtain network timing information
Produced fair amount of data
But occasional zero or negative inter-arrival time was spotted
Are results so far wasted?
Is TCPDUMP useable or not?
 Uli used GILK for comparison with TCPDUMP
» GILK provides high resolution packet arrival times
» TCPDUMP does generate erroneous results
» But previous results valid – statistical properties unchanged
What We Did
BSD Socket
Layer
{
Linux Network Stack
sock_recvmsg
 Browse Linux
source code and
call graph
» Identify highlevel socket
interface
» Add “rdtsc” timer
instrument
 Is socket
interface timing
what we want?
Ethernet
Driver
{
speedo_rx
» Find ethernet
device driver
» Instrument it to
time actual
packet transfer
Where was the Success?
 Uli’s not a kernel programmer
» Didn’t know beforehand what was needed
» Time wasted trying different instrumentation
 GILK is fast, and that’s important
» Instruments can be moved, adjusted, relaunched interactively
while the system is under the test load
» Allows us to try many different instrumentations
» Fits exploratory nature of debugging and modelling system
behaviour
How it Works
 GILK uses code splicing technology
»
»
»
»
Allows instrumentation of active kernel
Instrumentation performed at machine code level
Works on unmodified kernel image
No patching, restarting, recompiling etc
 Instruments written in ‘C’
» Implemented as Kernel Modules
» Access to full language features
» Access to kernel functionality
 System Requirements
» Only Intel x86 Architecture supported
» Kernel versions 2.0 – 2.2 supported
- Extension to 2.4 is trivial
Instruction Boundaries
Kernel
Function
{
Before
Kernel
Function
{
Code
Patch
{
After
splice
instrumentation
unused
Relocated sequence
Conclusions
 GILK quickly instruments Linux Kernel
» Runtime code splicing useful for instrumentation
» Some development work needed – ease of use, visualisation of instrument output
 Intriguing potential….
» Automatic instrumentation – bottleneck identification strategies
» Security applications –
-
Intercept, monitor for unusual behaviour
Characterise processes which initiate unexpected behaviour
Insert authorisation policies
check client process id, application image id, IP addresses, message contents etc
 Future work …
»
»
»
»
Simplify GUI, develop instrumentation strategies, visualization
Automate extraction of dynamic call graph, search for bottlenecks
Explore security policy ideas… context-dependent quality-of-service?
Later linux kernels (2.4 should be easy), SMPs, non-Intel platforms (for Solaris
see Paradyn project at Wisconsin)
» User-Space programs
 Our Research group is now doing this for Java …