Transcript Unit OS 11: System Performance Evaluation

Unit OS11: Performance Evaluation
11.1. System Performance
Windows Operating System Internals - by David A. Solomon and Mark E. Russinovich with Andreas Polze
Copyright Notice
© 2000-2005 David A. Solomon and Mark Russinovich
These materials are part of the Windows Operating
System Internals Curriculum Development Kit,
developed by David A. Solomon and Mark E.
Russinovich with Andreas Polze
Microsoft has licensed these materials from David
Solomon Expert Seminars, Inc. for distribution to
academic organizations solely for use in academic
environments (and not for commercial use)
2
Roadmap for Section 11.1
Performance Evaluation and Prediction
Tools for Monitoring Windows Internals
Performance Monitor and mmc
Scheduling-related Performance Counters
Memory-related Performance Counters
Windows Event Tracing
3
Performance Prediction and
Evaluation
Constructing a model of the system and then
using the model to predict the system's behavior
Model reflects system structure or organization as
well as its workload or input
Analyzed using mathematical techniques
Alternatively, the model may be simulated
Benchmarking & Monitoring
Evaluating behavior of a live system
Predefined workloads
4
Modeling Approaches
Analytic modeling techniques
Discrete- and continuous-time Markov chains
Queueing theory, and queueing networks
Approximate methods based on these techniques
Operational analysis
Non-stochastic, measurement-based perspective to
the analysis of computer systems
Modeled with discrete-event simulation
Performance metrics from stochastic simulations
are subject to statistical analysis (as are data
obtained from real systems)
5
Validity of Models
Models, whether analytic or simulation, can be
inaccurate or implemented incorrectly
An important aspect of any kind of performance
modeling study is to validate the model and its
implementation to whatever extent is possible
One way to do this is to study a system using more
than one model, e.g., a simulation model and an
analytic model
Analytic modeling of many systems is
computationally demanding
6
Monitoring Windows How to obtain Performance Data
Windows is thoroughly instrumented
Performance counters allow for monitoring of most kernel objects
Many tools available to dig into Windows internals
Helps to see internals behavior “in action”
Several sources of tools
Support Tools
Resource Kit Tools
Debugging Tools
Sysinternals.com
Additional tool packages with internals information
Platform Software Development Kit (SDK)
Device Driver Development Kit (DDK)
7
Tools for Windows Performance
Monitoring
Tool
Image Name
Origin
File Monitor
Global Flags
Handle Viewer
Kernel debuggers
FILEMON
GFLAGS
HANDLE
WINDBG, KD
Live Kernel Debugging
Open Handles
Page Fault Monitor
LIVEKD
OH
PFMON
Pending File Moves
Performance tool
Pool Monitor
Process Explorer
Process Statistics
PENDMOVES
PERFMON.MSC
POOLMON
PROCEXP
PSTAT
Quick Slice
Task (Process) List
Task Manager
TDImon
QSLICE
TLIST
TASKMGR
TDIMON
www.sysinternals.com
Support Tools
www.sysinternals.com
Debugging tools, Platform SDK,
Windows DDK
www.sysinternals.com
Resource kits
Support Tools, Resource kits,
Platform SDK
www.sysinternals.com
Windows built-in tool
Support Tools, Windows DDK
www.sysinternals.com
Support Tools, Windows 2000
Resource kits, Platform SDK,
www.reskit.com
Windows 2000 resource kits
Debugging tools
Windows built-in tool
www.sysinternals.com
8
Process Explorer (Sysinternals)
Shows performance-related data
…plus full image path, command line,
environment variables, parent process,
security access token, open handles,
loaded DLLs & mapped files
9
Obtain System Information
with Process Explorer
Click View->System Information
10
Overview of Performance Data
Collection
Windows defines performance data in terms of objects,
counters, and instances
A performance object is any resource, application, or service
that can be measured
System Monitor and Performance Logs and Alerts allow to
select performance objects, counters, and instances to collect
and present performance data
Objects have performance counters
Objects may also have instances, which are unique copies of
a particular object type
Not all object types support multiple instances
_Total instance represents the sum of the values for all
instances of the object for a specific counter
11
Vast Array of Performance Data
12
Performance Counter Aggregation
into Performance Logs (via mmc)
13
Real-time Data Collection with
Performance Monitor
14
Windows Performance Counters Categories
Monitoring Memory Management
Memory\ Page Reads/sec
Memory\ Page Writes/sec
Memory\ Available Bytes
Process\ Working Set
Process\ Private Bytes
15
Windows Performance Counters Categories (contd.)
Monitoring Physical and Logical Disk I/O
PhysicalDisk\ % Disk Time
PhysicalDisk\ Avg. Disk Queue Length
PhysicalDisk\ Current Disk Queue Length
PhysicalDisk\ Avg. Disk Sec/Read
PhysicalDisk\ Avg. Disk Sec/Write
PhysicalDisk\ Disk Read Bytes/sec
PhysicalDisk\ Disk Write Bytes/sec
PhysicalDisk\ Avg. Disk Bytes/Read
PhysicalDisk\ Avg. Disk Bytes/Write
PhysicalDisk\ Disk Reads/sec
PhysicalDisk\ Disk Writes/sec
16
Windows Performance Counters Categories (contd.)
Monitoring Network Activities
Network Interface\ Bytes Total/sec
Network Interface\ Bytes Sent/sec
Network Interface\ Bytes Received/sec
Protocol_layer_object\ Segments Received/sec
Protocol_layer_object\ Segments Sent/sec
Protocol_layer_object\ Frames Sent/sec
Protocol_layer_object\ Frames Received/sec
Server\ Bytes Total/sec
Server\ Bytes Received/sec
Server\ Bytes Sent/sec
Network Segment\ % Network Utilization
17
Analyzing Processor Activity
Determine the baseline on normal workload (from
several weeks to a month)
Processor\ % Processor Time counter
System\Processor Queue Length counter
Be aware of the Idle process …
The Idle process runs a thread on each processor
To measure the Idle process, use the Process(Idle)\
% Processor Time counter, or Processes tab in Task Manager
Zero idle time could mean that the processor is handling a lot
of work, but it could also mean that the processor or central
processing unit (CPU) is overloaded
18
Detecting Processor Bottlenecks
CPU bottlenecks are indicated by:
Processor\ % Processor Time often exceeds 80 percent (and
there is no compute-bound workload)
System\ Processor Queue Length is often greater than 2 on a
single-processor system
Queue Length is the single most important parameter
Other indications:
Unusually high values appear for the Processor(_Total)\
Interrupts/sec or System\ Context Switches/sec counters
19
Evaluating Memory Usage
Establish a reference point (or baseline) for physical
memory usage under normal workload
Create logs of memory usage over an extended period (from
several weeks to a month)
Relevant Performance Counters
\Memory\Available Bytes
\Paging File(_Total)\% Usage
Exclude spikes; the range of values that seem to appear
consistently constitutes your baseline
20
Detecting Memory Bottlenecks
Indication for insufficient memory:
Value for Memory\Available Bytes is consistently
low (e.g. less than 5% of RAM)
If available memory is consistently low, the
computer becomes unresponsive:
It is occupied exclusively with disk I/O operations
During paging due to low memory, the processor is
idle while waiting for the disk to finish
21
Examining Disk Performance
Monitor disk counters along with counters from other
objects. The following is a list of recommended counters.
LogicalDisk\% Free Space
PhysicalDisk\Disk Reads/sec
PhysicalDisk\Disk Writes/sec
PhysicalDisk\Avg. Disk Queue Length
Memory\Available Bytes
Memory\Cache Bytes
Memory\Pages/sec
Processor(All_Instances)\% Processor Time
System\Processor Queue Length
22
Detecting a Disk Bottleneck
Avg. Disk Queue Length for LogicalDisk or PhysicalDisk
If the value of Avg. Disk Queue Length exceeds twice the
number of spindles, then you are likely developing a
bottleneck
With a volume set, a queue that is never shorter than the
number of active physical disks indicates that you are
developing a bottleneck
Notice that this might overstate the true length of the queue,
because the counter includes both queued and in-service
requests
23
Counters by Feature
Windows services and apps may bring their
own performance objects
Internet Information Service
Active Server Pages
FTP Service
Web Service
Internet Information Services
Global
Indexing Service
Indexing Service
Indexing Service Filter
HTTP Indexing Service
Message Queuing
MSMQ Session
MSMQ IS
MSMQ Queue
MSMQ Service
Quality of Service (QoS) Admission
Control
ACS/RSVP Service
ACS/RSVP Interfaces
ACS/RSVP Policy
Routing and Remote Access
(RRAS)
RAS Port
RAS Total
File Replication Service
FileReplicaConn
FileReplicaSet
Terminal Service
Terminal Services Session
Active Directory™
NTDS
24
Kernel Event Tracing
Windows kernel and core device drivers are
instrumented to record trace data
Event Tracing for Windows (ETW)
Common infrastructure in the kernel that provides trace data to
the user-mode facility
ETW is accessed by:
Controllers that start and stop logging sessions and manages
buffer pools
Providers that define GUIDs for the event classes they can
produce traces for; act on Controllers’ commands
Consumer select one or more trace sessions for which the
want to read trace data (in real-time or in log files)
25
ETW Providers and Kernel Logger
Windows Server systems include several built-in
providers in user mode
Active Directory, Kerberos, and Netlogon
ETW defines a logging session with the name NT
Kernel Logger (kernel logger) for use by the kernel
and core drivers.
NT Kernel Logger provider is implemented in the
kernel
26
User mode controllers may enable
kernel logger - ETW Operation
ETW library sends I/O control request to the WMI driver to enable
tracing on a particular event class
ETW library is implemented in \Windows\System32\Ntdll.dll
If file logging is configured the WMI driver creates a system thread in
system process that creates a log file
Alternatively, logging may use an in-memory buffer
The WMI driver records trace events to a buffer
File logging thread wakes up once per second to dump the contents
of the buffers to the log file
Trace records generated for the kernel logger have a standard ETW
trace event header
Header records timestamp, process, and thread IDs, info on event class
Event classes can provide additional data specific to their events
27
Trace and Logging Data may be
accessed via mmc
28
Kernel Logger Trace Classes
Traces classes and their generating components
Disk I/O - disk class driver
File I/O - file system drivers
Hardware Configuration - plug&play manager
Image Load/Unload - system image loader in the kernele
Page Faults - memory manager
Process Create/Delete - process manager
Thread Create/Delete - process manager
Registry Activity - Configuration Manager
TCP/UDP Activity - TCP/IP driver
ETW controllers/providers described in Platform SDK
29
Using Event Logs and Performance
Counters for Optimization
Performance must be tuned to a workload
A sequence of service requests, commands, I/Os
that exercise the software
Often produced by workload generators rather than
real-world service provision
Allow for replay or generation of service requests at
a maximum rate so that bottlenecks can be
identified in systems
Most workloads are domain-specific
I.e.; TPC benchmarks and workloads
No “optimal” system configuration per se
30
Further Reading
Mark E. Russinovich and David A. Solomon,
Microsoft Windows Internals, 4th Edition, Microsoft
Press, 2004.
Chapter 1 - Concepts and Tools
Performance Tool, Support Tools, Resource Kits, pp. 25-34
Chapter 4 - Management Mechanisms
Windows Management Instrumentation, pp. 237-249
Chapter 3 - System Mechanisms
Event Tracing for Windows, pp. 177-180
Windows Performance Counter Reference
http://msdn.microsoft.com/library/default.asp?url=/library/enus/counter/default.asp
31
Source Code References
Windows Research Kernel (WRK):
\base\ntos\perf – core performance logging support
\base\ntos\ex\sysinfo.c – system/process
performance query functions
32