Transcript Windows Performance Troubleshooting and Analysis

Daniel Pearson
David Solomon Expert Seminars
SVR311
Daniel Pearson
Started working with Windows NT 3.51
Three years at Digital Equipment Corporation
Supporting Intel and Alpha systems running Windows NT
Seven years at Microsoft
Senior Escalation Lead in Windows base team
Worked in the Mobile Internet sustained
engineering team
Instructor for David Solomon, co-author of the
Windows Internals book series
Agenda
Components of performance analysis
Understanding the tools for troubleshooting
and analyzing performance issues
Troubleshooting CPU, memory and disk I/O
issues using various Windows tools
* Portions of this session are based on material developed by Mark Russinovich and
David Solomon
Components of Performance Analysis
Event Tracing for Windows
Core component of the operating system
Kernel mode data structures
Used to store information about the system and
system objects that can be read by various tools
e.g. dt nt!_KTHREAD KernelTime
CPU performance monitoring events
Refer to the Intel 64 and IA-32 Architectures
Software Developer’s Manual
Event Tracing for Windows
Built in to the system
High performance, low overhead and scalable
2.5% CPU usage for a sustained rate of 10,000 events/sec on
a 2 GHz CPU1
Operations throughout the system that are of
interest to performance are fully instrumented
e.g. process and thread activity, registry I/O, disk I/O
1. Milirud, Michael. 2008. Windows Performance Analysis: Using Windows Performance
Tools. Presented at Microsoft's WinHEC conference, November 5-7, Los Angeles, CA.
Event Tracing for Windows
Uses a buffering and logging mechanism
implemented in the kernel
Per-processor buffers that are written to disk by
an asynchronous writer thread
Ability to enable and disable tracing dynamically
Supports a managed code provider
Sysinternals Utilities
Sysinternals Utilities
Process Explorer
Useful for displaying which files, registry keys and
other objects processes have open and which DLLs
they have loaded
Process Monitor
Useful for showing real-time file system, registry
and process & thread activity
Available for download from the TechNet site
http://technet.microsoft.com/sysinternals
Resource Monitor
Resource Monitor
Included with Windows Vista and greatly enhanced in
Windows 7 and Windows Server 2008 R2
Allows the viewing of CPU, memory, disk and network
resources as well as handles and modules in real time
Ability to end, suspend and resume processes as well as
to start, stop and restart Windows services
Useful for identifying the highest resource consumers
by individual resource type, e.g. CPU
Able to list the wait chain tree of a process to
determine if a process is waiting on another
Using Resource Monitor
Performance Monitor
Performance Monitor
Queries performance counters that measure system
state or activity
Current values are read at specific intervals
Performance counters are included in the operating
system and can be included as part of applications
Able to collect event trace data from trace providers
that report actions or events
Can combine multiple trace providers into a single session
Configuration information can be collected from
registry keys at a specific time or interval
Using Performance Monitor
Windows Performance Analyzer
Windows Performance Analyzer
Part of the Windows Performance Toolkit
Support for both x86, x64, and IA64 architectures
Consists of three primary programs
xperf.exe
Used for controlling tracing and processing trace data
xbootmgr.exe
Automates on and off state transitions and captures traces during
those transitions
xperfview.exe
A graphical trace visualization tool to represent data in the form of
interactive graphs and summary tables
Windows Performance Analyzer
Primarily uses the Event Tracing for Windows
infrastructure built in to the system
Can be enabled or disabled at any time without
requiring a system or process restart
Supports symbol decoding, sample profiling,
and recording of call stacks on kernel events
Designed to be used during automation
All the functions of the tools are available via the
command line tool xperf.exe
Support for Earlier Systems
The Windows Performance Toolkit will fail to
install on Windows XP and on Windows Server
2003 although data collection is supported
Copy xperf.exe and perfctrl.dll
Trace analysis is only supported on Windows
Vista and later systems
Capturing a Performance Trace
Kernel options divided into two parts
Kernel Flags
Identified by the use of uppercase characters
e.g. PROC_THREAD, LOADER, PROFILE
Kernel Groups
Indentified by the use of title case characters
e.g. Base, Diag, Latency, FileIO
Kernel Groups are made up of a collection of Kernel Flags
e.g. SysProf = PROC_THREAD+LOADER+PROFILE
Flags and groups are separated by the ‘+’ token
e.g. xperf.exe -on FileIO+DISK_IO_INIT
Can use a mixture of upper and/or lower case text
e.g. xperf.exe -on sysprof+interrupt+dpc
Merging of Performance Trace Data
Traces can be copied to another system for analysis
The trace file should be “merged” on the collection system
before analysis to include additional system information
xperf -d trace.etl
System and symbol
information
Trace
Merged trace
Kernel trace
XPerf
Using the Windows Performance
Toolkit
Understanding CPU Activity
Windows uses 32 priority levels
The system implements a preemptive,
priority driven scheduler
Priority adjustments can be applied to
threads in the “dynamic” range
At least one runnable thread with the
highest priority will be running
31
Real time
16
15
Dynamic
0
Thread – Priority Current
Thread – Thread State
Context Switching
A switch from one thread to another is known as a
context switch
Switching involves saving the hardware state of a
thread and restoring the state of another
When a thread is scheduled, that thread’s context
switch count is also incremented
The context switch count represents how often a
thread begins running, not how long it ran
System – Context Switches/sec
Thread – Context Switches/sec
Time Accounting Quirks
Looking at total CPU time for each process may not
reveal where the system has spent its time
CPU time accounting is driven by an interrupt timer
which is set by the Hardware Abstraction Layer
Usually at either 10 or 15 msec intervals
Thread execution and context switches that happen
between clock intervals are not accounted for
e.g. a thread runs and enters a wait before the clock fires
Thus threads may run but never get charged
Process – % Privileged Time
Process – % User Time
Time Accounting Prior to Windows Vista
Windows accounted for CPU time based on the
interval clock timer
Thread quantum expiration was not always fair
A thread might get almost no turn
Threads were also charged for interrupts that
occurred while they were running
Idle
Idle
T1
T2
T2
Time Accounting Since Windows Vista
Windows Vista and later reads the Time Stamp Counter
during every context switch
The actual CPU cycles consumed are charged to a thread
Any interrupt time is not charged to the interrupted thread
Allows for more accurate quantum accounting
A thread gets at least one turn and at most will be given
one turn plus an additional tick
Idle
Idle
T1
T1
T2
Troubleshooting High CPU Utilization
Understanding Memory Management
Windows provides two system memory pools
Nonpaged Pool and Paged Pool
Used for system wide persistent data
Prior to Windows Vista, pool sizes were a function of
memory size and whether or not the system was
configured as a server or a workstation
Windows Vista introduced the concept of a dynamic
system address space
Memory – Pool Nonpaged Bytes
Memory – Pool Paged Bytes
Dynamic System Address Space
In 32-bit Windows Vista and later, virtual
memory is assigned as needed
Permits larger paged, nonpaged, and session pools
Components still cannot exceed 2 GB on
32-bit systems
On 64-bit systems, address space regions are
configured to their current maximum limits for
all memory sizes
Troubleshooting Memory Leaks
Understanding Disk I/O
Background I/O, e.g. indexing, disk
defragmenting, interferes with foreground
interactive tasks, e.g. reading e-mail
In earlier systems, the only way to prioritize
work was based on thread CPU priority
Windows Vista introduced I/O priority and
I/O bandwidth reservation
Physical Disk – Current Disk Queue Length
Physical Disk – Disk Transfers/sec
I/O Priority Levels
Critical – Used by the Memory Manager
High – Not currently implemented
Normal – Default priority, e.g. Microsoft Outlook
Low – Task priority, e.g. application prefetching
Very Low – Background activity, e.g. antivirus
Critical
MM
High
Normal
WMP
Hierarchy
Low
Outlook Prefetch
Very Low
Defrag
Indexer
Idle
AV
Troubleshooting Disk I/O
Additional Information
Windows Internals 5th edition
Windows Performance Analysis Developer Center
http://msdn.microsoft.com/performance
Windows Server Performance Team Blog
http://blogs.technet.com/winserverperformance
Ask the Performance Team Blog
http://blogs.technet.com/askperf
Additional Information
David Solomon Expert Seminars offers training
on Windows Internals both as public and private
workshops and public webinars via the Internet
Currently scheduled up and coming classes
Public workshop in London scheduled March, 2010
Public webinar scheduled for January, 2010
Visit http://www.solsem.com for further course
descriptions and up to date information
Resources
www.microsoft.com/teched
www.microsoft.com/learning
Sessions On-Demand & Community
Microsoft Certification & Training Resources
http://microsoft.com/technet
http://microsoft.com/msdn
Resources for IT Professionals
Resources for Developers
Complete an evaluation
on CommNet and enter to
win an Xbox 360 Elite!
© 2009 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should
not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS,
IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.