Transcript Start the presentation

Web Performance,
Load, and Stress Tests
Mainframe and
Open System
Network
Natural Conference
Boston
Dieter W. Storr
Los Angeles Times
May 2004
[email protected]
Performance, Load, and Stress Tests
Contents
What are performance, load, and stress tests?
Why and when should you test?
What response time do you expect?
What parts play a role in the network?
Network basics
Logical and physical transactions
May 2004
Dieter W. Storr [email protected]
2
Performance, Load, and Stress Tests
Contents
What tools can you use to test
these parts?
When should you start performance
tests?
Web site performance, analysis and
tuning
Checklist
Links
May 2004
Dieter W. Storr [email protected]
3
What are Performance Tests?
Test each part of the system,
such as web server, web
applications, databases, etc.
Tests can run usually without
requesting page images in
order to concentrate the testing
on the script and code itself
?
?
?
?
May 2004
Dieter W. Storr [email protected]
4
What are Load Tests?
Load tests are performed by
testing the website using the
best estimate of the traffic your
website needs to support.
Define the maximum time it
should take for a page to load -from a usability and customer
experience standpoint, not from
a technical one.
?
?
?
?
May 2004
Dieter W. Storr [email protected]
5
What are Stress Tests?
To simulate “brute force”
attacks that apply excessive load
to system, such as web server,
web applications, databases, etc.
To estimate the maximum load
that your system can support
To learn the traffic thresholds of
your system
How it will respond after
exceeding its threshold
May 2004
Dieter W. Storr [email protected]
?
?
?
?
6
What are Stress Tests?
Ramp test
A variation of stress test
The number of users is increased
over the life of the test
?
?
?
?
May 2004
Dieter W. Storr [email protected]
7
Why Performance Tests?
Determine that all functions do what they
meant to do (function test)
Ensure that tasks and objectives are easily
to be accomplished (usability test)
Guarantee a result from the website in an
acceptable time (performance test)
May 2004
Dieter W. Storr [email protected]
8
Why Performance Tests?
Optimize each part of the system for
increased web traffic
Ensure that the end-user’s mouse click is
not met with silence
May 2004
Dieter W. Storr [email protected]
9
Why Load Tests?
To compare the maximum time it should
take for a page to load with the effective
used time.
Calculate the impact of exceeding that
maximum time. Will you lose sales or
prospective customers?
Calculate the anticipated load and load
pattern for the website.
Determine to do some work on the
application or webserver.
May 2004
Dieter W. Storr [email protected]
10
Why Stress Tests?
Determine the maximum load
Test excessive system load
Determine the traffic thresholds of your
system
Experience the reaction of the system after
exceeding its threshold
Ramp tests can help to determine what
maximum load a server can handle while
providing optimal access to web resources
May 2004
Dieter W. Storr [email protected]
11
When Performance Tests?
You cannot start early enough!
Start before a single line of code is written





Network
Load balancer
Application
Database
Web servers
During development, all software engineers and
QA team should have access to performance test
tools
May 2004
Dieter W. Storr [email protected]
12
When Load Tests?
As soon as web
pages are working
QA team should
conduct the first load
tests
Performance testing
should be part of the
regular testing
routine each day for
each build of the
software
May 2004
Dieter W. Storr [email protected]
13
What Response Time Do You Expect?
Does your company set limits?
KEEP in mind:
Many websites today have a serious
business mission => to take money
Reason: Advertisement opportunities,
selling retail products, etc.
Websites and applications need to be
up and running at all times
May 2004
Dieter W. Storr [email protected]
14
What Response Time Do You Expect?
Response
Time
< 0.1 s
< 1.0 s
<10 s
> 10 s
User's View
User feels that the system is reacting
instantaneously
The user experience is not
compromised. Although the user is
unhappy with the wait, they are still
focused on the current web page
As wait times get close to 10s, studies
have shown that the likelihood of user
distraction increases greatly
User is most likely distracted from the
current website and loses interest
Source: Jakob Nielsen, Designing Web Usability, New Riders Publishing, 2000
May 2004
Dieter W. Storr [email protected]
15
What Response Time Do You Expect?
95% of all web requests should be
processed in less than 10 seconds!
BE AWARE:
If performance slows
even a little, fickle web
users are likely to jump
quickly to a competitor’s
site
May 2004
Dieter W. Storr [email protected]
16
What Parts Play a Role in The Network?
Basic Connections
Web Browser
Web Server
CGI
NATURAL
ADABAS
May 2004
Dieter W. Storr [email protected]
17
What Parts Play a Role in The Network?
Hubs
Switches
Routers
Firewalls
Load Balancer
Application Servers
Database Servers
Web Servers
May 2004
Dieter W. Storr [email protected]
18
Network Basics
Hub / Switch / Bridge / Router / Firewall
In data communications, a hub is a place of
convergence where data arrives from one or
more directions and is forwarded out in one or
more other directions. A hub usually includes a
switch of some kind.
See USB 7-Port Hub
To connect several
USB devices
May 2004
Dieter W. Storr [email protected]
19
Network Basics
Hub /
Switch / Bridge / Router / Firewall
In telecommunications, a switch is a network
device that selects a path or circuit for sending
a unit of data to its next destination.
See 8-Port Network Switch
May 2004
Dieter W. Storr [email protected]
20
Network Basics
Hub / Switch / Bridge / Router / Firewall
A switch may also include the function of the
router, a device or program that can determine
the route and specifically what adjacent
network point the data should be sent to.
In general, a switch is a simpler and faster
mechanism than a router, which requires
knowledge about the network and how to
determine the route
May 2004
Dieter W. Storr [email protected]
21
Network Basics
Hub / Switch / Bridge / Router / Firewall
In telecommunication networks, a bridge is a
product that connects a local area network
(LAN) to another local area network that uses
the same protocol (for example, Ethernet or
token ring).
May 2004
Dieter W. Storr [email protected]
22
Network Basics
Hub / Switch / Bridge /
Router / Firewall
On the Internet, a router is a device or, in
some cases, software in a computer, that
determines the next network point to which a
packet should be forwarded toward its
destination.
See 4-Port Cable/DSL Router
May 2004
Dieter W. Storr [email protected]
23
Network Basics
Hub / Switch / Bridge / Router /
Firewall
A system designed to prevent unauthorized
access to or from a private network. Firewalls
can be implemented in both hardware and
software, or a combination of both.
PIX Firewall from CISCO
Private Internet Exchange Firewall
Gauntlet Firewall from NAI
Network Associates Inc.
May 2004
Dieter W. Storr [email protected]
24
Network Basics
Bridge
May 2004
Dieter W. Storr [email protected]
25
Internet
Firewall
SSL
CISCO Router
Web Server
e.g. Apache
Appl. Server
Appl. Server
e.g. WebLogic
BEA
Screen Scraper
Screen Scraper
Web Server
Firewall
Open
System
Mainframe
e.g. Attachmate
Firewall
Com-plete
NATURAL
ADABAS
May 2004
CICS
COBOL
VSAM
Dieter W. Storr [email protected]
Oracle
FirstLogic
DBs
26
Logical and Physical Transactions?
Web Browser
Web Server
Appl. Server
NATURAL
ADABAS
May 2004
Dieter W. Storr [email protected]
27
Response Time / Click Time Tests
Function:
Display my
account
Web Browser
Web Server
Appl. Server
NATURAL
ADABAS
May 2004
Dieter W. Storr [email protected]
28
What Tools Can You Use to Test?
Web Server Stress Tool
Web Browser
User simulation
Logging period
Select URLs
URL recorder
Data merging
Browser simulation
Reviewing summary and detailed log results
Analyzing graphical results
May 2004
Dieter W. Storr [email protected]
29
CSS Stress Test
5 Users 10 Clicks
25 Sep 2003, 11:52-11:53
May 2004
Dieter W. Storr [email protected]
30
CSS Stress Test
5 Users 10 Clicks
14 Oct 2003, 14:05
May 2004
Dieter W. Storr [email protected]
31
09/25/2003 11:52:02 AM --- WEBSERVER STRESS TOOL
User #1 Click #1 started, used URL #1 (CSS-Prod-1) POST-DATA=
"[email protected]&password=<snip>
User #1 Click #1: Request 1 HTTP State: Initialized (0 ms)
User #1 Click #1: Request 1 HTTP State: DNS Lookup begin (7 ms)
User #1 Click #1: Request 1 HTTP State: DNS Lookup done (17 ms)
User #1 Click #1: Request 1 HTTP State: Connected (504 ms)
User #1 Click #1: Request 1 HTTP State: Waiting for header (512 ms)
User #1 Click #1: Request 1 HTTP State: Receiving Header (5937 ms)
(=Time to First Byte)
User #1 Click #1: Request 1 HTTP State: Waiting for body (5947 ms)
User #1 Click #1: Request 1 HTTP State: Done (5999 ms)
User #1 Click #1: CLICK-Request 1: Time=6008 ms, TFB=5937ms,
Bytes=16038, HTTP-StatusCode=200 (OK)
User #1 Click #1: !Success! All requests of click done (6046 ms)
May 2004
Dieter W. Storr [email protected]
32
What Tools Can You Use to Test?
APAS/INSIGHT
ADABAS
DWSCWUSE: SUMMARIZE (LS=150)
COUNT (HD='COMMANDS')
PERCENT (HD='% OF TOTAL' PF=G10)
SUM(TOTAL-IO) (HD='TOTAL IO')
PCT(TOTAL-IO) (HD='% OF TOTAL' PF=G10)
MEAN(TOTAL-IO) (HD='AVG IO')
MAX(TOTAL-IO) (HD='MAX IO')
MEAN(DURATION) (HD='AVG DUR')
MAX(DURATION) (HD='MAX DUR')
MEAN(ENQ-TIME) (HD='AVG ENQ')
MAX(ENQ-TIME) (HD='MAX ENQ')
BY USER-ID
WHERE USER-ID(2) = 'CW'
INSIGHT-LINES = 30
PRINT TO STRESS01;
May 2004
Dieter W. Storr [email protected]
33
ADABAS on Enterprise Server
CSS Stress Test
5 Users 10 Clicks
25 Sep 2003, 11:52-11:53
USER
ADABAS
ID
COMMANDS % OF TOTAL
-------- ---------- ----------CW0016
269 |======
CW0017
269 |======
CW0018
269 |======
CW0019
369 |=========
CW0020
369 |=========
CW0021
369 |=========
CW0022
369 |=========
CW0023
369 |=========
CW0024
269 |======
CW0025
269 |======
---------- ----------========
3,190 |========>
May 2004
AVG DUR SEC MAX DUR SEC
------------ --------0.00033
0.006
0.00052
0.008
0.00036
0.005
0.00058
0.014
0.00041
0.011
0.00043
0.008
0.00045
0.008
0.00038
0.007
0.00034
0.006
0.00049
0.009
------------ --------0.014
Dieter W. Storr [email protected]
34
What Tools Can You Use to Test?
APAS/INSIGHT
0090
0100
0110
0120
0130
0140
0150
0160
0170
0180
0190
0200
May 2004
ADABAS
DWSCWOBX: EXTRACT
(LS=150)
NAT-LOAD-COUNT (HD='OBJ-LOAD')
DATE TIME
NAT-PROGRAM CMD
USER-ID
GLOBAL-ID
WHERE USER-ID(2) = 'CW' AND
(NAT-PROGRAM = 'IWS00P00' OR
NAT-PROGRAM = 'IWD10P10')
INSIGHT-LINES = 200
PRINT TO STRESS01
;
Dieter W. Storr [email protected]
35
ADABAS on Enterprise Server
CSS Stress Test
5 Users 10 Clicks
25 Sep 2003, 11:52-11:53
NATURAL
PROGRAM OBJ-LOAD
-------- -------IWD10P10
50
-------50
May 2004
Dieter W. Storr [email protected]
36
Response Time / Click Time Tests
200ms - 20s
Function:
Display my
account
Web Browser
Web Server
Appl. Server
NATURAL
0.00033s - 0.014s
May 2004
ADABAS
Dieter W. Storr [email protected]
37
What Tools Can You Use to Test?
STRESS -- self-written
ADABAS
1 Start process with heavy CPU workload
2 Start process with many ADABAS Calls
3 Start process with heavy CPU workload and
many ADABAS calls
4 Start process with many ADABAS calls and
excessive terminal I/O's
5 Start asynchronous processes (ADABAS & CPU)
May 2004
Dieter W. Storr [email protected]
38
May 2004
Dieter W. Storr [email protected]
39
May 2004
Dieter W. Storr [email protected]
40
May 2004
Dieter W. Storr [email protected]
41
May 2004
Dieter W. Storr [email protected]
42
May 2004
Dieter W. Storr [email protected]
43
What Tools Can You Use to Test?
TP Monitor
Com-plete
UCTRL
NETSPY
TMON/MVS
May 2004
Dieter W. Storr [email protected]
44
UCTRL
17:49:14
TID
Function
-------------------ADABAS interface ...
ADABAS statistics ..
Buffer activities ..
Common storage .....
DB2 thread status ..
FBPM statistics ....
Message log ........
6
---
FC
-AI
AS
BA
CS
D2
FB
CM
COMTEST
USER ZINDWS
COM-PLETE CONTROL FACILITY ---
Function
-------------------Program lookaside ..
Program in-stor dir.
Resident programs ..
Find program .......
Queue overview .....
Roll activities ....
Server overview ....
Terminal overview ..
Terminal status ....
FC
-IP
OP
RP
FP
QO
RA
SO
TO
TS
Function
-------------------Task groups ........
Task list ..........
Thread groups ......
Thread subgroups ...
Thread list ........
User activities ....
User status.........
VSAM statistics ....
03/16/04
UCT1
FC
-PG
PL
TG
SG
TL
UA
US
VS
Print statistics ... PR
Select function ....
Operand ............
Operator command:
Enter-PF1---PF2---PF3---PF4---PF5---PF6---PF7---PF8---PF9---PF10--PF11--PF12--End
May 2004
Dieter W. Storr [email protected]
45
NETSPY
Time Host Avg-Resp User Avg-Resp Worst Host_Resp No. Session No. Input No. Output
13:30
0.1
0.1
10.9
14
3777
3996
13:45
0.0
0.0
0.9
4
749
780
14:00
0.1
0.1
5.7
25
10342
11029
14:30
0.1
0.1
4.1
2
2094
2114
Response Time in Seconds
Netspy Response Time
12.0
10.0
8.0
Host Avg-Resp
6.0
User Avg-Resp
4.0
Worst Host_Resp
2.0
0.0
13:30
13:45
14:00
14:30
Test Time
May 2004
Dieter W. Storr [email protected]
46
TP Monitor Com-plete Consumption
CPU TOTAL
19.31
CPU %
1.60
Paging Rate
0.00
IO Rate
26.50
COMTEST EXECUTION DETAIL
30.00
VALUE
25.00
20.00
15.00
Series1
10.00
5.00
0.00
CPU TOTAL
CPU %
Paging Rate
IO Rate
CATEGORY
May 2004
Dieter W. Storr [email protected]
47
TMON
Alerts
Average response time
CPU usage
I/O usage
Number of ended transactions rate
May 2004
Dieter W. Storr [email protected]
48
What Tools Can You Use to Test?
Screen Scraper
Attachmate
Run Time Application
Management & Control Services (MSC) -monitoring pools and sessions
Java console running in foreground
May 2004
Dieter W. Storr [email protected]
49
What Tools Can You Use to Test?
Screen Scraper
Attachmate Run Time Application
Number of sessions to host (mainframe)
Maximum sessions
Number of failures
Queue size
May 2004
Dieter W. Storr [email protected]
50
What Tools Can You Use to Test?
Attachmate Servers
NT utilities
Sampling at 10-second
intervals
May 2004
Dieter W. Storr [email protected]
Screen Scraper
51
What Tools Can You Use to Test?
Screen Scraper
Attachmate Servers
Analyze results:
CPU utilization
Memory utilization
Run queues
Swap space utilization
Disk I/O - most busy which disk hit the hardest
I/O wait > 10%
Page file - swaps
May 2004
Dieter W. Storr [email protected]
52
What Tools Can You Use to Test?
Cisco 2900 Switch
Manual testing/sampling
Content Server
Switch
Analyze results
Load on network components
Packet throughput
May 2004
Dieter W. Storr [email protected]
53
What Tools Can You Use to Test?
Cisco 4000 Router (10BaseT)
Manual testing/sampling
Router
Analyze results
Load on network components
http://www.scd.ucar.edu/nets/devices/routers/ios_analy
sis.html
Packet throughput
Router Configuration:
http://www.swcp.com/~jgentry/topo/cisco.htm
May 2004
Dieter W. Storr [email protected]
54
Dedicated line broadband speeds
and prices
 T-1 - 1.544 megabits per second (24 DS0 lines)
Ave. cost $400.-$650./mo.
 T-3 - 43.232 megabits per second (28 T1s)
Ave. cost $6,000.-$16,000./mo.
 OC-3 - 155 megabits per second (100 T1s)
Ave. cost $20,000.-$45,000./mo.
 OC-12 - 622 megabits per second (4 OC3s) no price
 OC-48 - 2.5 gigabits per seconds (4 OC12s) no price
 OC-192 - 9.6 gigabits per second (4 OC48s) no price
Source: http://www.infobahn.com/research-information.htm
prices updated: 16 March 2004
May 2004
Dieter W. Storr [email protected]
55
What Tools Can You Use to Test?
RDBMS
Oracle
Oracle utilities
Application-specific log files
Performance Monitoring and Tuning Tools
•
TKProf
•
UTLBSTAT.SQL and UTLESTAT.SQL - Begin and end
stats monitoring
•
Statspack
•
Oracle Enterprise Manager - Tuning Pack
May 2004
Dieter W. Storr [email protected]
56
What Tools Can You Use to Test?
Oracle
Performance Monitoring and
Tuning Tools (cont”)
•
•
RDBMS
DBXRAY: A real-time graphical snapshot of the health of your
Oracle database- anytime, anywhere.
Patrol: Intelligent and non-intrusive for proactive database
monitoring, including Oracle Parallel Server and Oracle
Symmetric Replication environments.
•
SQL-Explorer for Oracle The most advanced SQL tuning
tool available. Identify, model and tune problematic SQL
statements to boost application performance.
May 2004
Dieter W. Storr [email protected]
57
What Tools Can You Use to Test?
Oracle
Analyze results:
Number of Oracle processes
Buffer hit cache
Block I/O rate
Number of cursors created
Number of connections
Log information regarding number of
registrations, starts, etc.
May 2004
Dieter W. Storr [email protected]
RDBMS
58
What Tools Can You Use to Test?
SSL Accelerator
•
SSL
Secure Socket Layer (protocol between
HTTP and TCP) -- appliance or card
Manual testing/sampling
Analyze results:
Load on network components
Packet throughput
Optimizing SSL processing for Web
 http://www.nwfusion.com/news/tech/2000/1023tech.html?nf
May 2004
Dieter W. Storr [email protected]
59
What Tools Can You Use to Test?
PIX
Manual testing/sampling
Firewall
Analyze results:
Load on network components
Packet throughput
May 2004
Dieter W. Storr [email protected]
60
What Tools Can You Use to Test?
Gauntlet (several)
Manual testing/sampling
Firewall
Analyze results:
Memory
Disk I/O
Full system log
May 2004
Dieter W. Storr [email protected]
61
What Tools Can You Use to Test?
Sun OS
Solaris utilities
Sampling at 10-second intervals
May 2004
Dieter W. Storr [email protected]
Web Server
62
What Tools Can You Use to Test?
Sun OS
Web Server
Analyze results:
CPU utilization
Memory utilization
Run queues
Swap space utilization
Disk I/O -- most busy which disk hit the hardest
I/O wait > 10%
Page file -- swaps
May 2004
Dieter W. Storr [email protected]
63
What Tools Can You Use to Test?
Apache Application
Log Analyzer
Web Server
Analyze results:
Number of sessions
Bytes per session
Source address
Destination address
May 2004
Dieter W. Storr [email protected]
64
What Tools Can You Use to Test?
Sun OS
Solaris utilities
Sampling at 10-second
intervals
May 2004
Dieter W. Storr [email protected]
Application
Server
65
What Tools Can You Use to Test?
Sun OS
Application
Analyze results:
Server
CPU utilization
Memory utilization
Run queues
Swap space utilization
Disk I/O -- most busy which disk hit the
hardest
I/O wait > 10%
Page file -- swaps
May 2004
Dieter W. Storr [email protected]
66
What Tools Can You Use to Test?
BEA WebLogic Application
WebLogic administrative console
Spotlight
May 2004
Dieter W. Storr [email protected]
Application
Server
67
What Tools Can You Use to Test?
BEA WebLogic Application
Analyze results:
Number of active queues
Number of connections
Number of active sockets
Types of access
Time of access
May 2004
Dieter W. Storr [email protected]
Application
Server
68
Web Site Performance
Analysis and Tuning
In-House or Web Caching
Near the network edge, e.g. proxy cache or
browser: static HTML, JavaScript, and image files
Cache-control header “no-cache” “re-fetch”
“expiry date”
Compression
HTML is compressed before being sent through
the network.
May 2004
Dieter W. Storr [email protected]
69
Web Site Performance
Analysis and Tuning
SSL Acceleration (box or card)
Consider the number of transactions per second
(TPS). TPS will be more important on SSL
Acceleration appliances than cards.
An SSL Acceleration hardware appliance (box) can
deal with transactions for all your web servers and
scale quickly.
Network Load Balancing
Round Robin
Lowest response time
Least connections
Proximity
May 2004
Dieter W. Storr [email protected]
70
Checklist -- Getting Started?
Determine your environment
•
Mainframe, Open System, Middleware
Evaluate Stress Test Software
•
Write your own or buy one
Understand the different components
of a web site download
•
•
•
•
•
May 2004
DNS lookup
Redirect
First byte download
Base page download
Content download
Dieter W. Storr [email protected]
71
Checklist -- Getting Started?
What kind of content?
•
•
Dynamic or static
.html or image
Number of servers
Where does traffic comes from
•
May 2004
Internet, Intranet, or Extranet
Dieter W. Storr [email protected]
72
Create a Stress Test Plan
Test Organization
•
•
Overview
Participants
• Technical team
• Coordination team
• Application team
•
Objectives
• Infrastructure
• Application
•
•
May 2004
Assumptions / pretest conditions
Communication plan
Dieter W. Storr [email protected]
73
Create a Stress Test Plan
Environment / Test Components
•
•
•
Overview
Overall success criteria
Transactions to be tested
• Logical and physical
• Application functions
•
•
•
•
Data indentification
Script development
High-level system diagram
Components
• Responsibility
• Metrics
• Tool / method
May 2004
Dieter W. Storr [email protected]
74
Create a Stress Test Plan
Sequence of Events
•
•
•
May 2004
Summary
Process
Schedule of events
Dieter W. Storr [email protected]
75
Web Speed is no Magic!
May 2004
Dieter W. Storr [email protected]
76
Links to Interesting Websites
http://www.edwardtsai.com/web-siteperformance.html
http://www.nwfusion.com/news/tech/2000/102
3tech.html?nf
http://www.scd.ucar.edu/nets/devices/routers/
ios_analysis.html
http://www.swcp.com/~jgentry/topo/cisco.htm
May 2004
Dieter W. Storr [email protected]
77
Thank You!
May 2004
Dieter W. Storr [email protected]
78