Transcript Performance Testing - An Insider

performance testing –
an insider
Manoj Jain
Manager – Software Engineering, CA Tech.
Mar 26, 2011
agenda
− Understanding performance testing
− Case study
− Effective performance testing
− Activities Identification
− Best Practices
− Questions
2
April 6, 2016
Performance Testing - An Insider
Copyright © 2010 CA. All rights reserved.
understanding performance testing
performance testing
4
April 6, 2016
What
Testing conducted to evaluate the
compliance of a system or component
with specified performance
requirements (IEEE).
Why
To meet/exceed the customers’
expectations as per the agreed SLAs
(Service Level Agreements) for the
response time of different business
functions.
How
- Outsourcing
- In house with Purchased Tools
- In house with Leased Tools
Performance Testing - An Insider
Copyright © 2010 CA. All rights reserved.
a case study
Forward Inc. – A Case Study
Forward Inc. is a leading firm in providing Banking & Financial
solutions to medium and large scale organizations
Forward Inc. is eager to conduct performance testing as:
 More number issues are related to poor response time.
 Customers are demanding very high response time
To resolve the issues at the earliest, the company leased a
leading performance testing tool
6
April 6, 2016
Performance Testing - An Insider
Copyright © 2010 CA. All rights reserved.
Forward Inc. – Queries
 What other business functions
should be included in the
performance test?
 Which platform (HW/OS) should be
tested first?
 Which database should be used in
performance test?
 What should be the record volume
in the database?
 How do we monitor the resource
usage (CPU/Memory) at the
server(s)?
7
April 6, 2016
Performance Testing - An Insider
Copyright © 2010 CA. All rights reserved.
Forward Inc. – Issues Faced
— Performance test suite was aborted as
− Test master machine ran out of space
− Rollback segment was full due to large number of operations
− Test tool was not able to validate licenses from License Server
— Large number of users terminated abnormally as
− Users’ password expired during suite execution - 20%
− A show-stopper occurred due to a code fix - 10%
− An agent machine ran out of memory – 5%
8
April 6, 2016
Performance Testing - An Insider
Copyright © 2010 CA. All rights reserved.
Forward Inc. – Issues Faced (contd.)
 The report interpretation was very difficult as checkpoints are
listed in alphabetical order instead of their chronological order
 Test data needs to be prepared again as backup data cannot be
restored
 The response time was very poor as during suite execution,
some other activity was also being performed on the database
server
 Required HW/SW is not available for dial-up network testing
 QA team is idle when development team is using the testing
environment for fixing performance issues
9
April 6, 2016
Performance Testing - An Insider
Copyright © 2010 CA. All rights reserved.
effective methods of performance testing
performance testing phases & activities involved
Phase
Tool required
Activity
Project Kick-off
No
NA
Test Planning/Scheduling
No
NA
Test Bed/Test Data Preparation
No
NA
Virtual User Script Recording
Yes
• To record & validate VU scripts
• To define and load the data pool
Performance Test Suite
Preparation
Yes
To prepare performance test suit
with Agents, User Groups,
Scenarios etc.
Test Suite Execution
Yes
To execute the test suite with predecided number of user load
Publishing Performance Test
Results Report
No
11
April 6, 2016
Performance Testing - An Insider
Copyright © 2010 CA. All rights reserved.
best practices – project kick off
 Deciding test environment i.e. HW/SW/DB
 Deciding number of virtual users
 Deciding Expectation Matrix
 Escalation Matrix to handle any delays
 Deciding team size and identifying team members from QA, Dev., DBA &
Infrastructure team
 Identification of business functions (Workload Analysis)
12
April 6, 2016
Performance Testing - An Insider
Copyright © 2010 CA. All rights reserved.
best practices – test planning
 Arrange separate test environments for QA and Dev. team
 Time required by individual teams to conduct their assigned activities i.e.
 QA – Scripting/Test data/Suite Exec./Test Report
 Dev – Identification & fixing of performance issue, testing of tuned
sources, tuned source deployment
 DBA – Analysis of database usage reports, highlighting queries, which
are having high execution time
 Activities should be broken down to lowest possible level with
dependencies
 Time to backup and restore database with verification and analysis
13
April 6, 2016
Performance Testing - An Insider
Copyright © 2010 CA. All rights reserved.
best practices – VU scripting
 Script naming conventions
Convention: Version_Module_ScriptSequence_Script
Example:
r11_Acc_01_GenRep
Where: r11 – Version of the application under test
Acc – Abbreviation of the module e.g. Accounting
01 – The sequence number out of total virtual user scripts
GenRep – Abbreviation of the Script e.g. Generate Report
 Checkpoint naming convention
Convention: Version_Module_ActionSequence_Action
Example:
r11_Acc_GenRep_01_Login
Where: r11 – Version of the application under test
Acc – Abbreviation of the module e.g. Accounting
GenRep – Abbreviation of the action e.g. Generate Report
01 – The sequence number out of total virtual user scripts
Notes:
- The action sequence number is local within a script
- Actions are numbers within a script as per the chronological order
 No duplicate check point names
14
April 6, 2016
Performance Testing - An Insider
Copyright © 2010 CA. All rights reserved.
best practices – VU scripting (contd.)
Without Conventions
CloseRep
CloseRep
Gen
Gen
Login
Login
Logout
Logout
RepModule
RepModule
SelectRep
SelectRep
15
April 6, 2016
Performance Testing - An Insider
With Conventions
r11_Acc_GenRep_01_Login
r11_Acc_GenRep_02_RepModule
r11_Acc_GenRep_03_SelectRep
r11_Acc_GenRep_04_Gen
r11_Acc_GenRep_05_CloseRep
r11_Acc_GenRep_06_Logout
r11_Finan_GenRep_01_Login
r11_Finan_GenRep_02_RepModule
r11_Finan_GenRep_03_SelectRep
r11_Finan_GenRep_04_Gen
r11_Finan_GenRep_05_CloseRep
r11_Finan_GenRep_06_Logout
Copyright © 2010 CA. All rights reserved.
best practices – test data preparation
 Backup of database schema should be performed at each milestone.
 Test data prepared using SQL Scripts, tools should be validated.
 Database schema should be analyzed at regular intervals.
 Backup of database schema should NOT be stored on the same database
server.
 No resource consuming activity should be performed on the database
server during test data creation.
16
April 6, 2016
Performance Testing - An Insider
Copyright © 2010 CA. All rights reserved.
best practices – test suite preparation
 Test suite should be configured using a pre-decided think time value
 Logical group names should be assigned to user groups
 It is preferred to assign users as % instead of fixed nos.
 Agent machines should be logical
 While selecting an agent machine, it needs to be ensured that agent
hardware and OS is well supported by test tool
 It needs to be ensured that agent machine is neither over-allocated nor
under-allocated
17
April 6, 2016
Performance Testing - An Insider
Copyright © 2010 CA. All rights reserved.
best practices – test suite execution
 Ensure that all user ids used in the VU script are working fine and valid for
large number of days
 Ensure that server(s), test master and test agent machine(s) have enough
free disk spaces
 Paging file size of the agent machine should be increased to maximum.
 Ensure that all monitors have been configured properly to monitor the
servers’ resources usage
 Response time & working of the application under test should be verified
manually before executing the suite
 During suite execution, response time of the application under test might
be checked manually
 During suite execution, no other activity should be performed on the
machines being used in the test
18
April 6, 2016
Performance Testing - An Insider
Copyright © 2010 CA. All rights reserved.
best practices – test results report
 Documenting executive summary
 Test methodology followed
 Test results with different percentiles
 Test environment configuration
 Summary of Workload Distribution
 List of all scripts along with checkpoint names
 Comparative display of test results in graphical form
19
April 6, 2016
Performance Testing - An Insider
Copyright © 2010 CA. All rights reserved.
thank you
[email protected]