Transcript - ISQT International

Best Strategies to avoid worst
performance failures in Investment
Banking
Sujatha Badri Narayanan
Karthik Authoor Venkata
Infosys Limited (NASDAQ: INFY)
BFS-US SSDU Research
Abstract
• Investment banking functionalities are grouped into front office, middle
office and back office.
• Performance testing specifically related to trading applications within
investment banking plays a vital role.
• Era has shifted from traditional manual trading in stock exchanges like
NASDAQ, Globex to electronic trading.
• People trade millions of shares in micro seconds with various exchanges
these days using electronic trading.
BFS-US SSDU Research
2
Why this Paper?
“Every large system that works started as a small system that worked.” – Anonymous
“Fast, good, cheap: pick any two.” – Anonymous
“Lots of methodologies have come and gone, paradigms have changed but the requirements are
always the same; Make it good and make it fast.” – Anonymous
In Electronic trading, complex heterogeneous systems are spread over diverse technologies with various
platforms ranging from legacy mainframe to today’s web based technologies are used.
Today’s systems and interfaces are built with their own protocols both at application layer and network
layer.
Even minor performance improvement in these systems will yield mammoth benefits leading to savings in
millions of dollars.
In this whitepaper we will attempt to describe some of the good performance testing
strategies that can be followed to immunize ourselves from disasters in the Investment
banking domain specifically in the trading business.
BFS-US SSDU Research
3
Few Performance experiences as seen by User
Sl.No
User experience Scenarios in the Trading Business
1 Session login failed.
2 Market related data reaching the end user late due to network traffic rendering the info useless
3 Market data graphs failed to pull the real time data from the server
4 Viewing the detailed transaction statements on the web application giving distorted or half information due to delay
in response from server
5 CMP(Current Market price) dashboard showing old market price for the stocks as the real time market update is
slow
6 Placing a trade order from Stock trading web application during market mayhem (lot of trading activities happening
at the same time). Server timed out.
7 Placed a trade order from web application during heavy traffic. Trade got executed but the confirmation never came
back leaving the transaction in dilemma stage
8 Making a critical payment from bank web app to a third party using secure payment network. The money got
debited from my account but due to heavy traffic on the third party server, the transaction failed.
9 Downloading “My” bank statement on the desktop which failed
10
Making a mobile payment from network switches from 3G to Wireless and thus failing the transaction
BFS-US SSDU Research
4
Proposed Strategies for better user experiences
Following are the key strategies proposed, that might help to provide solutions to some of the
performance issues that have been experienced by users as listed in the previous slide:
1) Performance Testing from an Architect’s View – Performance test driven architecture
2) Infrastructure Optimization like Server, Database, Network etc 3) Optimal combination and usage of tools based on technology used – Fiddler, Web Load, VSTS,
Jmeter, LR, Wily Introscope, Dynatrace, Shunra, Strobe, Xcap etc.
4) Component based testing – Stressing the system at component level
5) Seamless Failover Testing – Time taken for switch over during disaster
BFS-US SSDU Research
5
Performance Strategy layers – An End-to-End View
Strategy 1 : Architect’s View
Web Server
App Server
App Server
Load
Balancer
Web Server
Strategy 3 : Optimal combination and usage of tools based on technology
MF
Strategy 5:
Seamless
Failover Testing
BFS-US SSDU Research
DB
Strategy 2:
Infrastructure Optimization
6
Strategy 1: Performance Testing from an Architect’s View
Life Cycle Phases
Performance Strategy Activities
Requirements gathering / analysis / 
validation



Architecture Analysis & Design
BFS-US SSDU Research





System understanding
NFR gathering
Understanding NFR & validating them
Modeling
o NFR
o Workload
Scalability assessment of the architecture
Validation of the design from a Performance stand point
Capacity and sizing planning of the infrastructure
Performance Predictive Modeling
Designing the performance Test strategy
7
Strategy 1: Performance Testing from an Architect’s View.. Contd
Following are the performance test strategy activities (as mentioned against each lifecycle phases in the
previous slide) that can be leveraged
1) Performance Testing from an Architect’s View – Performance test driven architecture
2) Infrastructure Optimization like Server, Database, Network etc 3) Optimal combination and usage of tools based on technology used – Fiddler, WebLoad, VSTS, Jmeter,
LR, Wily Introscope, Dynatrace, Shunra, Xcap etc.
4) Component based testing – Stressing the system at component level
5) Seamless Failover Testing – Time taken for switch over during disaster
BFS-US SSDU Research
8
Strategy 2: Infrastructure Optimization
Why Infrastructure Optimization ?
• Major component in an organization’s CAPEX
• Level of optimization is a key goal for many organizations
• Superior user experience which is the core business should also not get affected
Most Commonly used servers in Investment Banking
Web Server :Tomcat, IIS
App Server : Tomcat, IIS, IBM WebSphere
Database : Oracle 10g, 11g, Microsoft SQL Server
BFS-US SSDU Research
9
Strategy 2: Infrastructure Optimization - Web Server
Suggested optimizations on TomCat:
 Have the latest OS patches, Service packs and JDKs installed as higher versions always have better
performance and better security.
 Have windows server instead of a non server version as non server version will have various
limitations like number of users etc.
 Connector tuning will yield good results – Generally Client creates the connection and once server
sends response to the request connection is closed. One may land up opening 100s of connection to
render a single page – Modify “MaxKeepAliveRequests” to a higher value option to avoid this.
 Tomcat also has a thread pool. Make the best use of the same by adjusting minProcessors and
maxProcessors values.
BFS-US SSDU Research
10
Strategy 2: Infrastructure Optimization - Web Server … Contd
Suggested optimizations on IIS:
 Configure the following parameters like Shutdown worker processes when idle for good performance
 Managing log files will result in good performance in almost all the servers.
 Make sure that only required values are selected for logging and old log files are archived to avoid
space issue.
BFS-US SSDU Research
11
Strategy 2: Infrastructure Optimization - Application Server
Suggested optimizations on WebSphere Application Server for better performance :
Modify Initial Heap Size and Maximum Heap size to 3096
Have Loglevel severity to “error” or “fatal”
 “show application print statements” can be disabled
IBM Service Logs can be disabled (Process Defnitions -> Logging & Tracking -> IBM Service Logs)
Have a strategy around connection pool management in WAS
BFS-US SSDU Research
12
Strategy 2: Infrastructure Optimization – Database Server
Suggested optimizations on Oracle DB Server for better performance :
Ensure that the database is loaded to the required volume before Load testing.
As in any other server system resources monitoring plays a vital role in DB server also.
Besides CPU, Memory, ensure that Disk I/ O is monitored.
Ensure that all the disks are properly loaded.
Other major aspects of Oracle tuning are :Denormalizing a database – If your database is doing too many joins then you can consider
this option to improve performance
Indexing (B-Tree Concept)
- is used in databases to speed up retrieval process
-is one of the most time tested technique that can be used to improve performance
 Full Table Scan - it is better to go for full table scan for smaller tables
BFS-US SSDU Research
13
Strategy 2: Infrastructure Optimization – Database Server .. Contd
Tuning SQL itself is another interesting topic :
You can make use of AWR logs and other monitoring tools
to know what queries take more time and
what queries are called often and then select the queries that should be tuned.
Following are few samples of SQL tuning : If you are retrieving from multiple tables, the sequence of the tables matters a lot for
performance. Optimizers (if tuned properly – cost optimization)does help in making such
decisions.
If it is a read only table read once and cache them so that we can avoid reading multiple times
Do not read the entire table when only few columns are required. DB filters can be handy
here.
BFS-US SSDU Research
14
Strategy 3: Optimal combination and usage of tools
Selecting an optimal combination of tools plays a vital role in identifying bottlenecks.
While selecting load generation tools we may need to take into the account the following key
factors:
Protocol Support required for your application like :
Http, Win socket, SOAP, WAP, IMAP, POP3, EJB, RMI-Java, COM/DCOM, Tuxedo, Seibel, SAP
R/3, Terminal Emulation, SSL, XML, PeopleSoft, VT100-520, ODBC, FTP, Streaming Media,
LDAP, TCP/IP, UDP, MMS, Microsoft .NET SOAP Stack, CORBA, Oracle OCI, Ajax
Ease of Parameterization and Correlation
Network address management
User ramp up / down facility
Scheduling tests
License details
Customer Support
BFS-US SSDU Research
15
Strategy 3: Optimal combination and usage of tools… Contd
While selecting monitoring tools we may need to take into the account the following key factors:
Technology support required for your application - .Net, J2EE, Mainframe etc
Supported system resources monitoring
Depth in analysis For e.g. does it help in identifying bottlenecks at code level ? Query level ?
License details
Customer Support
There are many Performance tools in the market but based on our experience in Investment banking
we find that the following tools are most commonly used.
Strategic Areas / Technology
.Net
J2EE
Load Generation Tools
VSTS
Monitoring Tools
IIS Debug
HP Diagnostics, Wily Introscope, Dynatrace, Site Strobe
Diagnostics scope
BFS-US SSDU Research
HP LR , Radview’s Web load, Compuware
QALoad, Borland ‘s Silk Performer, Microsoft
Web Application Stress Tool (WAS)
Mainframe
16
HP LR using
RTE protocol
Strategy 3: Optimal combination and usage of tools… Contd
There are two types of monitoring tools
Agent based tools like HP Diagnostics ,Wily Introscope
Agent less tools like Site Scope
Sample Agent based Monitoring Tool output – Wily Introscope
Detailed analysis up to code / query level can be done in agent based monitoring.
BFS-US SSDU Research
17
Strategy 4: Component based Testing
SLAs should be broken down during architecture phase for each of the components.
Components should be individually load/volume tested to make sure that they meet required SLA.
This will ease bottleneck identification and help in addressing the same at an earlier stage rather
than struggling during final phase of the project.
BFS-US SSDU Research
18
Strategy 5: Failover Testing
In investment banking as availability at the right time matters most, failover testing is essential.
In failover testing , the behavior of the system will be tested by bringing down the servers in a logical
way – web, app etc.
In many instances CPU utilization was high and later during analysis we realized that Load Balancers
were not working as expected.
Make sure that your strategy covers testing load balancers also.
BFS-US SSDU Research
19
The Way ahead
Designing for the right performance and ensuring that performance is engineered into the product /
application is key.
Specifically in investment banking business , organizations cannot afford to incur higher expenditure on
application failures and higher opex on unnecessary infrastructure, hence performance testing
becomes an indispensable activity for optimizing the quality of the application
We recommend organizations to view performance quality from the bigger picture and adopt the best
strategies which were discussed during these sessions so that business at large is benefited thereby
making organizations less vulnerable to competitive pressures.
BFS-US SSDU Research
20
Conclusion and Results
Performance testers should also focus on performance engineering standpoint in addition to load
testing / volume testing strategies.
User experience Scenarios
1
Session login failed.
Strategy
Load beyond expected may be the reason. Capacity
Modeling could have avoided this situation – Strategy 1
2 Market related data reaching the end user late
Response time not meeting SLAs – Strategy 2 & 3
due to network traffic rendering the info useless
3 Market data graphs failed to pull the real time May be the server responsible for pulling the data is
data from the server
down. Strategy 5
4 Viewing the detailed transaction statements on
the web application giving distorted or half
Database Server response may be slow. Strategy 3
information due to delay in response from
server
5 CMP(Current Market price) dashboard showing
Possible reason could be bottleneck in one of the
old market price for the stocks as the real time
components – Strategy 4
market update is slow
BFS-US SSDU Research
21
Conclusion and Results … Contd
User experience Scenarios
Strategy
6 Placing a trade order from Stock trading web
Load beyond expected may be the reason.
application during market mayhem (lot of trading
Capacity Modeling / Proper Workload modeling
activities happening at the same time). Server timed
could have avoided this situation – Strategy 1
out.
7 Placed a trade order from web application during
Incomplete transaction due to poor response
heavy traffic. Trade got executed but the confirmation
time. Possible reason could be bottleneck in
never came back leaving the transaction in dilemma
one of the layers / components – Strategy 4
stage
8 Making a critical payment from bank web app to a
third party using secure payment network. The
Possible reason could be bottleneck in one of
money got debited from my account but due to
the layers / components – Strategy 4
heavy traffic on the third party server the
transaction failed.
BFS-US SSDU Research
22
References and Acknowledgements
References
 http://msdn.microsoft.com
 Tomcat – The definite guide by Jason Brittain & Ian F. Darwin
 Performance Tuning for Apache Tomcat by Mark Thomas
 Oracle Performance Tuning by Peter Corrigan and Mark Gurry
 Performance Testing Advanced Certification material from Infosys
Acknowledgements
 Vijay from Infosys Infrastructure Management Services
 Sanjeeb Kumar from Financial Services team for sharing the user experiences
 Finacle SPE team
BFS-US SSDU Research
Q&A:
[email protected]
[email protected]
BFS-US SSDU Research
24