Community Performance Testingx
Download
Report
Transcript Community Performance Testingx
Sakai Community
Performance Testing
Working Proof of Concept
Sakai Performance Working Group
Linda Place, Chris Kretler, Alan Berg
Universities of Michigan & Amsterdam
1 July 2008
Performance Testing Overview
• Part of QA process
– May include stress, capacity, scalability, reliability
as well as performance (response time,
throughput, bottleneck identification)
• Black box testing
– Running system with projected use-case
scenarios for acceptable throughput and response
times
• How will users experience application?
Performance Testing Overview
• White box testing
– Pushing system to identify application, database,
operating system, or network problems
• Tune environment to identify and address specific
problems
• Tests watched by developers, DBA, system and network
administrators, and performance engineers
• Resource intensive nature of process
– Infrastructure, personnel, tools, time
Community Performance Testing
• The Approach
• Common Environment/Data Creation
• Load Test Tool
• Proof of Concept
• The Practice
Organization vs. Community
Testing
• Organization approach
– Advantages
• Focus on exact infrastructure used in production
• Define use cases according to real use in production
• Only test tools specific to organization
– Limitations
• Very resource intensive (expensive)
• Hard to maintain
• Hard to be agile without careful advance planning
5
Organization vs. Community
Testing
• Community approach
– Advantages
• Pool of test scripts available for immediate use
• May not need full testing infrastructure to be confident in
production
• Increased confidence in adding new tools
• Total cost of testing shared
– Limitations
• Must clearly communicate results to community
• Collaboration seems harder than just doing it yourself
6
WG Project Objectives
• Create QA environment that enables
performance testing by community in shared
manner
– Plan available on Confluence
• http://confluence.sakaiproject.org/confluence/display/PE
RF/Home
• Have working proof of concept by Paris Sakai
Conference
7
Infrastructure Issues
• What do we learn from testing done on
infrastructure that doesn’t match our own?
– Advantages
• Software/application focus forces code improvements
• More efficient code more likely to scale to meet size and
performance needs
– Disadvantages
• Hardware and network bottlenecks NOT found
• Capacity limits of environment NOT discovered
Community Performance Testing
• The Approach
• Common Environment/Data Creation
• Load Test Tool
• Proof of Concept
• The Practice
Provisioning
• Alan's provisioning tool
– Perl scripts use web services to populate DB with
courses
– Configuration via property files
– Adds tools, instructors and students to courses
– Creates file structure and uploads resources to sites
• Other efforts are underway
Provisioning, Benefits
• Easy, fast creation of test environment
• Common data environment
– Common environment by sharing properties files
– Common data files for tests
Provisioning, Some Modifications
ORIGINAL
MODIFIED
1. provision.pl
1. provision.pl
2. make_media_list.pl
2. make_media_list.pl
3. make_local_resources.
pl
3. create_templates.pl
4. load_resources.pl
Data Environment Design
• Based on UM Winter 2008 semester
– Number of courses (2000) and students (20,000)
– Tool mix (+projection)
• Courses broken into 5x3 categories:
– 5 categories of roster size
– 3 categories of tool distribution
• Site name based upon category
Community Performance Testing
• The Approach
• Common Environment/Data Creation
• Load Test Tool
• Proof of Concept
• The Practice
Evaluation & Tool Selection
• Goal
– Enable near-enterprise level of tool quality using
Open Source tools
– Hewlett Packard LoadRunner used for comparison
15
Evaluated Test Software
Grinder
WebLoad Open Source
JMeter
TestMaker
CLIF
OpenSTA
Web Application Load
Simulator
Hammerhead
SlamD (Sun)
Funkload
Selenium
Pylot
httperf
Seagull
Deluge
Grinder: Plus
• Easy record mechanism
– Can record https on multiple platforms
• Like the scripting language (jython)
– Java extensible
– Allows for re-usable libraries
– Flexible reporting, data handling
• Run-time UI displays multiple test scripts
– We use 22 for our standard scenario!
Grinder: Plus
• Distributed load generation
– Multi-platform
• Active user/development community
– Open source
– Separate project for developing reports: “Grinder
Analyzer”
• Jython Eclipse plug-in (Grinderstone)
Grinder: Minus
• Default recorded script is complex
– Verbose results data
– Perl script cleans the default record script
• (J)Python not ubiquitous
– To do list shows support for java scripts
• Reporting project doesn't consolidate data easily
Community Performance Testing
• The Approach
• Common Environment/Data Creation
• Load Test Tool
• Proof of Concept
• The Practice
Proof of Concept
• Read-only scripts used
– Download file
– Viewing a grade
– “Idle” user
• Mix/Frequency of users corresponds to a
peak usage period
– Events table used to determine usage
Event Graphs
LoadRunner Script
Grinder Script
LoadRunner Results
Grinder Results
Database Results
Grinder: Some Ideas
• Consolidated & Expanded reporting
• Business Process Testing
– Makes test script development more flexible
– Potentially expand the pool of testers
Community Performance Testing
• The Approach
• Common Environment/Data Creation
• Load Test Tool
• Proof of Concept
• The Practice
Establishing Baseline(s)
• Identify “core” scripts for Sakai baseline
• Identify related script clusters to baseline tool
clusters
• Maximum time threshold for page loads
• Minimum user load for tools and kernel
• Database activity reports
Establishing Baseline(s)
• Identify “core” scripts for Sakai baseline
– All tools included in release?
– More limited subset?
• Identify related script clusters to baseline tool
clusters
– Tools with dependencies?
– Tools with similar actions?
– Usage scenarios associated with specific
campuses?
32
Community Sharing
• Scripts
–
–
–
–
Set up performance test branch in Sakai svn?
Associate test scripts with code in svn?
Expand Confluence Performance WG site?
Establish communication mechanisms during QA
release cycles?
33
Community Sharing
• Scripts
– Sakai performance test branch in svn?
– Packaged along with tools in svn?
• Results
– Expand Confluence site?
– Contribute to QA “score card”
• Documentation
– On confluence and/or in svn?
Contributing to Sakai QA
• Recommend performance “standards” to be
part of formal QA release process?
– Score cards
• Volunteer organization to test specific tools
based on usage needs
– Use shared scripts and contribute scripts/results
back to community
• Combined test results yield greater
confidence in Sakai code
Moving from Concept to
Community
• Performance Working Group BOF
– Wednesday, 11-12, BOF Room 1
• Check BOF schedule for exact location
• Looking to identify contributors
• Goal to establish new working plan for WG
contribution to next conference
Michigan’s Contribution
•
•
•
•
Leadership & initial design
All test scripts used at Michigan
All results from Michigan tests
All documentation needed to conduct testing
as currently implemented at Michigan
• All relevant materials for future tests
• Kernel testing for Next Generation Sakai