Transcript ProactiveTestPlanning - Seattle Area Software Quality

How Early, Proactive Test
Planning Contributes to
Project Success
Based on a paper to be presented at the
International Information Management
Association in Dublin, Ireland - September, 2005
August 18, 2005
Jim Nindel-Edwards
Presentation Outline
 Introduction
 Brief primer on requirements
 Where does the test plan fit in?
 Addressing requirements issues with the test
plan
 Conclusion
 Q & A, Comments, Discussion
August 18, 2005
Jim Nindel-Edwards
Survey of the Requirements Process
 The importance of requirements
 Just cannot succeed without them
 Requirements can be missed
 And too often are missed until late the process
 Missed functional requirements vs. other types of
requirements
 Development methodologies can help, or hinder the
requirements process
 How can the test plan help?
August 18, 2005
Jim Nindel-Edwards
Requirements – Definitions
Dorfman and Thayer (1990)
 A software capability needed by the user to
solve a problem to achieve an objective
 A software capability that must be met or
possessed by a system or system component
to satisfy a contract, standard, specification,
or other formally imposed documentation
August 18, 2005
Jim Nindel-Edwards
Unknown Requirements
Watts Humphrey (1990)
“Each error, when found, is a surprise whose
correction is both expensive and disruptive.”
 Unknown
“The users think they know what is wanted, but during
initial use they discover that their real needs are not
what they had thought.”
 Misunderstood Requirements
“Even when the requirements are known and stable, the
implementers often do not understand them in the
detail required to produce a satisfactory product.”
August 18, 2005
Jim Nindel-Edwards
What types of requirements are there
anyway?
 Functional – gets most of the attention
 Application Architecture
 Environmental
 Standards
 Functional needs - secondary user
groups
 Security
August 18, 2005
Jim Nindel-Edwards
Where does the Test Plan fit?
Functional requirements:
 Test what is supposed to work (of course)
 Test what is not supposed to work (Negative
testing)
 Testing in priority sequence (Pri1, Pri2, etc.)
 Regression Testing
 Testing conflicting requirements (Rich feature set,
ease of use)
August 18, 2005
Jim Nindel-Edwards
Application Architecture
Target hardware and OS environments
 Performance Test
 Stress Test
 Limits Testing
 Target installation environment vs. actual
implementation environment
 How does the application respond to
installation in unanticipated environments?
August 18, 2005
Jim Nindel-Edwards
Environmental Requirements
 Glenford Myers’s Project Objective List






Efficiency
Compatibility
Security (more later)
Reliability
Maintainability
Upgradeability
 Shared environments
August 18, 2005
Jim Nindel-Edwards
Standards Requirements
 Standards – Abstractions of hardware and
software




Target Operating System(s)
Target Hardware
Supported vs. non-supported environments
Industry standards (XML, SOAP, UDDI, etc.)
 Validation that standards are met
 Edge, boundary, and “corner” tests
August 18, 2005
Jim Nindel-Edwards
Secondary User Groups
Secondary users are users also!
 Installers, super users, DBAs
 Testing the documentation for all types of
users groups
 Testing the installation, upgrade, rollback, and
trouble shooting processes
 Conversion and upgrades from other
software packages and/or old versions
 Reliability, efficiency, storage requirements
August 18, 2005
Jim Nindel-Edwards
Product and System Security
 Security requirements – Application or
Environment?

Security holes typically lie in-between these
two areas
 Making certain the security requirements are
clearly documented, and testable
 Continuous testing of security features, and
searching for security holes
 Tests must include all user groups
August 18, 2005
Jim Nindel-Edwards
Back to the test plan
 Verify functionality, find defects
 Use a checklist (handout)
 Build the test plan early in the project, and
keep it up to date

You will affect the requirements document in this
process!
 Test all aspects of the product at every
milestone

Early test failure(s) need not indicate a poor product,
just an “immature” product
August 18, 2005
Jim Nindel-Edwards
Conclusion
 Not having a test plan is not a good idea!
 Having a test plan, even late in the project, is better
than not having one at all

At least you can document what was tested, and not
tested, in the development process
 Early test planning leads to more comprehensive
tests earlier in the development process
 Test plans at the very start of a project can uncover
missing and misunderstood requirements before they
add delays to the project
August 18, 2005
Jim Nindel-Edwards
Q & A, Comments, Suggestions
 Question: What is a “good” bug?

And what might you change in your project to catch this one
early?
August 18, 2005
Jim Nindel-Edwards
Thanks!
 Jim Nindel-Edwards, SQA Lead, Costco
Wholesale, Costco.com
 Email: [email protected]
 My thanks to SASQAG

And my Graduate Advisor, Dr. Gerhard Steinke,
Seattle Pacific University
August 18, 2005
Jim Nindel-Edwards