Transcript Workshop Template - University of Wisconsin–Platteville

Software Engineering Institute (SEI)
High Maturity Workshop:
Reducing Defects Through Improved
Requirements-Based Testing Models
Craig Hale, Process Improvement Manager and Presenter
Mike Rowe, Staff Engineer
Tom Bragg, VP of Operations
Mara Bechwar, B&M Lead
Esterline Control Systems - AVISTA
Software Requirements-Based Testing Defect Model
• Focus: requirements-based test (RBT) reviews
– Quality imperative, but cost impacts
– Large amount of historical data
• Model: defects per review based on number of requirements
– Suspected review size a factor
– Used for every review
– Looked at controllable factors to improve reviews effectiveness
• Stakeholders:
– Customers
– Project leads and engineers
– Baselines and models team
Model Goals
• Improve overall quality of safety-critical systems
• Focus on improving review process
– Maximize defect detection rate
• Minimize defect escapes
– Reduce defect injection rate
• Reduce cost of poor quality
• Defect process performance baselines split
– Embedded vs. non
– Complexity level
– Application type – avionics, medical, etc.
Predicted Outcomes
• Expected defects in the review per
number of requirements
• Important to understand if exceeding
expected defects
• Valuable to understand if all defects were
detected
Factors
• Metrics: 738 reviews over three years
• Y Variables
 Number of defects per review (D/R) discrete: ratio data type
 Defects per requirement (D/Rq) continuous: ratio data type
Modeling Techniques
• Non-linear regression
• Power function correlation
• Standard of error estimate varied considerably
– Partitioned into nine intervals
– Monte Carlo simulation
• Standard of error estimate did not change by more than 0.000001
for ten iterations
• Determined standard of error estimate for each partition
• Limitation: variations in customers and projects impacts
accuracy
Factors and Correlation Tables
D = Defects
PT = Preparation Time
R = Review
Rq = Requirement
Data Collection: Requirements Count
Data Collection: Partitioning of Reviews
Output from Model
4 Requirements
20 Requirements
Pilot Results
• Determined to automate model
• Needed statistical formula for variance
• More guidance on what to do when out of range
Project
Organization
Mean
Standard
Deviation
Mean
Standard
Deviation
Review Size
-7.17%
+209.9%
-46.24%
-67.62%
Defects Per
-13.55%
-16.71%
-36.68%
-41.83%
Results, Benefits and Challenges
• Points to decreasing variation in defects
• Provides early indicator to fix processes and reduce defect
injection rate
• Indicates benefits for small reviews and grouping
• Challenged with gaining buy-in, training and keeping it simple
Organization
Mean
Standard
Deviation
Review Size
-34.85%
-53.44%
Defects Per
-24.70%
-31.36%
Summary
• What worked well
– Utilizing historical data to predict outcomes
– Encouragement of smaller data item reviews
– Reducing defect injection
• Future plans: Enhance the model
–
–
–
–
–
Requirement complexity
Expand Lifecycles
Expand Activities
Safety criticality
Staff experience