Transcript T&E Part 5

EMIS 7307
T&E Part 5
Test Planning and Analysis
Landing distance of an F-18
1
EMIS 7307
T&E Part 5
Test Planning and Analysis
2
EMIS 7307
T&E Part 5
Test Planning and Analysis
3
EMIS 7307
T&E Part 5
Test Planning and Analysis
• What if 60% isn’t sufficient confidence?
– Typically 95% is the desired confidence.
• Need more data points!
– How many?
– Can’t say for sure.
– Depends on s and x with the new data.
4
EMIS 7307
T&E Part 5
Test Planning and Analysis
Let’s assume that if we collect more data the standard deviation and
(x -V*) don’t change. This not a very good assumption, but possibly
the best we can do. Then we can rewrite ts as ts= k *√(n-1).
With this assumption k = .351.
For:
n =2,
ts= .351,
α=.39, (1-α) = .61
n =5,
ts= .702,
α=.27, (1-α) = .73
n =10, ts= 1.053, α=.16
(1-α) = .84
n = 17, ts = 1.404, α=.09
(1-α) = .91
n = 26, ts = 1.755, α=.04
(1-α) = .96
n = 37, ts = 2.106, α=.015 (1-α) = .985
5
EMIS 7307
T&E Part 5
Test Planning and Analysis
•From the previous chart we get a sense of
the “cost” of confidence.
•For an increase of only three tests (from 2 to 5) we
can boost our confidence by 12%.
•But the change from 17 to 26 only further increased
our confidence by 5%!
•This illustrates why 90% confidence is often chosen
as good enough. It’s a compromise of increased
confidence versus cost.
6
EMIS 7307
T&E Part 5
Test Planning and Analysis
ANOVA
R.
Walpole and R. Myers, [1993], Probability and Statistics for Engineers and
Scientists, Prentice-Hall, Englewood Cliffs, New Jersey.
7
EMIS 7307
T&E Part 5
Test Planning and Analysis
R.
Walpole and R. Myers, [1993], Probability and Statistics for Engineers and
Scientists, Prentice-Hall, Englewood Cliffs, New Jersey.
8
EMIS 7307
T&E Part 5
Test Planning and Analysis
ANOVA
R.
Walpole and R. Myers, [1993], Probability and Statistics for Engineers and
Scientists, Prentice-Hall, Englewood Cliffs, New Jersey.
9
EMIS 7307
T&E Part 5
Test Planning and Analysis
R.
Walpole and R. Myers, [1993], Probability and Statistics for Engineers and
Scientists, Prentice-Hall, Englewood Cliffs, New Jersey.
10
EMIS 7307
T&E Part 5
Test Planning and Analysis
R.
Walpole and R. Myers, [1993], Probability and Statistics for Engineers and
Scientists, Prentice-Hall, Englewood Cliffs, New Jersey.
11
EMIS 7307
T&E Part 5
Test Planning and Analysis
R.
Walpole and R. Myers, [1993], Probability and Statistics for Engineers and
Scientists, Prentice-Hall, Englewood Cliffs, New Jersey.
12
EMIS 7307
T&E Part 5
Test Planning and Analysis
R.
Walpole and R. Myers, [1993], Probability and Statistics for Engineers and
Scientists, Prentice-Hall, Englewood Cliffs, New Jersey.
13
EMIS 7307
T&E Part 5
Test Planning and Analysis
R.
Walpole and R. Myers, [1993], Probability and Statistics for Engineers and
Scientists, Prentice-Hall, Englewood Cliffs, New Jersey.
14
EMIS 7307
T&E Part 5
Test Planning and Analysis
R.
Walpole and R. Myers, [1993], Probability and Statistics for Engineers and
Scientists, Prentice-Hall, Englewood Cliffs, New Jersey.
•In this example the table f value is 2.78. Use 4 and 25
as the degrees of freedom.
•The calculated f is 4.3.
•Therefore, reject the null hypothesis.
15
EMIS 7307
T&E Part 5
Test Planning and Analysis
16
EMIS 7307
T&E Part 5
Test Planning and Analysis
• ANOVA is a powerful data analysis tool
– Using a few samples answers the question, does
an effect on a subgroup of a hypothetical
population cause that subgroup to not really be
from the hypothetical population?
• We hypothesized that the five different aggregates
were from a population that absorbed moisture
similarly.
• The ANOVA showed that they were not!
17
EMIS 7307
T&E Part 5
Test Planning and Analysis
• ANOVA is readily available in many
spreadsheet packages.
– Excel has it under tools then click on data
analysis and select ANOVA.
18
EMIS 7307
T&E Part 5
Test Planning and Analysis
• Bottom line.
– Use student t or normal distribution (n>30)
when comparing to a spec.
– Use ANOVA when looking to see if two groups
are statistically the same to some confidence
level.
19
EMIS 7307
T&E Part 5
Test Planning and Analysis
• Bottom line (cont’d)
– For both test planning and data analysis at least
one member of the team needs to be statistics
savvy.
– From a management perspective.
• This facilitates planning for adequate data collection
to meet the desired confidence.
• Assures data collection ends when sufficient data
exist.
20
EMIS 7307
T&E Part 5
Special Topic
Software testing
• In an earlier lecture I presented the results
of a study showing the economic impact of
poor or limited software testing.
• I’ve also made the case that earlier problem
finding and fixing is exponentially cheaper.
• Here I want to introduce several software
testing concepts.
21
EMIS 7307
T&E
Part 5
Percent of Specification
Requirements
Involving Software
100
F-22
Percent
80
B-2
60
F-15
F-16
Series1
40
F-111
20
F-4
A-7
0
1960 1964 1970 1975 1982 1990 2000
Ref: Lockheed - Martin Corp.
22
EMIS 7307
T&E Part 5
Special Topic
Software testing
•Human testing is defined as informal, noncomputer-based
methods of evaluating architectures, designs and interfaces. It
can consist of:
• Inspections – The programmer explains his/her work to
a small group of peers with discussion and direct feedback
on errors, inconsistencies and omissions.
• Walk-through – A group of peers develop test cases to
evaluate work to date and give direct feedback to the
programmer.
• Desk Checking – A self evaluation is made by the
programmer of his/her own work. There is a low
probability of identifying his/her errors of logic or coding.23
EMIS 7307
T&E Part 5
Special Topic
Software testing
• Peer Ratings – Mutually supportive, anonymous reviews
are performed by groups of peers with collaborative
evaluations and feedback.
• Design Reviews – Preliminary design reviews (PDRs)
and critical design reviews (CDRs) provide milestones in
the development efforts that review development and
evaluations to date.
24
EMIS 7307
T&E Part 5
Special Topic
Software testing
• Computerized software testing
– Black box
• Functional testing of a software unit without knowledge of how the
internal structure or logic will process the input to obtain the
specified output. Within-boundary and out-of-boundary stimulants
test the software’s ability to handle abnormal events. Most likely
cases are tested to provide a reasonable assurance that the software
will demonstrate specified performance. Even the simplest software
designs rapidly exceed capacity to test all alternatives.
25
EMIS 7307
T&E Part 5
Special Topic
Software testing
• Computerized software testing
– White box
• Structural testing of the internal logic and software
structure provides an opportunity for more extensive
identification and testing of critical paths. The process
and objectives are otherwise very similar to black box
testing.
26
EMIS 7307
T&E Part 5
Special Topic
Software testing
• IV&V
– Independent verification and validation are riskreducing techniques that are applied to major software
development efforts. The primary purpose of IV&V is
to ensure that software meets requirements and is
reliable and maintainable. The IV&V is effective only
if implemented early in the software development
schedule. Requirements analysis and risk assessment
are the most critical activities performed by IV&V
organizations.
27
EMIS 7307
T&E Part 5
Special Topic
Software testing
• IV&V
– The desire to boast of a high SEI rating in proposals
and to generally improve the corporations efficiency
has moved many companies to significantly improved
software development capabilities.
– This in turn has reduced the perceived need by
customers to require and pay for IV&V.
28