Transcript 9-2-Outsourcing

Outsourcing, subcontracting and
use of COTS
Torbjørn Skramstad
Contents
We will cover the following topics
• Testing as a confidence building activity
• Testing and outsourcing
• Testing COTS components
• Sequential testing
• Simple Bayesian methods
Responsibility
It is important to bear in mind that
• The company that brings the product to the
marketplace carries full responsibility for the
product’s quality.
• It is only possible to seek redress from the
company we outsourced to if we can show
that they did not fulfill their contract
Testing and confidence
The role of testing during:
• Development – find and remove defects.
• Acceptance – build confidence in the component
• When we use testing for COTS or components
where the development has been outsourced or
developed by a subcontractor, we want to build
confidence.
A product trustworthiness pattern
System
definition
Product is
trustworthy
Trustworthiness
definition
Environment
definition
Product
related
Process
related
People
related
Means to create product trust
Based on the product trust pattern, we see that
we build trust based on
• The product itself – e.g. a COTS component
• The process – how it was developed and
tested
• People – the personnel that developed and
tested the component
A process trustworthiness pattern
Activity is
trustworthy
Argument by
considering
process
Team is
competent
Method
address
problem
Trustworthiness
definition
Process
definition
Process is
traceable
Means to create process trust
• If we apply the pattern on the previous slide
we see that trust in the process stems from
three sources:
– Who does it – “Team is competent”
– How is it done – “Method addresses problem”
– We can check that the process is used correctly –
“Process is traceable”
Testing and outsourcing
• If we outsource development, testing need to
be an integrated part of the development
process. Testing is thus a contract question.
• If we apply the trustworthiness pattern, we
need to include requirements for
– The component - what
– The competence of the personnel – who
– The process – how
Outsourcing requirements - 1
When drawing up an outsourcing contract we
should include:
• Personnel requirements – the right persons
for the job. We need to assess the CV for each
person.
• Development process – including testing. The
trust can come from
– A certificate – e.g. ISO 9001 or a CMMI
assessment (3rd party assessment)
– Our own process audits (2nd party assessment)
Outsourcing requirements - 2
Last but not least, we need to see and inspect
some important artifacts:
• Project plan – when shall they do what?
– Activities, Milestones, Toll gates
• Test strategy – how will they test our
component requirements?
• Test plan – how and when will the tests be
run?
• Test log – what were the results of the tests?
Trust in the component
• The trust we have in the component will
depend on how satisfied we are with the
answers to the questions on the previous
slide.
• We can, however, also build our trust on
previous experience with the company.
– The more we trust the company based on earlier
experiences, the less rigor we will need in the
contract.
Testing COTS
• We can test COTS by using e.g. black box
testing or domain partition testing.
• Experience has shown that we will get the
greatest benefit from our effort by focusing on
tests for
– Internal robustness
– External robustness
Robustness – 1
• There are several ways to categorize these two
robustness modes. We will use the following
definitions:
– Internal robustness – the ability to handle faults in
the component or its environment. Here we will
need wrappers, fault injection etc.
– External robustness – the ability to handle faulty
input. Here we will only need the component “as
is”
Robustness – 2
• The importance of the two types of
robustness will vary over component types.
– Internal robustness - components that are only
visible inside the system border
– External robustness – components that are part of
the user interface.
Internal robustness testing
Internal robustness is the ability to
• Survive all erroneous situations, e.g.
– Memory faults – both code and data
– Failing function calls, including calls to OS
functions
• Go to a defined, safe state after having given
the error message
• Continued after the erroneous situation with a
minimum loss of information.
Why do we need a wrapper
By using a wrapper, we obtain some important
effects:
• We control the component’s input, even
though the component is inserted into the
real system.
• We can collect and report input and output
from the component.
• We can manipulate the exception handling
and effect this component only.
What is a wrapper – 1
A wrapper has two essential characteristics
• An implementation that defines the functionality
that we wish to access. This may, or may not be an
object (one example of a non-object implementation
would be a DLL whose functions we need to access).
• The “wrapper” class that provides an object interface
to access the implementation and methods to
manage the implementation. The client calls a
method on the wrapper which access the
implementation as needed to fulfill the request.
What is a wrapper – 2
A wrapper provides interface for, and services to,
behavior that is defined elsewhere
Fault injection – 1
• On order to test robustness, we need to be
able to modify the component’s code –
usually through fault injection.
• A fault is an abnormal condition or defect
which may lead to a failure.
• Fault injection involves the deliberate
insertion of faults or errors into a computer
system in order to determine its response. The
goal is not to recreate the conditions that
produced the fault
Fault injection – 2
There are two steps to Fault Injection:
• Identify the set of faults that can occur within an
application, module, class, method. E.g. if the
application does not use the network then there’s
no point in injecting network faults
• Exercise those faults to evaluate how the
application responds. Does testing the application
detect the fault, is it isolated and does the
application recover?
• Typically used in robustness testing
Software implemented fault
injection
• Compile-time injection: Source code is modified to
inject simulated faults into the system
– Changes existing lines of code
– Adds code. Use of perturbation functions. Simple functions
which take an existing value and perturb it
• Run-time injection
– Corruption of memory
– Intercepting operating system calls made by user-level
software and injecting faults into them
– Network level fault injection. Corruption, loss or
reordering of network packets
Change / add code example
Change code:
a = a + 1 to a = a – 1
Add code:
int pFunc(int value) {
return value + 20;
}
int main(int argc, char * argv[]) {
int a = pFunc(aFunction(atoi(argv[1])));
if (a > 20) { /* do something */
} else { /* do something else */
}
}
External robustness testing – 1
Error handling must be tested to show that
• wrong input gives an error message
• the error message is understandable for the
intended users
• application continues after the error with a
minimum loss of information.
External robustness testing – 2
External robustness is the ability to
• Survive the input of faulty data – no crash
• Give an easy-to-understand error message
that helps the user to correct the error in the
input
• Go to a defined state
• Continue after the erroneous situation with a
minimum loss of information.
Easy-to-understand message – 1
• While all the other characteristics of the
external robustness are easy to test, the error
message requirement can only be tested by
involving the users.
• We need to know which info the user needs in
order to:
– Correct the faulty input
– Carry on with his work from the component’s
current state
Easy-to-understand message – 2
• The simple way to test the error messages is
to have a user to
– start working on a real task
– insert an error in the input at some point during
this task
• We can then observe how the user tries to get
out of the situation and how satisfied he is
with the assistance he get from the
component (the error message).
Sequential testing*
In order to use sequential testing we need:
• Target failure rate p1
• Unacceptable failure rate p2 and p2 > p1
• The acceptable probability of doing a type I (false
positive) or type II decision error (false negative)
– a and b. These two values are used to
compute a and b, given as
1 b
b
b  ln
a  ln
a
1a
*Sequential testing is statistical testing where the sample size is not fixed in advance
Type I and Type II error
• A type I error occurs when the null hypothesis (H0) is true, but is rejected
• A type II error occurs when the null hypothesis is false, but erroneously
fails to be rejected
Background - 1
We will assume that the probability of failure is Binomially
distributed *. We have:
n x
n x


f ( x, p, n)    p (1  p)
 x
The probability of observing the number-of-defects sequence x1,
N
N
x2,…xn can be written as
xi
Nn  xi
f ( x1 , x2 ,...xN , p, n)  Cp

i 1
(1  p)

i 1
*The binomial distribution is the discrete probability distribution of the number
of successes in a sequence of n independent yes/no experiments
Background - 2
We will base our test on the log likelihood ratio,
which is defined as:
 f ( xi , p1, n)
N
p1 
 1  p1
ln   ln
  xi ln   Nn   xi  ln
 f ( xi , p2 , n) i1 p2  i1  1  p2
N
For the sake of simplicity, we introduce
p1
1  p1
u ln
, v  ln
p2
1  p2
The test statistics
Using the notation from the previous slide, we
find that
n
b  ln   a  b  Nnv  (u  v) xi  a  Nnv
i 1
N
b  Mv
a  Mv
  xi 
u v
u v
i 1
We have p1, p2 << 1 and can thus use the
approximations ln(1-p) = -p, v = (p2 – p1) and
further that (u – v) = u
Sequential test – example
• We will use a = 0.05 and b = 0.20. This will
give us a = -1.6 and b = 2.8.
• We want a failure rate p1 = 10-3 and will not
accept a component with a failure rate p2
higher than 2*10-3. Thus we have u = - 0.7 and
v = 10-3.
• The lines for the “no decision” area are
– Sxi(reject) = - 4.0 + M*10-3
– Sxi(accept) = 2.3 + M*10-3
Sequential test – example
Sx
2.3
M
4*103
-4.0
Accept
Sequential testing - summary
• In statistics, sequential analysis is a statistical analysis
where the sample size is not fixed in advance. Data are
evaluated as they are collected. Further sampling is
stopped in accordance with a pre-defined stopping rule
as soon as significant results are observed.
• Testing software – e.g. p < 10-3:
The method needs a large number of tests. It should
thus only be used for testing robustness based on
automatically generated random input.
• Inspecting documents – e.g. p < 10-1:
The method will give useful results even when
inspecting a reasonable number of documents
Simple Bayesian methods
• Instead of building our trust on only test
results, contractual obligations or past
experience, we can combine these three
factors.
• One way to do this is to use Bayesian statistics.
• We will give a short intro to Bayesian statistics
and show one example of how it can be
applied to software testing
Bayes theorem
In a simplified version., Bayes’ theorem says that
P ( B | A)  P( A | B) P ( B )
When we want to estimate B, we will use the likelihood
of our observations as our P(B|A) and use P(B) to
model our prior knowledge.
A Bayes model for reliability
For reliability it is common to use a Beta distribution for
the reliability and a Binomial distribution for the
number of observed failures. This gives us the
following results:
P( X | obs)  p (1  p)
x
P( X | obs)  p
n x
x a 1
a 1
p (1  p)
(1  p)
n  x  b 1
b 1
Estimates
A priori we have that
^
R
a
a b
If x is the number of successes and n is the total
number of tests, we have posteriori, that
ax
R
a b n
^
Some Beta probabilities
Testing for reliability
We use a Beta distribution to model our prior
knowledge. The knowledge is related to the
company that developed the component or
system, e.g.
• How competent are the developers
• How good is their process, e.g.
– Are they ISO 9001 certified or CMMI assessed
– Have we done a quality audit
• What is our previous experience with this
company
– Are there any changes?
Modeling our confidence
Several handbooks on Bayesian analysis contain
tables where we specify two out of three
values:
• R1: our mean expected reliability
• R2: our upper 5% limit. P(R > R2) = 0.05
• R3: our lower 5% limit. P(R < R3) = 0.05
When we know our R-values, we can read the
two parameters n0 and x0 out of a table.
The result
We can now find the two parameters for the
prior Beta distribution as:
• a = x0
• b = n0 – x0
if we run N tests and observe x successes then
the Bayesian estimate for the reliability is:
R = (x + x0) / (N + n0)
Sequential test with Bayes
• We can combine the info supplied by the
Bayesian model with a standard sequential
test chart by starting at (n0 - x0, n0) instead of
starting at origo as shown in the example on
the next slide.
• We have the same number of tests necessary,
but n0 of them are virtual and stems from our
confidence in the company.
Sequential test with Bayes – example
Sx
2.3
n0
-4.0
M
4*103
Accept