Transcript lec1110

Software Quality:
Testing and Verification II
Software Flaws are identified at three levels
1. A failure is an unacceptable behaviour exhibited by a system
— The frequency of failures measures software reliability
Low failure rate = high reliability
— Failures result from violation of a requirement
2. A defect is a flaw that contributes to a failure
— It might take several defects to cause one failure
3. An error is a software decision that leads to a defect
2
Eliminating Failures: Testing vs Verification
Testing = running the program with a set of inputs to gain
confidence that the software has few defects
Goal: reduce the frequency of failures
When done: after the programming is complete
Methodology: develop test cases; run the program with
each test case
Verification = formally proving that the software has no defects
(in this case, the program is said to be “correct”)
Goal: eliminate failures
When done: before, during and after the programming is complete
Methodology: write separate specifications for the
code; prove that the code and the specifications
are mathematically equivalent
3
Program Correctness
The correctness of a program is based on a specific standard.
That standard is called a specification.
int max (int a, int b) {
int m;
if (a >= b)
m = a;
else
m = b;
return m;
}
E.g., a specification for the above program might be that it “finds the
maximum value of any two integers.”
4
Formalizing a Specification
A Formal specification is written as a logical expression called an assertion.
An assertion describes the state of the program’s variables.
Two key assertions are the program’s precondition and its postcondition.
int max (int a, int b) {
int m;
if (a >= b)
m = a;
else
m = b;
return m;
}
Precondition P = what’s required
for the program to begin its run.
Postcondition Q = what’s ensured to
be true when the program finishes
A domain is a set of values over which a variable is well defined.
The primitive types (int, float, boolean, etc.) and standard Java classes (String,
Vector, HashMap, etc.) provide domains for reasoning about programs.
5
Pre- and Postconditions
Postconditions describes what it will compute.
For max, a postcondition is Q = m = max(a, b), where max is a
mathematical function that delivers the larger of two integers.
Preconditions describe minimum requirements for the program to run.
For max, a and b can be any integers, so the precondition is P = true.
Before proving a program’s correctness, we first write its specifications:
{true}
int max (int a, int b) {
int m;
if (a >= b)
m = a;
else
m = b;
return m;
}
{m = max(a, b)}
6
Another Example – computing a factorial (n!)
{n > 1}
int Factorial (int n) {
int f = 1;
int i = 1;
while (i < n) {
i = i + 1;
f = f * i;
}
return f;
}
{f = n!}
Precondition P
Postcondition Q
This example raises two issues. What happens if:
1. the program has a loop that never terminates?
2. The program terminates abnormally (e.g., an exception is raised)?
7
Defining away the problem: “partial correctness”
Let’s assume that nothing bad ever happens…
• programs always terminate after a finite number of steps, and
• termination is always normal.
Then such a program is partially correct if, for every set of input
values that satisfies precondition P, the program computes a result
that satisfies postcondition Q.
E.g., Factorial is partially correct if for every value of n that
satisfies n > 1, it computes f =n!
8
(Partial) Correctness Proofs
Let’s think generally about any program or sequence of statements s,
whose pre- and postconditions are P and Q.
A “Hoare triple” is a predicate of the form
{P} s {Q}
which asserts that “execution of statements s, beginning in a state
that satisfies P, results in a state that satisfies Q.”
If we can prove that this Hoare triple is valid, (i.e., it is true for all
assignments of values to variables in P, Q, and s) then the
program s is said to be correct.
But how can we construct such a proof?
9
Constructing a Correctness Proof
If the program is a series of statements:
s1; s2; …; sn
We start with the Hoare triple {P} s1; s2; …; sn {Q}, and
• use inference rules for programs to
• derive an intermediate triple {Pi} si {Pi+1} for every
statement si in this sequence.
When done, we also ensure that P  P1 and Pn1  Q.
Note: This process is similar to a direct proof in logic, where we
• use inference rules to

• derive a series of assertions that logically link the premises
to the conclusion.
So, what are the inference rules for programs?
10
Inference Rules are tied to program statement types

11
Formal methods and real programs
Major question: Where do formal methods fit in the software
process? That is,
1. How do we integrate them into object oriented programs?
2. How do we write pre- and postconditions P and Q for
– methods?
– classes?
– systems?
3. Once written, how are these used to ensure that software
is correct?
12
Recall the spiral model for software design
Review
Deployment (v 1.0)
v 1.1
v 1.0
Testing and
Verification
Prototype
Coding/
Integration
Requirements
Specifications
(use cases)
Design
13
Formal methods in the object oriented design process
In OO design, we focus on classes and objects
Methods and messages are subordinate
Collections of objects have state, which is the set of all active
objects and the values of their variables at any moment of
run time.
Formal specifications P and Q are therefore logical
expressions about each object’s state.
Tools for the formal design process
• Specifications : Java Modeling Language (JML)
• Design: Unified Modeling Language (UML and JML)
• Coding: Java and JML
• Verification: JML
14
Correctness of OO systems
Where?
• Method level: pre- and post-conditions, loop invariants
• Class level: class invariant (class state)
• System level: intra-class invariants (system state)
When?
• Specification and design phases:
Write specifications for all classes and methods (UML/JML)
• Coding phase:
Develop code from the specifications (UML/JML/Java)
• Validation phase:
Prove (mathematically) that specifications and code are
logically equivalent (JML <==> Java)
15
What is JML? (www.jmlspecs.org)
History
– Emerged in early 2000s out of ESC/Java2
Goals
— Integration of formal methods throughout the software process
— Formal specification accessible to programmers
— Direct support for design by contract
— Integration with a real language (Java)
JML allows us to mix specifications directly with the Java code
— Preconditions
— Postconditions
— Loop invariants
— Class invariants
16
JML Basics
JML specifications are special comments in a Java program:
//@
for one-liners
/*@
….
@*/
for multiple-liners
The Hoare triple
{P} s1; s2; …; sn {Q}
is written in JML/Java as
(P and Q are written as Java
boolean expressions, and use
parameters, local, and class
variables as arguments.)
/*@ requires P ;
ensures Q ;
@*/
type method (parameters)
{
local variables
s1; s2; …; sn
}
17
JML Language Summary
Note: p and e are also good old fashioned Java boolean expressions,
possibly augmented by JML-specific operations.
18
Here’s a simple example, first as a Hoare triple:
{n > 1}
P (precondition)
int Factorial (int n) {
int f = 1;
int i = 1;
{1 < i  i < n  f = i!}
R (loop invariant)
while (i < n) {
i = i + 1;
f = f * i;
}
return f;
}
{f = n!}
Q (postcondition)
19
And again with JML expressions for P, Q, and R:
/*@ requires 1 <= n ;
P
Q
R
ensures \result == (\product int i; 1<=i && i<=n; i) ;
@*/
static int Factorial (int n) {
int f = 1;
int i = 1;
/*@ loop_invariant i <= n && f == (\product int j; 1 <= j && j <= i; j);
@*/
while (i < n) {
i = i + 1;
f = f * i;
}
return f;
}
20
JML-based software tools
(1 and 3 are built into Eclipse)
1. Compiling (use jmlc instead of javac)
• Does syntactic and type checking, and byte code generation for all
JML assertions and Java code
2. Static checking (ESC/Java2)
3. Runtime assertion checking (use jmlrac instead of java)
• Checks truth of precondition P at entry to every call
• Checks truth of postcondition Q at exit from every call
• Checks truth of loop invariant R before every iteration
• Issues a Java Exception when any of these is not true
Note: this is not formal verification
(Checking truth for one instance of a call is not the same as checking truth
for all instances. The latter is proof of correctness!)
4. Proof assistance tools (Daikon, LOOP)
21
JML Eclipse Environment
1. Compiling
3. Runtime assertion checking
22
Continuing our JML example, let’s wrap Factorial
inside a simple class:
public class myFactorial {
/*@ requires 1 <= n;
ensures \result == (\product int i; 1<=i && i<=n; i);
@*/
static int Factorial (int n) {
…
}
public static void main(String[] args) {
int n = Integer.parseInt(args[0]);
System.out.println("Factorial of " + n + " = " +
Factorial(n));
}
}
23
… compile it with jmlc, and run it twice with jmlrac:
% jmlc -Q myFactorial.java
normal run
% jmlrac myFactorial 3
abnormal run
Factorial of 3 = 6
(throws a JML exception)
% jmlrac myFactorial -5
Exception in thread "main”
org.jmlspecs.jmlrac.runtime.JMLEntryPreconditionError:
by method myFactorial.Factorial regarding specifications at
File "myFactorial.java", line 3, character 15 when
'n' is -5
at myFactorial.checkPre$Factorial$myFactorial(myFactorial.java:240)
at myFactorial.Factorial(myFactorial.java:382)
at myFactorial.main(myFactorial.java:24)
24
JML Exceptions
25
JML helps identify errors
Example 1: Suppose we change the while loop from
while (i < n)
to while (i <= n)
so that n! will be computed incorrectly.
Here’s the result:
invariant not satisfied
% jmlrac myFactorial 3
Exception in thread "main"
org.jmlspecs.jmlrac.runtime.JMLLoopInvariantError: LOOP INVARIANT:
by method myFactorial.Factorial regarding specifications at
File "myFactorial.java", line 9, character 24 when
'n' is 3
at myFactorial.internal$Factorial(myFactorial.java:102)
at myFactorial.Factorial(myFactorial.java:575)
at myFactorial.main(myFactorial.java:211)
26
JML Example 2
Suppose we change the while loop from
while (i < n) to
while (i <=n )
and also remove the JML loop invariant. Now we get:
postcondition not satisfied
% jmlrac myFactorial 3
Exception in thread "main"
org.jmlspecs.jmlrac.runtime.JMLNormalPostconditionError:
by method myFactorial.Factorial regarding specifications at
File "myFactorial.java", line 4, character 23 when
'n' is 3
'\result' is 24
at myFactorial.checkPost$Factorial$myFactorial(myFactorial.java:321)
at myFactorial.Factorial(myFactorial.java:392)
at myFactorial.main(myFactorial.java:24)
27
JML Example 3
Disagreement between a JML specification and a program may
signal an error in the specification.
E.g., if the loop invariant had specified j <= i rather than j < i the
following outcome would occur:
invariant not satisfied
% jmlrac myFactorial 3
Exception in thread "main"
org.jmlspecs.jmlrac.runtime.JMLLoopInvariantError: LOOP INVARIANT:
by method myFactorial.Factorial regarding specifications at
File "myFactorial.java", line 9, character 24 when
'n' is 3
at myFactorial.internal$Factorial(myFactorial.java:101)
at myFactorial.Factorial(myFactorial.java:573)
at myFactorial.main(myFactorial.java:209)
28
But beware… JML is no panacea
jmlrac doesn’t trap all errors… here are two “normal” runs:
% jmlrac myFactorial 21
Factorial of 21 = -1195114496
% jmlrac myFactorial 32
Factorial of 32 = -2147483648
wrong results!
Recall: (1) Java has no ArithmeticOverflow exception, but
(2) Factorial(n) for n > 12 should give a result > 231-1
Note: jmlrac computes the same wrong result when it checks the
postcondition as the Factorial method computes, so that this
error goes undetected.
Conclusion: the program and its specifications are both wrong.
29
Exception Handling in JML
We can throw Java Exceptions, and then validate their
circumstances in JML whenever they occur.
/*@ requires P ;
ensures Q ;
signals (exception) expression;
@*/
type method (parameters) {
locals
s1; s2; …; sn
(includes “throw new exception ;”)
}
JML executes this
whenever this happens
Two outcomes: 1) expression is true and normal Java
exception handling proceeds, or 2) expression is false and
JMLExceptionalPostconditionError is reported.
30
JML Example 4: Throwing and Checking Exceptions
/*@ requires 1 <= n;
ensures \result == (\product int i; 1<=i && i<=n; i);
signals (ArithmeticException) n > 12;
@*/
static int Factorial (int n) {
if (n > 12) throw new ArithmeticException();
else { …
Normal Java exception
handling occurs, since
% jmlrac myFactorial 22
n > 12 is true.
Exception in thread "main" java.lang.ArithmeticException
at myFactorial.internal$Factorial(myFactorial.java:9)
at myFactorial.Factorial(myFactorial.java:610)
at myFactorial.main(myFactorial.java:213)
31
Additional Points about JML
1. We can sometimes avoid a signals clause by strengthening the
2.
3.
4.
5.
precondition. E.g., for Factorial, we could have said:
requires 1 <= n && n < 13 ;
Specifications are always declarative; they never affect the state of
the program.
Runtime assertion checking is not proof, but it does provide a
rigorous framework for debugging.
JML provides a language in which formal methods and Java
programs can be integrated.
There’s a lot more to JML. We are particularly interested in:
a. class level specifications?
b. Tools for static checking of specifications?
c. Tools for proving correctness?
32
Readings (at the course web site)
Hoare, An axiomatic basis for computer programming,
Communications of the ACM 12(10):576-580.
Leino, Hoare-Style Program Verification I
Leino, Hoare-Style Program Verification II
Gordon, Specification and Verification I
Dwyer et al., Software Specifications
Leavens and Cheon, Design by Contract with JML
Poll, Kiniry, and Cok, Introduction to JML
Burdy et al., Overview of JML Tools
JML Reference Manual
33