Transcript Bug Localization with Association Rule Mining

Random Test Generation of Unit Tests:
Randoop Experience
Wujie Zheng
[email protected]
1
Outline




Automated Test Generation of Unit Tests
Randoop: Feedback-directed Random Test
Generation
Randoop Experience
Potential Improvements
2
Outline




Automated Test Generation of Unit Tests
Randoop: Feedback-directed Random Test
Generation
Randoop Experience
Potential Improvements
3
Automated Test Generation of Unit Tests

Problem
Given a method of a class, how to
automatically generate test cases to execute
most of its code, i.e., to achieve high code
coverage.
4
Automated Test Generation of Unit Tests

A simple example
class under test: java.util.Stack
method under test:
public synchronized int search(Object o) {
int i = lastIndexOf(o);
if (i >= 0) {
return size() - i;
}
return -1;
}
5
Automated Test Generation of Unit Tests

What need to be generated?

Test Inputs


not only primitive values but also objects, including the
receiver object
Test Oracles


Could use some basic contracts
For not much test inputs, manually added by testers
6
Automated Test Generation of Unit Tests

An example test case
// a test case for Stack.search(Object)
// existing (previous generated) sequences that create
parameters
Stack var0 = new Stack();
String var1 = "hi!";
var0.push((Object)var1);
// the method under test
int var2 = var0.search(var1);
// test oracles
assertTrue(var0.equals(var0));
assertTrue(var2==1); // Regression assertion
7
Automated Test Generation of Unit Tests

Main solutions

Random Approaches

Randomly create and mutate objects



For a method under test, select objects created in existing
sequences as its parameters
Simple, scalable, but may not efficient
Systematic Approaches


Explore the possible paths systematically, for each path,
collect the path condition and solve it to generate test
inputs
Potential high coverage, but complex
8
Outline




Automated Test Generation of Unit Tests
Randoop: Feedback-directed Random Test
Generation
Randoop Experience
Potential Improvements
9
Randoop: Feedback-directed Random Test
Generation

The basic idea


Undirected random test generation is not efficient
because it does not prune redundant or error
objects (generated by duplicate or different
method sequences)
Identify redundant or error objects early can prune
large search space
10
Randoop: Feedback-directed Random Test
Generation

The basic idea

Identify redundant objects


Use the equals() method
Identify error objects

Exceptions.


Null


Exceptions frequently correspond to precondition violations for a
method, and therefore there is little point extending them
Null dereference exceptions caused by using null as an argument
are often uninteresting
Set the redundant or error objects as not extensible
11
Randoop: Feedback-directed Random Test
Generation

Sequence generation for a method


To execute a method m(T1,…,Tk), we need to find
existing sequences that generate the parameters
For each Ti



Primitive type, randomly select a value from a pool
Reference type, select a extensible (non-redundant and
legal) value v from existing sequences, or use null
Generate a new sequence, including the method
m and the required sequences to generate the
parameters
12
Randoop: Feedback-directed Random Test
Generation

The Randoop algorithm
Repeat until timeout
 Randomly select a method m(T1,…,Tk)
 Generate a new sequence for m
 Execute the new sequence
 Check simple oracles, set the extensible flags of
the objects
Output the generated sequences as unit tests
13
Outline




Automated Test Generation of Unit Tests
Randoop: Feedback-directed Random Test
Generation
Randoop Experience
Potential Improvements
14
Randoop Experience

Subject programs



Java.util
Berkeley DB
Interested Metrics



Number of tests
Code coverage
Error detection
15
Randoop Experience

Experiments on Java.util






7 classes in Java.util, used in other researchers’
experiments
Run Randoop 60 seconds
4834 tests
50.8% instruction coverage
195 failed tests, many false positives!
Demo
16
Randoop Experience

Experiments on Berkeley DB







Java edition of Berkeley DB, com.sleepycat.je
Run Randoop 60 seconds
Around 20,000 tests
30% instruction coverage
300 failed tests, many false positives!
Need to prepare some environments, e.g., files in the disk
Can more tests achieve higher coverage?


Run Randoop 120 seconds: out of memory
Demo
17
Outline




Automated Test Generation of Unit Tests
Randoop: Feedback-directed Random Test
Generation
Randoop Experience
Potential Improvements
18
Potential Improvements

The problems of Randoop

Not enough code coverage


Difficult to generate interesting sequences quickly
before running out of memory
Too large test suites

Need to perform test selection (reduction)
19
Thank you!
20