Transcript Model-based testing

AUTOMATED TEST
GENERATION
Muhammed İ. KALKAN
Automatic Test Generation (ATG)

Index








What is ATG?
ATG Types
Why do we need ATG?
What are the main benefits of ATG
ATG Methods
Testing Methodologies
Automatic Test Pattern Generation
Relevant Examples
What is ATG?



Testing an engineered system is the
most important part of the product
revealed.
Engineers are working on and
putting alot of effort into this issue.
Since testing process is required for
all obtained engineering products,
engineers decided to develop an
automated test mechanism instead
of generating test cases manually.
What is ATG?



Automated Test Generation includes
test cases , scenarios and the
evalution of the obtained result such
as passed and failed.
Given inputs and obtained outputs
are evaluated automatically.
The Main Idea :
 TEST COMPLETELY
AUTOMATICALLY
ATG or ATPG?



ATG is an engineering subject which
can be applied to both Computer
Sciences and other engineering
specialities like Electronics or
hardware circuits.
ATG is a term which is generally
used for Computer Sciences
especially software testing.
On the other hand for hardware
testing Automatic Test Generation is
usually referred as Automatic Test
Pattern Generation.
ATG or ATPG?



ATPG is defined as follows:
ATPG is an electronic pattern
automation method/technology used
to find an input (or test) sequence
When applied to a digital circuit, it
enables testers to distinguish
between the correct circuit behavior
and the faulty circuit behavior
caused by defects.
ATG…….But WHY??





As years have passed,computer hardwares
and its capabilities grow bigger and bigger.
Result 1: Software types and their
capabilities are varied.
Result 2: New software demand on market
for jobs.
Result 3: Bigger income ,more firms to
compete for market ration.
Result 4: Requirement for strong,bug free
and developed in small amount of time
softwares
ATG…….But WHY??

Result 5 : Software Engineering
and Testing studies.
ATG…….But WHY??

Software testing is important
Software errors cost the U.S.
economy about $59.5 billion each
year
 Improving testing infrastructure
could save 1/3 cost


Software testing is costly

Account for even half the total cost
of software development
Benefits



Reduces Testing Time
Reduces Testing Cost
Reduces human errors while
testing
ATG Methods


Data-Driven ATG
xUnit Frameworks for ATG



Examples : JUnit and NUnit
Model-Based ATG
Keyword-Driven ATG
xUnit


Various code-driven testing
frameworks have come to be
known collectively as xUnit.
Based on a design by Kent
Beck, originally implemented for
Smalltalk as SUnit, but are now
available for many programming
languages and development
platforms.
xUnit




The overall design of xUnit frameworks
depends on several components.
Test Fixtures
 A test fixture is the set of preconditions or
state needed for a test to run. Also known
as a test context.
Test Suites
 A test suite is a set of tests that all share the
same fixture.
Test Execution
 The execution of an individual unit
test proceeds as follows:
xUnit

The setup() and teardown() methods serve to
initialize and clean up test fixtures.

Assertions
 An assertion is a function or macro that
verifies the behavior of the unit under test.
Failure of an assertion typically throws an
exception, aborting the execution of the
current test.
Data-Driven ATG




Data-driven testing is a methodology
used in Test automation
Test scripts are executed and verified
based on the data values stored in one or
more central data sources or databases.
Databases can range from datapools,
ODBC sources, csv files, Excel files, DAO
objects, ADO objects, etc.
Data-driven testing is the establishment of
several interacting test scripts together with
their related data results in a framework
used for the methodology.
Data-Driven ATG

Variables are used for both input values and
output verification values: navigation through
the program, reading of the data sources, and
logging of test status and information are all
coded in the test script. Thus, the logic
executed in the script is also dependant on the
data values.

An example of xUnit Framework Automatic
Testing by using Data-Driven Test Methodology
is as follows:
Example


Put the data that
varies from test
to test into the
Data-Driven Test
file that the
interpreter reads
to execute the
tests
For each test it
does the same
sequence of
actions to
implement the
Four-Phase Test
Example (FPT)




Set-up the test fixture
Try the test case
Evaluate the result
Release the fixture
Keyword-Driven ATG




It is also known as table-driven testing, actionword testing
It can be used for manuel testing as well as
automated testing
The advantages for automated tests are the
reusability and therefore ease of maintenance
of tests which have been created at a high
level of abstraction.
The Keyword-Driven ATG methodology divides
test creation into two stages.


Planning Phase
Implementing Phase
Keyword-Driven ATG
Planning Phase

The application (or the
requirements for the application)
is analyzed to determine which
operations and objects will need
to be tested.
Example
A web-questionnaire application will
require a large amount of text entries. By
identifying which operations should be
encapsulated
into
keywords,
the
efficiency and maintainability of the tests
are maximized.
Keyword-Driven ATG
Implementation Phase




The implementation stage differs depending on the
tool or framework used.
Often, automation engineers implement a
framework that provides keywords like “check” and
“enter”.
Testers or test designers (who don’t have to know
how to program) write test cases based on the
keywords defined in the planning stage, which have
been implemented by the engineers.
The test is executed using a driver which reads the
keywords and executes the corresponding code.
Keyword-Driven ATG




Other methodologies use an all-in-one
implementation stage.
Instead of separating the tasks of test
design and test engineering, the test
design is the test automation.
Keywords, such as “enter” or “check” are
created using tools in which the necessary
code has already been written.
This removes the necessity for extra
engineers in the test process, because the
implementation for the keywords is already
a part of the tool.
Keyword-Driven ATG





Benefits
This methodology requires more planning and a
longer initial time-investment than going directly to
the test creation stage and recording a test.
It does make the test creation and test maintenance
stages more efficient and keeps the structure of
individual tests more readable and easier to modify.
The more abstract keywords are, the more reusable
they are, and therefore the easier a test is to
maintain.
As well as reducing the cost and time spent
maintaining and updating tests, the modular
structure of Keyword-Driven ATG means that new
tests can easily be created from pre-existing
modules.
Keyword-Driven ATG



Another advantage is the reduction
in technical know-how required for
the test automation process.
In the first approach, technical knowhow is only required by the
engineers that implement the
keywords.
In the second approach, even this is
not required, which means that the
test team is capable of entirely
automating tests, even without
programming knowledge.
Model Based ATG

Model-based testing is software testing in
which test cases are derived in whole or in part
from a model that describes some (usually
functional) aspects of the system under test
process

The model is usually an abstract, partial
presentation of the system under test's desired
behavior.

The test cases derived from this model are
functional tests on the same level of
abstraction as the model

Can not run abstracted test cases.Need
executable ones.
Model Based ATG

Diagram description :
Model-Based Test Systems
Model Based ATG




Many ways to produce test cases.
No best method for producing tests.
Use your test interest to define your test
cases.
Design decisions is often known as
"test requirements", "test purpose" or
even "use case"
Model Based ATG


Use model and test requirements to
derive abstract test cases
Implementation Extra Information to
abstract test suit to produce executable
test cases
Model Based ATG


Execute the test
Evaluate the reports
Model Based ATG

There are a few ways to deploy the
Model-Based Testing


Online Testing: A model-based testing
tool connects “directly” to a system
under test and tests it dynamically.
Offline Generation of Executable
Tests: A model-based testing tool
generates test cases as a computerreadable asset that can be later
deployed automatically.

A collection of Python classes that embodies
the generated testing logic
Model Based ATG


Offline Generation of Manually
Deployable Tests: A model-based
testing tool generates test cases as a
human-readable asset that can be later
deployed manually.
 A PDF document in English that
describes the generated test steps
Another Step :

Test Generation
Model Based ATG




Model-Based Testing has a good potential in
aspect of automation.
Test case generation is one of the issues that
ATG is subject to
To find test cases, Model-Based structure is
interpreted as a kind of Finite State Machine to
see different states that the system might be in
There are a few technics for generating test
cases.Most knowns are as follows:
Model Based ATG

Test case generation by theorem proving



System model is defined as predicates,kinds of
logical expressions
These expression are processed and proved
afterwards
Test case generation by constraint logic
programming

Selecting test cases satisfying specific
constraints by solving a set of constraints over a
set of variables
Model Based ATG

Test case generation by model checking


Test case generation by symbolic execution


We provide a model of the system under test and a
property we want to test to the model checker
Searching for execution traces in an abstract model.
In principle the program execution is simulated using
symbols for variables rather than actual values
Test case generation by using an event-flow
model
 With a graphical user-interface (GUI) front-end is
called the event-flow model that represents
events and event interactions
Example
TestMaster™ automates the generation of
tests for call processing features developed
for the 5ESS®-2000 Switch
This test uses Model-Based Test Generation



Results obtained here as follows.

A test generation productivityimprovement
of just over 90%
Automatic Test Pattern
Generation



ATPG is an electronic design automation
method/technology used to find an input (or
test) sequence that, when applied to a digital
circuit, enables testers to distinguish between
the correct circuit behavior and the faulty circuit
behavior caused by defects
The generated patterns are used to test
semiconductor devices after manufacture,
In some cases to assist with determining the
cause of failure
Automatic Test Pattern
Generation


The effectiveness of ATPG is
measured by the amount of modeled
defects, or fault models, that are
detected and the number of
generated patterns.
We can estimate test quality by
effectiveness of ATGP
Automatic Test Pattern
Generation




A defect is an error introduced into a device
during the manufacturing process.
A fault model is a mathematical description of
how a defect alters design behavior.
A fault is said to be detected by a test pattern
if, when applying the pattern to the design, any
logic value observed at one or more of the
circuit's primary outputs differs between the
original design and the design with the fault.
There are two steps that ATPG should take to
detect fault.
Automatic Test Pattern
Generation


Fault activation: Establishes a signal
value at the fault model site that is
opposite of the value produced by the
fault model
Fault propagation: Moves the resulting
signal value, or fault effect, forward by
sensitizing a path from the fault site to a
primary output.
Sequential ATPG

Searches for a sequence of vectors to detect a
particular fault through the space of all possible
vector sequences

Vector Sequence: A sequence of values for
circuit input

In Sequential Circuits due to the presence of

memory elements, the controllability and
observability of the internal signals (in
general)are much more difficult than those
in a combinational circuit
Complexity of sequential ATPG much higher
than that of combinational ATPG.
Algorithmic Methods



Testing very-large-scale integrated circuits with
a high fault coverage is a difficult task because
of complexity.
Many different ATPG methods have been
developed to address combinatorial and
sequential circuits.
Some examples:

Pseudorandom test generation is the simplest
method of creating tests. It uses a
pseudorandom number generator to generate
test vectors
Algorithmic Methods


The D Algorithm was the first practical test
generation algorithm in terms of memory
requirements. The D Algorithm introduced D
Notation which continues to be used in most
ATPG algorithms.
Fan-Out Oriented Algorithm: It limits the
ATPG search space to reduce computation
time and accelerates backtracing.
Automatic Test Pattern
Generation

ATPG can fail to find a test for a
particular fault in at least two cases.
 The fault may be intrinsically
undetectable
 it is possible that a pattern(s)
exist, but the algorithm cannot find
it
References




Automated Test Generation, (From a Behavioral
Model), James M. Clarke, Lucent Technologies
Automatic Test Pattern Generation ,
http://en.wikipedia.org/wiki/Automatic_test_pattern_
generation
Data Driven Tests, Gerrard Meszaros,
http://xunitpatterns.com/Data-Driven%20Test.html,
2007
Data-Driven Testing,
http://en.wikipedia.org/wiki/Data-driven_testing
References




Test Automation,
http://en.wikipedia.org/wiki/Test_automation
Test Automation Framework,
http://en.wikipedia.org/wiki/Test_automation_framew
ork
A Survay on Automatic Test Case Generation,
M.Prasanna, S.N. Sivanandam, R.Venkatesan,
R.Sundarrajan, Department of Computer Science
and Engineering, PSG College Of Technology,
http://www.acadjournal.com/2005/v15/part6/p4/
Keyword-Driven Testing ,
http://en.wikipedia.org/wiki/Keyword-driven_testing
References

Model – Based Testing,
http://en.wikipedia.org/wiki/Model-based_testing

Improving Automation in Developer Testing:
Achievements and Challenges, Tao Xie,
Department of Computer Science, North Carolina
State University

xUnit , http://en.wikipedia.org/wiki/XUnit
The End
Thank You
Q&A