Transcript Data-Flow testing

Quality Assessment I
Peter Dolog
dolog [at] cs [dot] aau [dot] dk
2.2.05
Intelligent Web and Information Systems
October 28, 2010
This Lecture
Performance and Reliability Testing
Data Flow Testing (White Box)
Terminology in the quality assessment
Peter Dolog, Web Engineering 2010, Quality Assesment I
2
Performance and Reliability Testing
Functional Steps
© Mauro Andreolini, Michele
Colajanni, and Riccardo Lancellotti:
Web System Reliability and
Performance. In Web Engineering,
2006, Springer, pages: 181-218.
Peter Dolog, Web Engineering 2010, Quality Assesment I
4
Thoughput with technologies
© Mauro Andreolini, Michele Colajanni, and Riccardo Lancellotti: Web System
Reliability and Performance. In Web Engineering, 2006, Springer, pages: 181-218.
Peter Dolog, Web Engineering 2010, Quality Assesment I
5
Typical tasks
Representation of workload model definition
Traffic generation
Data Collection and analysis
Peter Dolog, Web Engineering 2010, Quality Assesment I
6
Representation of Workload Model
File lists with access frequences and analytical distribution
good enough for static resources, less flexible
File traces
Depends on representativeness of log
Finate State Automata
Flexible and yet suitable for complex patterns
Needs to be defined by hands which can be error prone and
labourous
Peter Dolog, Web Engineering 2010, Quality Assesment I
7
Traffic Generation
Emulates user interaction
Needs to generate and take into account cookies, session
management, and security (e.g. HTTPS)
Needs to take into account network delays
Given workflow model is taken as an input for traffic generation
Peter Dolog, Web Engineering 2010, Quality Assesment I
8
Data Collection and Analysis
You have to choose a metric and granularity samples
Mean, minimum and maximum may not be sufficient for
modelling performance of advanced applications
Advanced statistics is needed then
Subject of white box or black box strategies: resource vs. system
based metrics
Peter Dolog, Web Engineering 2010, Quality Assesment I
9
Black Box Testing
The whole web system as seen from outside
Served pages, hits, bytes per second -> administrator to check
the architecture
Response time: elapsed time between user click until arrival of
last object from the request -> interesting for users (not so
much throughput)
Response time higher than 30seconds is unacceptable for users
according to IBM
Less then 20 seconds is considered adequate
Peter Dolog, Web Engineering 2010, Quality Assesment I
10
Testing Approach
Define several workload models as patterns for you
Verify whether the performance stays stable
If not, analyse what could have caused problems
Web usage logs could give you a hint of a bottelnecks
Web application needs to operate with safe limits
Peter Dolog, Web Engineering 2010, Quality Assesment I
11
Response time as a function of traffic volume
© Mauro Andreolini, Michele Colajanni, and Riccardo Lancellotti: Web System
Reliability and Performance. In Web Engineering, 2006, Springer, pages: 181-218.
Peter Dolog, Web Engineering 2010, Quality Assesment I
12
White Box testing
Evaluates internal web applications components for different
client request rates (usually maximum)
You need to run it after black box in case of performane
problems
Reveals what causes the problem
Metrics look at resources such as CPU utilization, network,
central memory, file and socket descriptors, process table
entries, …
Metrics look also at the functions such as function call frequency,
percentage of time spent by program in each main function …
Peter Dolog, Web Engineering 2010, Quality Assesment I
13
Performance improvement
See last lecture
Peter Dolog, Web Engineering 2010, Quality Assesment I
14
Example workload models (400 concurrent
users)
© Mauro Andreolini, Michele Colajanni, and Riccardo Lancellotti: Web System
Reliability and Performance. In Web Engineering, 2006, Springer, pages: 181-218.
Peter Dolog, Web Engineering 2010, Quality Assesment I
15
System Design
HTML,
PhP (4),
MySQL (4),
Apache (2.0)
Linux, 2.4GHz, 1GB main memory, 80GB disks. Fast ethernet
Peter Dolog, Web Engineering 2010, Quality Assesment I
16
Testing Loop
© Mauro Andreolini, Michele Colajanni, and Riccardo Lancellotti: Web System
Reliability and Performance. In Web Engineering, 2006, Springer, pages: 181-218.
Peter Dolog, Web Engineering 2010, Quality Assesment I
17
Results
© Mauro Andreolini, Michele Colajanni, and Riccardo Lancellotti: Web System
Reliability and Performance. In Web Engineering, 2006, Springer, pages: 181-218.
Peter Dolog, Web Engineering 2010, Quality Assesment I
18
Additional response time testing and stressing
© Mauro Andreolini, Michele Colajanni, and Riccardo Lancellotti: Web System
Reliability and Performance. In Web Engineering, 2006, Springer, pages: 181-218.
Peter Dolog, Web Engineering 2010, Quality Assesment I
19
White box testing (2)
© Mauro Andreolini, Michele Colajanni, and Riccardo Lancellotti: Web System
Reliability and Performance. In Web Engineering, 2006, Springer, pages: 181-218.
Peter Dolog, Web Engineering 2010, Quality Assesment I
20
White Box Testing
© Mauro Andreolini, Michele Colajanni, and Riccardo Lancellotti: Web System
Reliability and Performance. In Web Engineering, 2006, Springer, pages: 181-218.
Peter Dolog, Web Engineering 2010, Quality Assesment I
21
Data Flow White Box Testing
Control Flow Graph
ORD = (V, E)
V … Set of nodes representing objects
E … VxV relationships betwen objects
Objects: client pages, server pages, components
Objects have attributes: variables or document elements
Objects have operations: functions in scripting or prog. Language
Graph can be defined recursively
Relationships: inheritance, aggregation, association, request,
response, navigation, and redirect
Peter Dolog, Web Engineering 2010, Quality Assesment I
23
Example: bidding web application
© Liu, C., Kung, D. C., Hsia, P., and Hsu, C. 2000. Object-Based Data Flow Testing of Web Applications.
In Proceedings of the the First Asia-Pacific Conference on Quality Software (Apaqs'00) (October 30 - 31, 2000).
APAQS. IEEE Computer Society, Washington, DC, 7.
Peter Dolog, Web Engineering 2010, Quality Assesment I
24
Principles
© Liu, C., Kung, D. C., Hsia, P., and Hsu, C. 2000. Object-Based Data Flow Testing of Web Applications.
In Proceedings of the the First Asia-Pacific Conference on Quality Software (Apaqs'00) (October 30 - 31, 2000).
APAQS. IEEE Computer Society, Washington, DC, 7.
Peter Dolog, Web Engineering 2010, Quality Assesment I
25
Composite CFG
© Liu, C., Kung, D. C., Hsia, P., and Hsu, C. 2000. Object-Based Data Flow Testing of Web Applications.
In Proceedings of the the First Asia-Pacific Conference on Quality Software (Apaqs'00) (October 30 - 31, 2000).
APAQS. IEEE Computer Society, Washington, DC, 7.
Peter Dolog, Web Engineering 2010, Quality Assesment I
26
© Liu, C., Kung,
D. C., Hsia, P.,
and Hsu, C.
2000. ObjectBased Data
Flow Testing of
Web
Applications.
In Proceedings
of the the First
Asia-Pacific
Conference on
Quality
Software
(Apaqs'00)
(October 30 31, 2000).
APAQS. IEEE
Computer
Society,
Washington,
DC, 7.
Peter Dolog, Web Engineering 2010, Quality Assesment I
27
OCFG for the example
© Liu, C., Kung, D. C., Hsia, P., and Hsu, C. 2000. Object-Based Data Flow Testing of Web Applications.
In Proceedings of the the First Asia-Pacific Conference on Quality Software (Apaqs'00) (October 30 - 31, 2000).
APAQS.
IEEE Computer
Society,2010,
Washington,
DC, 7.
Peter Dolog,
Web Engineering
Quality Assesment
I
28
5-level Data flow testing
Function level (CFG)
Function cluster level (ICFG)
Object level (OCFG + criterion)
Object cluster level (ICFG cross object, CCFG for http)
Application level (CCFG)
Peter Dolog, Web Engineering 2010, Quality Assesment I
29
Def use chains from graphs
© Liu, C., Kung, D. C., Hsia, P., and Hsu, C. 2000. Object-Based Data Flow Testing of Web Applications.
In Proceedings of the the First Asia-Pacific Conference on Quality Software (Apaqs'00) (October 30 - 31, 2000).
APAQS. IEEE Computer Society, Washington, DC, 7.
Peter Dolog, Web Engineering 2010, Quality Assesment I
30
Summary
Can uncover error at the lovest levels
Provides a system
However, does not scale for complex applications
You can use it in your project for the most important functions
Peter Dolog, Web Engineering 2010, Quality Assesment I
31
Terminology
Quality
According to ISO 8402-86 Quality is the totality of features and
characteristics of software product that relate to its ability to
satisfy stated or implied needs.
Peter Dolog, Web Engineering 2010, Quality Assesment I
33
Quality Attributes (ISO/IEC 9126-1)
Functionality – correct and adequate functions
Reliability – capability of ensuring given level of performance
Usability – related to ease of use by end users
Efficiency – related to the ratio of the performance to the
number of used resources
Maintanability – ease of modification even after deployment
Portability – being able to run on different platforms
Peter Dolog, Web Engineering 2010, Quality Assesment I
34
Quality Model
Quality model is a selection of criteria to assess properties
related to quality characteristics
Quality assessment => measuring the properties related to the
above according to specified quality metrics for those
Peter Dolog, Web Engineering 2010, Quality Assesment I
35
Quality Characteristics (ISO/IEC 9126)
© Springer, Web Applications Engineering 2009
Peter Dolog, Web Engineering 2010, Quality Assesment I
36
Quality Perspectives
© Springer, Web Applications Engineering 2009
Peter Dolog, Web Engineering 2010, Quality Assesment I
37
Quality in Use
Effectiveness – capability of reaching the users goals with
accuratness and completeness
Productivity – capability of spending appropriate amount of
resources in relation to effectiveness to be achieved
Safety – capability of achieving acceptable level of risks for the
users, the usage environment, and the users’ activities
Satisfaction – a capability to satisfy the user in a given context of
use
Peter Dolog, Web Engineering 2010, Quality Assesment I
38
Evaluation of Quality
Internal attributes – e. g. static measures on the code
External attributes – e. g. dynamic measures of behavior of the
code
Quality in use attributes – e. g. reduction of time to produce an
invoice
Peter Dolog, Web Engineering 2010, Quality Assesment I
39
Web Application Functionality
Content
Navigation
Correctness and Adequateness
=> Functional Testing
Peter Dolog, Web Engineering 2010, Quality Assesment I
40
Performance
Throughput:
number of requests per unit of time
Response time:
the time needed to serve a request
Evaluation in average and peak conditions
Peter Dolog, Web Engineering 2010, Quality Assesment I
41
Usability
Easy-to-use navigation mechanisms
Quickly reaching relevant content and operation
Not getting lost
Accessibility
Peter Dolog, Web Engineering 2010, Quality Assesment I
42
Testing Web Applications
White Box Testing
Black Box Testing
Gray Box Testing
Unit Testing
Integration Testing
System Testing
Peter Dolog, Web Engineering 2010, Quality Assesment I
43
Types of Web Applications Testing
Functional testing: Ghost pages, Unreachable pages
Model-based testing: using design models to find out whether
quality criteria are satified
Data-flow testing: to reveal invalid use of data values due to
coding errors based on control flow
Peter Dolog, Web Engineering 2010, Quality Assesment I
44
Structural Model supporting Testing
© Springer, Web Applications Engineering 2009
Peter Dolog, Web Engineering 2010, Quality Assesment I
45
Data-Flow testing
Path coverage
Branch coverage
Node coverage
Peter Dolog, Web Engineering 2010, Quality Assesment I
46
Control Flow Model
© Springer,
Web
Applications
Engineering
2009
Peter Dolog, Web Engineering 2010, Quality Assesment I
47