Transcript Test and Metrics Plugin Framework

WP4: Testing Tools and
Methodologies
Status Report
ETICS Review – 15 February 2008
Éva Takács (4D SOFT)
www.eu-etics.org
INFSOM-RI-026753
Content
•
•
•
•
•
•
•
•
WP4 in numbers
Goals
Challenges
Distributed testing
Achievments
Case study
Future plans
Conclusion
INFSOM-RI-026753
WP in numbers
• 4D Soft: PM – 24 ?
• UW: PM –18
To be completed following the template
INFSOM-RI-026753
Staffing and Resources/ 4D Soft
?????
• István Forgács
• Tasks:
– Coordination Etics, Diligent
• Eva Takács – full time Etics
Tasks:
– Implementation of plugins
– Implementation and testing of installation modules
– Diligent system testing – modelling in ETICS
• Andor Dirner – 30% Etics, 60% Diligent, 10% system
administration
• Tasks:
– Implementation and testing of installation modules
– Diligent testing
INFSOM-RI-026753
4
Deliverables Status
• ETICS-D4.1-Requirements for unit, functional and
regression testing – M6
• ETICS-D4.2-Distributed testing – M12
• ETICS-D4.3-Distributed testing – draft M18
• ETICS-D4.3-Distributed testing – final M22
INFSOM-RI-026753
ETICS 3rd All-Hands Meeting - University of Wisconsin, Madison - 23-25/05/2007
5
Goals
The major goal of WP4 is to realize distributed testing by:
• Identification and analysis of test metrics
• Identification and selection of software testing tools
and metric calculation utilities
• Adopting of the selected testing tools and metric
calculation utilities to the ETICS build and test system
(Test and Metrics Plugin Framework) by plugin
implementation
• Implementation of distributed testing mechanisms
INFSOM-RI-026753
Challenges
• For the WP4 as technical work
– Realising distributed testing – a quite new aspect in software
engineering
– Customising existing ‘traditional’ testing technics for cutting-edge
new technologies
– Identifying the most suitable open source test tools that fit te best
in the ETICS system and could help the users with testing
facilities after performing builds and before releasing software
packages
• For 4D Soft as company
– Organisational challenge – being a WP leader as an SME in a
project having mostly large academic institutions
– Technical challenge - working with open source operation
systems/tools widely adopted by academic institutions in
contrast to the industry working mostly with commercially
available software
INFSOM-RI-026753
Distributed testing in ETICS
The aim of ETICS is to provide a distributed build/test
framework where the functionalities of the framework
and the testing tools are clearly separated. Having
these principles in mind in ETICS distributed testing
environment consists of four major elements:
• Test and Metrics Plugin framework – as part of the
ETICS client (command line interface)
• Testing tools and Tools for Metrics collection as
plugins of the Test and Metrics Plugin framework
• NMI/Metronome Build&Test system – as execution
framework
• co-scheduling mechanism – feature that allows automated
multi-site deployment and test execution
INFSOM-RI-026753
Test and Metrics Plugin Framework
In order to host the wide range of testing tools (client-side), without
breaking the ETICS’s core functionality, a plugin-framework was
developed
The goal of the plugin framework can be summarised as follows:
• Provide an extendible framework for common actions to be
performed during build and test execution
• Provide natural separation between specific build and test
execution and analysis and metrics collection
• Provide open interface for users to register their own plugins
(comercially available tools) to be executed during build and test
• Provide raw data for build and test and repository services
especially for tracking trends, having statistics
INFSOM-RI-026753
Why using Plugin Framework ?
ETICS ships with some built-in plugins
but the rational behind using „Eclipse style” plug-in
architecture is
• that most of present and/or future ETICS users will
come with some tests and testing tools already in place
and of course they would like to use them when
working with ETICS
• Users can plug comercially available testing tools
• Commercial providers can extend the sytem with their
products
INFSOM-RI-026753
Examples of metrics provided
• Static analysis
–
–
–
–
–
–
Single line of code count (SLOC)
Cyclomatic complexity
Depth of inheritance
IPv6 compliance
Check style
Nr of possible bugs
• Dynamic analysis
– Code coverage
– Memory leaks
• Test execution
–
–
–
–
–
Java: JUnit
C++: CPPUNIT (CPPTEST)
Python: PyUnit
Shell: shutil
Script/Executable
INFSOM-RI-026753
Implemented Test and Metrics
plugins
•
•
•
•
•
•
•
•
•
•
PyUnitPlugin.py (pyton unit tests)
PyCoveragePlugin.py (pyton coverage)
PyComplexityPlugin.py
SLOCCountPlugin
IPv6Plugin
WSInteroperabilityPlugin
JUnitPlugin.py (junit)
JUnitreportsPlugin.py
JUnitemmaPlugin.py (emma coverage for junit)
JCoberturaPlugin.py (cobertura coverage for junit)
INFSOM-RI-026753
12
Implemented Test and Metrics
plugins
•
•
•
•
•
•
JCnnPlugin.py (javaccn)
JDependPlugin.py (jdepend – dependency analyser)
JPmdPlugin.py (pmd – source code quality)
JFindbugsPlugin.py (findbugs – bug pattern analysis)
JRat.py (java profiler)
CFlawfinderPlugin.py (flawfinder - static analyser for
C/C++)
• CPyPHPRatsPlugin.py (Rats – static analyser for
C/C++, Perl, python, php)
• CValgrindPlugin.py (Valgrind – C profiler)
INFSOM-RI-026753
13
Metrics and their tool support in
ETICS
Metrics
Type
Programming
languages/ technologies
Tool
Etics Plugin
complexity
static
Java/Python
Javancss/ x.py
JCcnPlugin/
PyComplexityPlugin.py
design quality
static
Java
JDepend
JDependPlugin
nr of security bugs
static
C/C++/Python/Perl/ PHP
Flawfinder,
RATS
CFlawfinderPlugin/
CPyPhpRatsPlugin
nr of warnings
static
Java
PMD, Findbugs
JPmdPlugin,
JFindbugsPlugin
lines of code
static
all
coverage
dynamic
Java/Python
Emma/
Cobertura/
coverage.py
JUnitemmaPlugin
unit test/unit test reports
dynamic
Java/ Python
JUnit/PyUnit
JUnitPlugin/
JUnitreportsPlugin.py/
PyUnitPlugin.py
interoperability
static
IPv6/web service
profiler
dynamic
C/C++/Java
INFSOM-RI-026753
SLOCCountPlugin
IPv6Plugin/
WSInteroperabilityPlugin
Jrat, Valgrind
JRatPlugin/ CValgrindPlugin
Installation modules
• Additionally to plugins so-called installation modules
have been implemented
• The aim of installation modules is to assist users in
their installation and testing procedure by providing
some general purpose installation modules, for
example, tomcat, mysql or wscore.
• The installation modules are components with
appropriate test commands to install, start and stop the
services on which other test components can set
dependencies in order to get those services running
before the test is executed.
INFSOM-RI-026753
Major Tasks
Performed/Achievments
Distributed testing facilities are in place
Particularly,
– distributed testing environment is in place and co-schedulig
mechanism is implemented
– metrics and testing tools have been identified
– Test and Metrics Plugins Framework as part of ETICS client is in
place
– approximately 15 test and metrics plugins have been
implemented
– general purpose deployment scripts to assist testing have been
implemented
INFSOM-RI-026753
Case study - Diligent system testing
• Remote regression testing using the Diligent system
testing environment as a private resource
– grid7.4dsoft.hu/grid5.4dsoft.hu are attached to the etics pool
• Role of regression tests:
– Initially, to assure the testers that the test environment system is
testable. Successfull running of regression tests means for testers that
they can start working on the implementing new test cases.
– Later on, the initial set of test cases have been augmented with the new
test cases implemented by testers.
INFSOM-RI-026753
Case study - Diligent system testing
• Applying Java static analysis plugins for core Diligent
services
Applied plugins: JFindbugsPlugin, JDependPlugin
JPmdPlugin, JCcnPlugin
• The principal aim here has been to model a system
testing environment in ETICS for Diligent components
where the java static analysis plugins can run easily in
an automatic way on a periodic manner.
INFSOM-RI-026753
Future plan
• As the test execution mechanism assuaring test
process reproducibility is in place in ETICS
the future work in ETICS2 is
• to implement a modelling environment for creating
advance test cases/ test workflows
INFSOM-RI-026753
19
Conclusion
INFSOM-RI-026753
20