Transcript Plugin - CERN Indico

WP4: Plugins and
installation modules
Status Report
ETICS All-Hands – 23 May 2007
Presenter: Eva Takács (4D SOFT)
www.eu-etics.org
INFSOM-RI-026753
Location of the plugins
• Location of the plugins:
\ETICS_HOME\etics\lib\python2.3\sitepackages\plugins directory of the etics client installation.
INFSOM-RI-026753
ETICS 3rd All-Hands Meeting - University of Wisconsin, Madison - 23-25/05/2007
2
Location of plugins
• The \ETICS_HOME\etics\lib\python2.3\site-packages\plugins
directory has additional subdirectories:
• \plugins_externals – containing the tools and their
dependencies
• \plugins_buildfiles – containing the ant scripts with
the core functionality of the implemented plugins. These ant
scripts are wrapped by the execute() function of the
PluginManager itself.
• \plugins_properties –containing the property files of
the ant scripts
• \plugins_docs –containing the brief description of the
implemented plugins
INFSOM-RI-026753
Included tools as plugins
• At present the following tools are included as plugins:
• \cobertura – containing the libraries of the java coverage
tool cobertura
• \emma – containing the libraries of the java coverage tool
emma
• \junit –containing junit library
• \pmd – containing the libraries of the pmd tool
• \findbugs – containing the libraries of FindBugs
• \javaccn – containing the libraries of javancss
• \jdepend – containing the libraries of jdepend tool
INFSOM-RI-026753
ETICS 3rd All-Hands Meeting - University of Wisconsin, Madison - 23-25/05/2007
4
Naming convention
• Naming convention of the plugins:
JtoolnamePlugin.py where
• ‘J’ – indicates that java programs are under analysis
• ‘toolname’ – refers to the integrated tool
• ‘Plugin’ – should be in the name of the module to
enable the PluginManager.py to load the plugin
during the execution of the build
INFSOM-RI-026753
ETICS 3rd All-Hands Meeting - University of Wisconsin, Madison - 23-25/05/2007
5
JUnit plugin
• Definition: this plugin is responsible for running JUnit
test cases/suites using the widely-used ant
mechanism.
• Input directory:
• {workspaceDir}/repository/{projectName}/{mod
uleName}/{version}/noarch/lib
• Output: xml, html reports located in
• {workspaceDir}/reports/JUnitReports
INFSOM-RI-026753
ETICS 3rd All-Hands Meeting - University of Wisconsin, Madison - 23-25/05/2007
6
Emma JUnit plugin
• Definition: this plugin is responsible for calculating
EMMA coverage metrics for JUnit test cases.
• Input directory:
• {workspaceDir}/repository/{projectName}/{mod
uleName}/{version}/noarch/lib
• Output junit results: xml, html reports located in
• {workspaceDir}/reports/JUnitReports-emma
• Output emma coverage results: xml, html reports
located in
• {workspaceDir}/reports/coverage-emma
INFSOM-RI-026753
ETICS 3rd All-Hands Meeting - University of Wisconsin, Madison - 23-25/05/2007
7
Cobertura JUnit plugin
• Definition: this plugin is responsible for calculating
coverage metrics for JUnit test cases using Cobertura.
• Input directory:
• {workspaceDir}/repository/{projectName}/{mod
uleName}/{version}/noarch/lib
• Output junit results: xml, html reports located in
• {workspaceDir}/reports/JUnitReports-cobertura
• Output emma coverage results: xml, html reports
located in
• {workspaceDir}/reports/coverage-cobertura
INFSOM-RI-026753
ETICS 3rd All-Hands Meeting - University of Wisconsin, Madison - 23-25/05/2007
8
JDepend plugin
• Definition: this plugin is responsible for scanning Java
byte code and generating dependency info. The open
source jdepend tool is used in this plugin.
• Input directory:
{workspaceDir}/repository/{projectName}/{mod
uleName}/{version}/noarch/lib
• Output:
• Html(jdepend-report.html)
and
report.xml)
reports
are
{workspaceDir}/reports directory.
INFSOM-RI-026753
xml(jdependcreated
in
JPmd plugin
Definition: this plugin is responsible for scanning Java
source code and looking for potential problems like:
• Possible bugs - empty try/catch/finally/switch
statements
• Dead code - unused local variables, parameters and
private methods
• Suboptimal code - wasteful String/StringBuffer usage
• Overcomplicated expressions - unnecessary if
statements, for loops that could be while loops
The open source pmd tool is used in this plugin.
INFSOM-RI-026753
JPmd plugin
• Input directory:
{workspaceDir}/repository/{projectName}/{mod
uleName}/{version}/noarch/src
• Output:
• Html (pmdReport.html) and xml (pmdReport.xml)
reports are created in {workspaceDir}/reports
directory.
INFSOM-RI-026753
Interpreting the results
• PMD works by scanning Java code and
checks for violations in three major
areas:
INFSOM-RI-026753
ETICS 3rd All-Hands Meeting - University of Wisconsin, Madison - 23-25/05/2007
12
Interpreting the results
• Compliance with coding standards such as:
– Naming conventions - class, method, parameter and variable
names
– Class and method length
– Existence and formatting of comments and JavaDocs
INFSOM-RI-026753
Interpreting the results
• Coding antipatterns such as:
–
–
–
–
Empty try/catch/finally/switch blocks
Unused local variables, parameters and private methods
Empty if/while statements
Overcomplicated expressions - unnecessary if statements, for
loops that could be while loops
– Classes with high Cyclomatic Complexity measurements
INFSOM-RI-026753
Interpreting the results
• More precisely, PMD defines rulesets and inside each ruleset a set
of rules in order to verify the source code.
• The default of rulesets are as follows:
• # Basic (rulesets/basic.xml) – looks for some basic problems, for
example, catch blocks shouldn't be empty, override hashCode()
anytime you override equals(), etc.
• # Naming (rulesets/naming.xml) -- Tests for the standard Java
naming conventions: variable names should not be too short;
method names should not be too long; class names should begin
with an uppercase letter, method and field names should begin
with a lowercase letter, etc.
INFSOM-RI-026753
Interpreting the results
• # Import statements (rulesets/imports.xml) -- Checks
for minor issues with import statements, such as
importing the same class twice or importing a class
from java.lang.
• # JUnit tests (rulesets/junit.xml) -- Looks for specific
issues with test cases and test methods, such as
correct spelling of method names and whether suite()
methods are static and public.
INFSOM-RI-026753
Interpreting the results
• # Strings (rulesets/string.xml) -- Identifies common problems that
when working with strings, such as duplicate string literals, calling
the String constructor, and calling toString() on String objects.
•
• # Braces (rulesets/braces.xml) -- Checks whether for, if, while, and
else statements use braces.
•
• # Code size (rulesets/codesize.xml) -- Tests for overly long
methods, classes with too many methods, and similar candidates
for refactoring.
•
• # Javabeans (rulesets/javabeans.xml) -- Inspects JavaBeans
components for violations of JavaBeans coding conventions,
such as unserializable bean classes.
INFSOM-RI-026753
Interpreting the result
• # Strict exceptions (rulesets/strictexception.xml) -- A few more
tests for exceptions: methods should not be declared to throw
java.lang.Exception, exceptions should not be used for flow
control, Throwable should not be caught.
• # Controversial (rulesets/controversial.xml) -- This rule set
contains some of the more questionable checks, including
assigning null to a variable, multiple return points from a method,
and importing from the sun packages.
• # Logging (rulesets/logging-java.xml) -- Searches for strange uses
of java.util.logging.Logger, including nonfinal, nonstatic loggers,
and more than one logger in a class.
INFSOM-RI-026753
JFindBugs plugin
• Definition: this plugin is responsible for scanning Java
byte code and looking for potential bugs. The open
source findbugs tool is used in this plugin.
• Input directory:
{workspaceDir}/repository/{projectName}/{mod
uleName}/{version}/noarch/lib
• Output:
Html (findbugsReport.html) and xml
(findbugsReport.xml) reports are created in
{workspaceDir}/reports directory.
INFSOM-RI-026753
Interpreting results
• FindBugs results are stored in html/xml reports.
• The html report contains a summary of the bug
categories indicating their severity.
• Then each bug is detailed and the location of it in the
source file is also specified.
• Lastly, an explanation of the bugs found are put at the
end of the report. One example of such a report can be
found
INFSOM-RI-026753
Interpreting results
• FindBugs looks for potential problems by matching the
bytecodes against a list of bug patterns.
• Bug patterns are code idioms that are often errors.
• FindBugs doesn't focus on style or formatting; it
specifically tries to find real bugs or potential
performance problems.
• Bug patterns in principally fall in the following
categories:
–
–
–
–
Single-threaded correctness issue
Thread/synchronization correctness issue
Performance issue
Security and vulnerability to malicious untrusted code
INFSOM-RI-026753
OS - Open Stream
• When a program opens an input or output stream, it is good
practice to ensure that the stream is closed when it becomes
unreachable. Although finalizers ensure that Java I/O streams are
automatically closed when they are garbage collected, there is no
guarantee that this will happen in a
• timely manner. There are two reasons why streams should be
closed as early as possible. First, operating system file
descriptors are a limited resource, and running out of them may
cause the program to misbehave. Another reason is that if a
buffered output stream is not closed, the data stored
• in the stream’s buffer may never be written to the file (because
Java finalizers are not guaranteed to be run when the program
exits).
•
INFSOM-RI-026753
OS - Open Stream
• An example of an open stream found by the bug
detector is as follows:
•
• private static File _parsePackagesFile(File
packages, File destDir) {
•
try {
•
FileReader fr =new
FileReader(packages);
•
BufferedReader br =new
BufferedReader(fr);
• ...
• // fr/br are never closed
• }
INFSOM-RI-026753
NP – Null pointer dereference
• NP – Null pointer dereference
• Calling a method or accessing an instance field of a null value
results in a NullPointerException at runtime. This detector looks
for instructions where a null reference might be dereferenced.
•
• An example of a null pointer dereference found by the NP bug
detector is as follows:
• if (entry == null) {
•
IClasspathContainer
container=JavaCore.getClasspathContainer (
•
entry.getPath(), // entry is null!
•
root.getJavaProject());
• .....
• }
INFSOM-RI-026753
UR - Uninitialized Read In
Constructor
• When a new object is constructed, each field is set to the default
value for its type. In general, it is not useful to read a field of an
object before a value is written to it. This detector finds fields that
are read in constructors before they're initialized.
•
• public class Thing {
•
private List actions;
•
public Thing(String startingActions) {
•
StringTokenizer tokenizer = new
StringTokenizer(startingActions);
•
while (tokenizer.hasMoreTokens()) {
•
actions.add(tokenizer.nextToken());
•
}
•
}
• }
• In this example, line red will cause a null pointer exception
because the variable actions has not been initialized.
INFSOM-RI-026753
Installation modules
• The aim of installation modules is to assist users in
their installation procedure by providing some general
purpose installation modules, for example, for tomcat,
mysql, wscore
INFSOM-RI-026753
Installation modules
• In order to be more flexible, it was intended to
implement a general installation procedure for all
software packages while the individual features such
as the commands for testing the installation should be
included in a separate config file.
INFSOM-RI-026753
Steps of installation
• To have (download) the requested package
• The implemented automatism guess the file type from
the mime type and unpacks or installs the packages
accordingly
• Then the package is installed in the specified location
• The last step is to check the installation using the
testing command specified in the config file. By default
the testing command is provided, but the user can
change it.
INFSOM-RI-026753
Config file
• [TOMCAT]
• tomcat.url =
http://xenia.sote.hu/ftp/mirrors/www.apache.org/tomc
at/tomcat-6/v6.0.10/bin/apache-tomcat-6.0.10deployer.tar.gz
• tomcat.destdir = /home/takacs/Applications/tomcat
• tomcat.installcmd = mkdir -p %(tomcat.destdir)s &&
rm -fr %(tomcat.destdir)s/* && cp -r
%(tomcat.unpackpath)s/* %(tomcat.destdir)s/
• tomcat.javahome = $(dirname $(dirname $(which
java)))
• tomcat.startcmd = (JAVA_HOME=%(tomcat.javahome)s
%(tomcat.destdir)s/apache-tomcat6.0.10/bin/startup.sh)
• tomcat.checkcmd = (sleep 21 && wget
http://localhost:8080 -O /tmp/TOMCAT.TEST && cat
/tmp/TOMCAT.TEST)
• tomcat.stopcmd = (JAVA_HOME=%(tomcat.javahome)s
%(tomcat.destdir)s/apache-tomcatINFSOM-RI-026753
Etics
•
•
•
•
•
•
•
•
•
•
1. etics-show-module-structure externals | grep tomcat
Result:
>Component : tomcat
2. etics-checkout --project-config externals tomcat
Result:
>Downloading configuration 'tomcat v. 5.0.28' from
>repository
'http://eticssoft.web.cern.ch/eticssoft/repository//externals/tomcat/
5.0.28/noarch/tomcat5->5.0.28.tar.gz'
•
• At this point ’tomcat5-5.0.28.tar.gz’ is located in
{workspaceDir}/repository/{projectName}/{moduleName}
/{version}/noarch.
INFSOM-RI-026753
YUM
• If the ETICS repository does not contain the required
tool, then the next option is to use the YUM
mechanism. In order to successfully use YUM the
/etc/yum.repos.d/*.repo files should be present and
filled with the necessary data.
•
•
•
•
which yum
yum update (updating just metadata not libraries)
yum list all | grep tomcat
yum install tomcat
INFSOM-RI-026753
Apt
• Lastly, if no YUM mechanism is provided then apt is
used. The following commands are run:
• which apt
• apt-search tomcat (the biggest version of the
component is searched)
• apt-get update (updating metadata)
• apt-get upgrade (upgrading the libraries)
• apt-get install
INFSOM-RI-026753