Transcript DevCOP: A Software Certificate Management System for Eclipse

DevCOP: A Software
Certificate Management
System for Eclipse
Mark Sherriff and Laurie Williams
North Carolina State University
ISSRE ’06
November 10, 2006
Agenda
• Background
– Motivation and Hypothesis
– Software Certificates and Software Certificate
Management Systems (SCMS)
• DevCOP (Defect Estimation with V&V Certificates
on Programming)
– Research Goal and Methodology
– DevCOP Certificates
– The DevCOP SCMS Eclipse plug-in
• Limitations
• Current Status
• Questions
2
Motivation
• Software Reliability
– Often not estimated until development is complete
– Actual reliability not known until system is shipped to
customers
• Corrective action is more expensive later in the
process
• If defect density could be estimated in-process…
– Steps could be taken to address issues early
– More economical, could improve development effort
3
Hypothesis
• Defect density estimation can be based upon the
history of verification and validation techniques
that have been performed on the project.
Questions that need to be answered:
• What is the best way to record V&V techniques?
• How do I build a model that can predict defect
density with V&V information?
4
Recording V&V Techniques
• Software Certificates
– A record of a verification and validation (V&V) practice
employed by developers and used to support
traceability between code and evidence of the V&V
technique used
• Software certificates could be any type of record:
– Logs from test case runs
– Reports from code inspections
– Details on pair programming assignments
• However, these are all different forms of V&V
information and maintaining this information could
be expensive.
5
Recording V&V Techniques
• Software Certificate Management Systems
– Provides an interface and infrastructure to
automatically create, maintain, and analyze
software certificates
– Benefits:
• Software maintenance
• Analysis of V&V technique effectiveness
• Reference in future projects
• Current research:
– OGI/PSU: Programatica, a SCMS for Haskell
6
DevCOP
• Research hypothesis: Defect density
estimation can be based upon the history of
V&V techniques that have been performed
on the project.
• Methodology: Build a parametric model
using software certificate information and
previous project defect data to create a
prediction model for future projects.
7
DevCOP Certificates
• Records:
– identifying information for the function it is
associated with including its name, signature,
class, and file location;
– identifying information for the developer that
created it;
– the type of V&V technique used;
– a hash of the function’s abstract syntax tree
(AST); and
– a significance weight.
8
DevCOP Certificates
• Types of Certificates
– Manual
• Includes all manual techniques, such as pair programming and
code inspections
– Automated Static Analysis
• Includes all techniques that can be run automatically on
uncompiled code
– Dynamic
• Includes all techiques performed automatically at run time,
such as black box testing
– Formal
• Includes all formal methods, such as lambda calculus and
proofs
9
DevCOP SCMS Eclipse Plug-in
• Currently supports manual V&V techniques
and jcoverage certificates
• Provides different methods for examining
and managing certificate data
• Demo
10
Potential Side Effects
• Retrospective Causal Analysis
– Once certificates are recorded on a project and bugs
are reported, developers can use certificate and defect
information to evaluate the efficacy of their V&V
practices.
• Building certificate information in with compiled
code base
– If certificate information could travel with compiled
code, it could be referenced at runtime so that other
systems could evaluate whether it was to work with
that system.
11
Original Hypothesis
• Parametric Modeling: Method by which
dependant variables are related to one or
more independent variables with regard to
previous data
• In Software Engineering…
– Purpose is to provide an estimated answer to a
software development question earlier in the
development lifecycle
• Famous SE parametric models: COCOMO
81 and COCOMO II
12
Original Hypothesis
• Early Research: Software Testing
Reliability Early Warning
– Uses a suite of metrics gathered on static code
to provide a reliability estimate
• If a reliability estimate can be created from
testing and static metrics, could it be
improved if we added other verification and
validation information to the model?
13
Limitations of DevCOP
• Eclipse Environment
– Even though numerous companies do use
Eclipse, it might not be standardized
• Tool Selection
– DevCOP uses JCoverage, which might not be
the approved coverage tool
• Does add minimal overhead
– Even adding one step can be too much…
14
Evolution of Research
• The core theory behind DevCOP is that
artifacts generated during software
development can be useful during the
development process.
• A certificate is simply an artifact of software
development.
• Can we gather certificates or other artifacts
with NO ADDITIONAL developer
involvement?
15
Evolution of Research
• SDAA: Software Development Artifact
Analysis
– By mining source control data, testing records,
and other V&V information, new associations
can be found between code
units.
– The type of association is
determined by the type
of development artifact
used.
16
Uses of SDAA
• Identification of Areas of Risk
– Associations generated from defect data
• Impact Analysis
– What files are likely to
change together?
• Test Case Prioritization
– How does a particular test
relate to a given
association?
17
Thank you!
• Questions? Queries? Quandries?
Contact Information:
Mark Sherriff
[email protected]
DevCOP Project:
http://agile.csc.ncsu.edu/devcop/
18