Transcript José Maldonado: Readers Project

13th International Software Engineering Research Network
ISERN 2005
Readers Project:
Quality aspects in Empirical Studies
José Carlos Maldonado
[email protected]
ICMC/USP
Instituto de Ciências Matemáticas e de Computação
Universidade de São Paulo – São Carlos
Noosa Heads, Austrália - November, 2005
Readers’ Team
ISERN-2005
READERS:
PBR studies
University of Maryland
Universidade de São Paulo
Victor Basili
José Carlos Maldonado
(Brazil Coordinator)
(USA Coordinator)
Forrest Shull
Jeffrey Carver
PBR
Secondary Study
Competence
Centers Portal
Universidade Federal
do Rio de Janeiro
Universidade Federal de São Carlos
Sandra Fabbri
Guilherme Horta Travassos
Manoel Mendonça
Universidade Salvador
ISERN-2005
Readers Project
Readers Project
NSF-CNPQ Readers Project
- A Collaborative Research to Develop,
Validate, and Package Reading Techniques for
Software Defect Detection
•Investigating replications and transfer of
experimental knowledge issues
• Basili-Selby (Extended) Experiment
• Implementation Level
• Inspection technique X Testing techniques
• PBR Experiment
• Requirement Specification Level
• Inspection techniques
ISERN-2005
Readers: PBR studies
Readers Project
PBR teams
X
Non-PBR teams
ISERN-2005
PBR Studies: questions
Readers Project
1) Do teams applying PBR detect more defects than
teams applying the comparison technique?
2) Do individual reviewers using PBR find more
defects than those using the comparison
technique?
3) Does a reviewer’s experience affect her
effectiveness?
4) Does a reviewer, individually, find out different
defects applying PBR and non-PBR techniques?
5) Do the PBR perspectives have the same
effectiveness and efficiency?
6) Do PBR perspectives find different defects?
ISERN-2005
Replications: difficulties
Readers Project
• Difficulty to assemble a complete and consistent lab
package for the replication;
• Version control and configuration problems;
• Replications had different cultural settings;
• Language, application domain, …
• Training;
• Meta-analysis: data collection, …
• Pilot study
• To understand lab package and how the
experimental process should be conducted.
ISERN-2005
Readers Project
Replications: Lab Package Evolution
• Criteria for artifact (lab package) evolution???
• Lists of
• defects
• false positives
• non-defects (should be elaborated?)
• Updated Forms
• Defect Collection (additional fields)
• Annalist Survey
• Feedback
• Training material
• Steps
• High x low level
• PBR underlying techniques
•
Error guessing
• Equivalence partitioning
• Smaller Requirement Document
ISERN-2005
Experimental Knowledge Sharing Model
Readers Project
SHULL, F.; MENDONÇA, M.; BASILI, V.; CARVER, J.; MALDONADO, J.; FABBRI, S.;
TRAVASSOS, G.; OLIVEIRA, M. Knowledge-Sharing Issues in Experimental Software
Engineering. Empirical Software Engineering An International Journal. Kluwer Academic
Publishers, v.9, n.1-2, p.111 - 137, 2004.
ISERN-2005
Replications: collaboration mechanisms
Readers Project
• Knowledge Transfer Process
• Series of team’s workshops (6 months interval)
• e-mail and Web Portals
• Knowledge Repository
• e-Workshops and Local Workshops
>>>>supporting the steps of EKSM
ISERN-2005
Experimental Improvement Paradigm
Readers Project
MENDONÇA, M., MALDONADO, J. C., OLIVEIRA, M. C. F., FABBRI, S., SHULL, F., CARVER, J.;
BASILI, V. and TRAVASSOS, G. H. The Experimentation Improvement Paradigm for
Software Engineering. (to be submitted).
ISERN-2005
Readers Project
Lab Package Quality Goals
• Access to experimental artifacts
• Adequate and complete training materials
• Accurate estimates for the time required for
subjects to execute the experiment
• Presence of oracles and golden artifacts
• Ease of package evolution and experimental
feedback
• Clear description of communications channels
by which the original experimenters and, If
possible, related experimental community can
be contacted
ISERN-2005
Readers Project
Lab Package Quality Goals
• Complete descriptions of the
• analysis and goal of the experiment
• experimental design
• Including threats to validity
• context(s) in which the experiment was run
• process to run the experiment
• Process conformance
• Tacit knowledge
• Collaboration structure
ISERN-2005
Readers Project
Knowledge Sharing in a Technology Transfer
to Industry using a Lab Package
• Lab Package adaptation
• Application domain and techniques training
• Selection/elaboration of specific domain artifacts
• Running and evaluating pilot study;
• Evaluation and adjustment of the lab package in the specific
context
• Running another pilot study to valuate the adapted lab
package.
• Replicator training
• Study lab package,
EKSM in a
Technology
Transfer to
Industry
experimentation
process
• Pilot study:
observation, running
and evaluation
• Disseminate into the
organization
ISERN-2005
Readers Project
Replications: Main Agents
Replicator:
• What are the roles of the Replicator?
• What would be the “conditions”
(requirements) to be satisfied?
• Should the Replicator be trained?
• Has s/he been a subject?
• Has s/he been a observer of a replication?
• Should s/he be evaluated against the concepts of
experimental studies and experimental projects?
• ???
Replications:
Main Agents
ISERN-2005
Readers Project
Replications: Main Agents
Trainer:
• What would be the “requirements” to be a
trainer?
• Should the trainer be trained?
• Has s/he been a subject?
• Has s/he been a replication observer?
• Should s/he be evaluated against the techniques
involved (independent variables)?
• ??? Others..
Replications:
Main Agents
ISERN-2005
Readers Project
Replications: Main Agents
Data Analyzer:
• What would be the requirements to be a
Data Analyzer?
• Background on statistical techniques?
X
• Background on experimentation and
the techniques under study?
Replications:
Main Agents
ISERN-2005
Secondary studies
Systematic
Review
Secondary Study:
observations on PBR studies
Systematic Review
• 19 experimental studies about PBR found.
• 14 studies about PBR effectiveness against
(Checklist or ad-hoc) effectiveness
• 57% - PBR is more effective.
• 29% - not succeed to indicate PBR effectiveness
superiority.
• 14% - not obtain expressive difference.
14%
57%
29%
PBR superior
PBR not superior
no difference
MAFRA, S. N.; TRAVASSOS, G. H. Técnicas de Leitura de Software: Uma Revisão
Sistemática. In Proceedings of 19th Brazilian Symposium on Software
Engineering. P. 72-87. October, 2005.
Secondary Study:
observations on PBR studies
ISERN-2005
Secondary studies
Systematic
Review
Systematic Review
• Controversial Points:
•
•
•
•
PBR process conformance
Critical defects
PBR perspectives overlaying
Reviewer’s expertise X effectiveness
MAFRA, S. N.; TRAVASSOS, G. H. Técnicas de Leitura de Software: Uma Revisão
Sistemática. In Proceedings of 19th Brazilian Symposium on Software Engineering.
P. 72-87. October, 2005.
Competence Center Portals
ISERN-2005
Competence Center
Portals
• Highly connected clusters (“communities”)
[close-knit circles of “competences” (friends)]
linked
by a few weak ties between acquaintances belonging
to different “circles” of “competences”
•
•
•
•
•
•
Experimentation Process
VV&T Competence
Formal Methods
Software Process
Open Source Software
….
Competence Centers Portal
ISERN-2005
Competence Centers
Portal
• VV&T Competence Center Portal
• Lessons Learned
• Experiments
•
•
•
•
•
•
•
•
Lab packages
Experiment Types
Visual Data Mining
Distributed Experiments
Open Source Tools
Open Source Training Material
ESE maturity process
Open
ISERN-2005
Competence Centers
Portal
Competence Centers Portal
• VV&T Competence Center Portal
• Lessons Learned
• Experiments
•
•
•
•
Lab packages
Experiment Types
Visual Data Mining
Distributed Experiments
• Open Source Tools
• Open Source Training Material
• ESE maturity process
• Open Source Software Competence Center
ISERN-2005
Experiment Types
Competence Centers
Portal
Experiment
Types
TRAVASSOS, G.H.; BARROS, M.O. Contributions of In virtuo and In Silico
Experiments for the future of Empirical Studies in Software Engineering.
Proceedings of the ESEIW 2003 Workshop on Empirical Studies in Software
Engineering - WSESE 2003 Workshop Series on Empirical Studies in Software
Engineering (discussed at the ISERN 2003)
ISERN-2005
Competence Centers
Portal
Competence Centers Portal
• VV&T Competence Center Portal
• Lessons Learned
• Experiments
•
•
•
•
Lab packages
Experiment Types
Visual Data Mining
Distributed Experiments
• Open Source Tools
• Open Source Training Material
• ESE maturity process
• Open Source Software Competence Center
ISERN-2005
Competence Centers
Portal
Visual Data
Mining based
process
Visual Data Mining
ISERN-2005
Competence Centers
Portal
Competence Centers Portal
• VV&T Competence Center Portal
• Lessons Learned
• Experiments
•
•
•
•
Lab packages
Experiment Types
Visual Data Mining
Distributed Experiments
• Open Source Tools
• Open Source Training Material
• ESE maturity process
• Open Source Software Competence Center
ISERN-2005
Competence Centers
Portal
Competence Centers Portal
• VV&T Competence Center Portal
• Lessons Learned
• Experiments
•
•
•
•
Lab packages
Experiment Types
Visual Data Mining
Distributed Experiments
• Open Source Tools
• Open Source Training Material
• ESE maturity process
• Open Source Software Competence Center
ESE maturity process
ISERN-2005
Competence Centers
Portal
ESE maturity
process
ISO 12207,
generic
experimentation
process
Packaging
process
Types of
Experiments,
experimentation
cycle, instruments,
techniques, tools
Definition
Standard Experimentation Process (SEP)
Specialization by type of experiment (In
Virtuo, In Vivo, In Vitro, In Silico)
SEP
Specialized 1
SEP
Specialized 2
...
SEP
Specialized N
Instantiation by experiment
Experiment’s
features
Process of the
Experiment 1
Process of the
Experiment 2
...
Process of the
Experiment N
Defining experimentation processes for eSEE using TABA Workstation infrastructure (see ISESE 2005 late breaking paper session)
www.cos.ufrj.br/~ese
ISERN-2005
Competence Centers
Portal
Competence Centers Portal
• VV&T Competence Center Portal
• Lessons Learned
• Experiments
•
•
•
•
Lab packages
Experiment Types
Visual Data Mining
Distributed Experiments
• Open Source Tools
• Open Source Training Material
• ESE maturity process
• Open Source Software Competence Center
ISERN-2005
Competence Centers
Portal
Qualipso Proposal
ISERN-2005
Readers Project
Industry and Academy cooperation
Cooperation and integration between industry and
academy in the technology development and
evaluation are fundamental.
Industry
Experimental
Studies
Industry and
Academy
Cooperation
Academy
Real
Projects
Experimental
Studies
Knowledge Base
(ISERN, CeBASE, …)
HÖHN, E. N.; MALDONADO, J. C.; MENDONÇA, M.; FABBRI, S. VILLAS BOAS, A. L.;
TAMBASCIA, C.; FREITAS, M. E.; PAGLIUSO, P. PBR: Transferência de Tecnologia Baseada
em Pacotes de Experimentação. In Proceedings of SBQS. 2004.
ISERN-2005
Quality depends on the
ESE Process Maturity Level
• ()1
• ()2
• ()3
• ()4
• ()5
ISERN-2005
ESE Process Maturity Level??
• ()1
• ()2
• ()3
• ()4
• ()5

FIM