Transcript presentation ( -- 776 KB)

The Impact of Software Reuse and
Incremental Development on the Quality
of Large Systems
Parastoo Mohagheghi,
Dept. Computer and Information Science (IDI),
University of Science and Technology (NTNU),
Trondheim
[email protected]
1
Parastoo Mohagheghi
07 September 2004
What is this presentation about?
• It presents results of a series of empirical studies
at Ericsson in Grimstad in 2001-2004.
• The studies are performed during a doctoral work
in the INCO project and supervised by Prof.
Reidar Conradi.
• 11 students were involved in different studies, an
example of university-industry cooperation.
• For more details, see the PhD thesis at:
http://www.idi.ntnu.no/grupper/su/publ/phd/moha
gheghi-thesis-10jul04-final.pdf
2
Parastoo Mohagheghi
07 September 2004
Motivation
• Systematic software reuse, product family
development and incremental development seem
be potent technologies to achieve benefits in
productivity, quality and maintainability, and to
reduce risks of changes.
Quantifiable benefits,
but few empirical studies in industry
that can verify these benefits
or possible other impacts.
3
Parastoo Mohagheghi
07 September 2004
Company background
• Ericsson has developed two large-scale telecom
systems (470 KSLOC or 1000 KSLOC in
equivalent C each) that share
–
–
–
–
Software architecture,
Components and a component model,
Development environment,
Software process.
• Systems are developed incrementally, and lots of
data is gathered during 5 years on defects,
changes, characteristics etc.
4
Parastoo Mohagheghi
07 September 2004
Overview of the Ericsson packetswitched core network- GPRS
MSC/
VLR
SMS-GMSC
HLR SMS-IWMSC
EIR
Other
networks
IP
network
SGSN
Backbone
network
GGSN
BSC/
RNC
Packet-switched
core network
Other
SGSN
BGW
5
Parastoo Mohagheghi
07 September 2004
Evolution of the GSN software architecture
and the software process model
The original architecture
The hierarchical architecture
Rev. &
Re-eng.
GPRS for WCDMA
GPRS for GSM
3’ GPRS
4 for GSM
1’
GPRS for GSM
1
4
2
5
3
6’
6
Wireless Packet Platform (WPP)
5’
2
8
7
Application-specific
layer
Business-specific
layer
Common services Reused
Layer (including
component framework)
Wireless Packet Platform (WPP)
Adaptation of RUP
Initial Process
6
Parastoo Mohagheghi
Adapted RUP
07 September 2004
The GSN RUP
7
Parastoo Mohagheghi
07 September 2004
Research questions
• RQ1. Why a reuse program is initiated and how is
it implemented?
• RQ2. What is the impact of software reuse,
Component-Based Development (CBD) and
incremental development on the quality? The
impact of development approaches on product
quality metrics and on project attributes such as
schedule or effort are sought.
• RQ3. How to improve the practice of incremental
development of product families in some aspects?
8
Parastoo Mohagheghi
07 September 2004
Research methods
• Quantitative studies by exploring (‘data mining’) company
databases and assessing hypotheses.
• Quantitative results are discussed with experienced
personnel, and combined with qualitative feedbacks and
studies of the process and practice to assess validity of the
results.
• Several case studies, a small survey and an experiment.
• Combining results of several studies in the interpretation
phase:
– The impact of introducing reuse or incremental development is
widespread.
– Taking benefit of all available data,
– Confirming results by other studies/data.
9
Parastoo Mohagheghi
07 September 2004
Qualitative study of RUP in the context of
reuse
• The approach to initiating a product family was an
extractive one.
• Software architecture has evolved to support reuse, while
GSN RUP has not been adapted for reuse to the same
degree.
• Examples of proposals:
– “Domain analysis” and “Make vs. Reuse vs. Buy” decisions in the
inception phase.
– “Update reuse documentation” for reusable components.
– “Record reuse experience” in the conclusion phase.
10
Parastoo Mohagheghi
07 September 2004
Quantitative study of Trouble Reports
(TRs)
• For defects detected during integration testing, system
testing, or later in maintenance.
• For defects detected in previous releases.
• For all types of defects (software, hardware, toolbox, and
documentation).
• We have ‘data mined’ databases:
– 13000 TRs for several systems and releases were parsed, data was
extracted, and inserted in a SQL database by a C# program.
– Inconsistencies were resolved -> Data was not analyzed.
•
11
The company provided us excel sheets on software size
(software module level).
Parastoo Mohagheghi
07 September 2004
Hypotheses: Software Reuse and Quality
(Defect-density and Stability)
12
H01
Reused components have the same
defect-density as non-reused ones.
H02
There is no relation between number
of defects and component size for
all/reused/non-reused components.
Not-rejected
Not-rejected
Rejected
H03
There is no relation between defectdensity and component size for
all/reused/non-reused components.
H04
Reused and non-reused components
are equally modified.
Not-rejected
Not-rejected
Not-rejected
Rejected
Parastoo Mohagheghi
Rejected
07 September 2004
H1: Reuse and defect-Density, Release 3
and blocks
Defect-density of blocks
Mean
Median
Variance
#TRs/KLOC, Reused
1.32
0.76
1.70
#TRs/KLOC, Non-Reused
3.01
2.44
4.39
#TRs/MKLOC, Reused
3.50
1.78
21.26
#TRs/MKLOC, Non-Reused
5.69
3.73
21.76
•Reused components have lower defect density.
•Reused components have proportionally more
defects with Severity A (highest severity).
•Reused components have proportionally less defects
after delivery.
13
Parastoo Mohagheghi
07 September 2004
H2: Size vs. the number of Defects
180
160
R2 = 0,7698
140
Reused
R2 =0,7213
120
#TR
Non-reused Blocks
100
Linear (Non-reused Blocks)
80
Linear (Reused)
R2 = 0,5876
R2 =0,573
60
Poly. (Reused)
Poly. (Non-reused Blocks)
40
20
0
0
5000
10000
15000
20000
25000
30000
35000
40000
45000
LOC
14
Parastoo Mohagheghi
07 September 2004
H3: Size vs. Defect-Density
8
7
#TRs/KLOC
6
5
Reused
4
3
Nonreused
2
1
0
0
5
10
15
20
25
30
35
40
45
KLOC
15
Parastoo Mohagheghi
07 September 2004
Discussion
• Reuse benefits: Reused components are less defect-prone
and more stable.
• Why? Reused components are designed more thoroughly,
and changed with care.
– Confounding factors (conclusion validity):
• Programming language. Not applicable.
• Type of functionality. Not applicable for stability, but may be
for defect-density.
• Developers’ experience: Not applicable.
• Internal validity: Missing data, but not systematic.
• Construct validity: Are defect-density and stability
indicators of quality?
• External validity: At least inside the company for similar
systems, or in the domain.
16
Parastoo Mohagheghi
07 September 2004
Contributions
• Research:
– Empirical verification of reuse benefits. No other
studies on large industrial systems.
– Combined with other studies in the company to assess
development approaches and identifying metrics.
• Company:
– Evaluating the trouble reporting system.
– Evaluating the measurement program: Granularity of
component definition and data for incremental
development.
– Identifying defect-prone and unstable components.
17
Parastoo Mohagheghi
07 September 2004
Software evolution and incremental
development
• With incremental development, software changes:
– during each release (development)
– between successive releases (evolution)
– and after delivery (maintenance).
• Some changes are pre-planned (incremental development)
and some are not (iterative improvement, adaptive changes
etc.).
• The IEEE Standard 1219 on software maintenance defines
software maintenance as “The modification of a software
product after delivery to correct faults (i.e. corrective), to
improve performance or other attributes (i.e.
perfective/preventive), or to adapt the product to a
modified environment (adaptive)”.
18
Parastoo Mohagheghi
07 September 2004
Quantitative study of Change Requests
(CRs)
• Ericsson defines requirements for each release in an ARS
(Application Requirement Specification).
• Changes to ARS or previous deliveries (releases) are
handled by issuing Change Requests (CRs).
– CRs handle coarse-grained changes to add/delete functionality,
improve performance or improve design.
– CRs cover changes during developing each release as well as
between releases (in addition to new requirements stated in the
ARS).
• How this data can be used in our research?
– 169 CRs for 4 releases of one system were analyzed.
– Exploratory study: Origin (internal vs. external), acceptance rate
and classes of changes are studied.
19
Parastoo Mohagheghi
07 September 2004
Origin of CRs
Perfective/
Functional
No
40
Perfective/
Quality
Attributes
(QA)
74
Adaptive Preventive
35
30
Other
(cost)
8
•18 CRs have indicated two reasons.
•Most perfective CRs are to improve QA (also
preventive CRs improve quality).
•23 of 169 CRs are issued because of customer
demands. These and adaptive CRs have external
origin (34% of all CRs).
20
Parastoo Mohagheghi
07 September 2004
Acceptance Rate over Releases
100 %
80 %
60 %
Rej ec t ed CRs (%)
40 %
A c c ept ed CRs (%)
20 %
0%
Release 1 Release 2 Release 3 Release 4
Acceptance rate is increasing (59% in total); i.e. the
product is getting more change-prone.
21
Parastoo Mohagheghi
07 September 2004
Conclusions
• Most changes originate from the project
organization itself in order to improve quality and
enhance functionality. The share of the first group
is higher.
• The practice indicates that the iterative realization
and improvement of quality attributes has great
impact on the evolution of the products.
• We could not observe any significant difference
between reused and non-reused components in the
number of CRs per KLOC.
22
Parastoo Mohagheghi
07 September 2004
Conclusions (continued)
• Most CRs are accepted and the acceptance rate
can have impact on the project plans in terms of
decreasing planning precision (observed).
Incremental development opens for (more)
changes.
• Most CRs are issued pre-delivery, and especially
in the short time right after requirement baseline.
The organization possibly takes the decision to
baseline requirements too early.
23
Parastoo Mohagheghi
07 September 2004
Overview of all studies
Phase 1
P1
P7
Phase 2
C1
P8 Study of
Trouble Reports
2003
Study of
Reuse Practice
2002
C6b
Experiment on
Inspection
P4
2002
Combining results in Phase 3
P11 Developing Data
Mining Method
2004
C1
C2
P10
Study of
Change Requests
2003
P2 P9
P6
Survey
2002
Study of Software
Process & RUP
2001-2002
March 2001
24
P12
C3
Parastoo Mohagheghi
P11 Assessing the Impact of
Dev. Approaches &
Identifying Metrics
2003-2004
Study of
Effort
2003-2004
Developing Effort C3
Estimation Method
2003-2004
P13
Prototype
Study of MDA
P3
2003
Quantitative study
Qualitative study
C4
P5
C6a
C5
July 2004
P
Paper
C
Contribution
Input
07 September 2004
Impact of dev. approaches and context
Reuse
Software
evolution
25
Incr. dev.
CBD
Large
systems
Change rate
Earlier rel.
not evolved
Granularity
Improved
incr.
Long-lived
Effort
estimation
ROI?
Software
No data for
from a
components
previous rel.,
Incr. changes
in
requirements
, Large CM
Large CM
and testing
effort
Metrics
yes
yes
Different
granularity
Parastoo Mohagheghi
yes
07 September 2004
Mining industrial databases
Theory or
Hypothesis
confirmatory
theoretical
Literature
Search
Research
Question
exploratory
Pre-Study
Data
Select/Define
Theory or Hyp.
Select
Methods &
tools
preparation
Sampling
yes/no
Explore
Data
Modify
Data
Model
Assess
execution
Package
Results
conclusion
Top-down theory search integrated with
bottom-up data exploring.
26
Parastoo Mohagheghi
07 September 2004
Assessing development approaches
Development Approaches
Practices
Product/process Quality
Requirement Modification +/-
Incremental &
Iterative
Development
Planning Precision
+/Incremental Delivery
Success
Solution Modification
Needed Effort
Software Reuse
Incremental Planning
Component Stability
+
Incremental Integration
Component-Based
Development
+
Reusable Artifacts/
Components
+
Reduced Defect-Density
Changeability
Leads to
Development approaches should be seen as coherent systems of practices,
be chosen depending on the attribute that should be optimized and
may trade-off for other practices [IEEE software, MacCormack, 2003 ]
27
Parastoo Mohagheghi
07 September 2004
Main Contributions
• Empirical verification of reuse benefits.
• Increased understanding of the origin and type of
changes in each release and between releases.
• Identifying metrics for a combination of reuse and
incremental development.
• A data mining method for exploring industrial
databases.
• An effort estimation method for incremental
development.
28
Parastoo Mohagheghi
07 September 2004
Discussion
• Studies are performed in a real context.
– Context-independent knowledge is rare in SE and generalization is
difficult.
• Studies combine different methods and data
(triangulation).
– Integration of the results is a challenge: physical and conceptual
integration challenges.
• Incremental development is combined with reuse of
software in multiple products and CBD.
– All aspects of software development need adaptation; e.g.
estimation method, software process model, data collection
routines etc.
29
Parastoo Mohagheghi
07 September 2004
Discussion (continued)
• Own experience and access to internal data
– Confidentiality of some results and other ethical issues should be
respected.
• Some studies are evaluation of others’ observations or
methods in a new context (e.g. estimation method, reuse
benefits or distribution of maintenance categories), while
others contain new observations and methods.
• Being exposed to organizational changes leaded to an
emerging research design and changes in plans.
• Working in the field needs flexibility and a suitably timeframe.
30
Parastoo Mohagheghi
07 September 2004
Why such studies are important for
industry?
• Methods should be tested for development in
large.
• Companies are increasingly using mainstream
methods, tools and programming languages.
• The studies analyzed data that was hardly studied
before: evaluation of data collection routines and
metrics for the company, in addition to concrete
results.
31
Parastoo Mohagheghi
07 September 2004
Acknowledgements
• Thanks to INCO, Simula and NTNU for financial
support.
• Thanks to my supervisor, Prof. Reidar Conradi,
and colleagues at INCO and IDI.
• Thanks to Ericsson for the permission to perform
studies and to publish the results.
• Thanks to the SPIKE seminar and the audience.
32
Parastoo Mohagheghi
07 September 2004