Transcript Predictor of Customer Perceived Software Quality

Predictor of Customer
Perceived Software Quality
By
Haroon Malik
Introduction
• Predict the customer’s experience within
first three months
• Quantifying the relative importance of
various processes and product factors on
customer experience
• Deployment issues
– Usage patterns,
– Software platforms &
– Hardware platform.
Introduction
• Come up with a model that can be easily
adapted and used at other organization
with little or more tailoring.
Driving force
• Techniques already exists to predict
how many faults remains in
unchanging software system, changes
or module will have defect and even
how much effort defect repairs will
require.
• Many researches examined the effect
of software contents and development
process on measure of customer
perceived quality.
Driving force
• Most of which ignore the
–
–
–
–
Hardware configurations,
Software platform,
Usage patterns &
Deployment issues
• End users experience the software
typically experience the quality of the
entire “Solution”.
• Past researches do consider the
importance of these factors but no
solid work is done to validate the claim
or quantify these factors.
The Software project
• Call processing software for AVYA
telephony systems
– Established product
– Seven million lines of code mostly in C and C++
• Multiple releases are in field and are
being used by tens of thousands of
customers.
• Used by clients whose business depend
on the high availability of the product.
Capturing Interaction
• Four database are considered for
capturing customer interaction
measures
– Customer issue tracking system
• Trouble ticket database
– The equipment database
– Change management
• Sablime database
The Process
• Avaya uses a tiered support process.
• Trouble Ticket
• Half of the over 4 million tickets
created in 2003 are related to product
analyzed in this project.
• Equipment Database
– Software release
– Number of licensed ports
– 4 million systems listen in equipment
databases.
Customer perceived quality
models
Model
Model
• Business activity distraction ---Negative effect
• To major aspect of customer perceived
quality:
– Impact of problem occurrence
– Frequency of problem occurrence
• Interest
– Rare high-impact problems
• Equipment services outages
• Malfunction resulting in software
modification
Customer perceived quality
models
– Frequent low impact problems
Model
Model
• Technician dispatches
• Customer calls
• Alarm reports
Factors to predict
• Factors
–
–
–
–
lack empirical
Deployment issues
validation
Usage patterns
Software platform
Hardware configurations
• Prior work examines
– Software product
– Development process
They are not good predictors (Static.
do not vary for a single release)
Predictor Measures ?
• Value Predictors
– System size.
–
–
–
–
Operating system
Ports
Total Deployment time
Software Upgrades
• Nuisance Predictors
– US of international installation
– Service contracts
– Missing configuration information
System size
• This predictor measures
– Hardware configuration factors
– Software platform factors and
– Usage patterns factor.
• Introduced “LARGE” variable indicator in the
model
• Small and medium systems have:
–
–
–
–
Fewer customer interactions
Few settings to configure
Fewer systems to interface with
Likely not to be involved in business critical
application requiring 7x24 uptime.
– Less likely to experience and report issues.
Operating System
• Operating system predictor measures:
– Software platform factor &
– Hardware configuration factor
• Considered
– An open Linux
– proprietary
– Commercial windows
• Very small number Os systems used NT/Win2000
• Off-the-shelf operating system introduce
unnecessary complexity and configuration issue.
Ports
• The port predictor measures
– Usage pattern factor &
– Hardware configuration factors
• The number of ports indicates how
many licensed end points are
supported by the system.
• Model encoded the log number of
ports with log(nPort) variable
Total deployment time
I need it on
Timeeee!!!!
• Deployment time predictor
measure the deployment issues
• Anticipation: fewer customer
interaction as the total
deployment time increases.
Software upgrades
• The software upgrades predictor
measures the deployment issues
factor.
• Encoded the existence of upgrade using an
indictor variable called “Upgr”.
• Upgrade serves to keep the machine running
properly by incorporating the latest fixes and
refinements to the system
• Upgrades have clearly defined purpose of
making the system more stable, so expect to
have that effect.
Nuisance Factors
• These factors are likely to identify peculiarities
of data reporting and collection process, but not
necessarily differences in underlying customer
perceived quality.
– US of International installation
– Service contracts
– Missing configuration information
Factors Vs Predictors
• Predict for each
customer (outputs):
–
–
–
–
–
Software defects
System outages
Technician dispatches
Calls
Automated alarms
• Using Logistic
regression and Linear
regression
• Using predictors (inputs):
–
–
–
–
–
Total deployment time
Operating system
System size
Ports
Software upgrades
• For a real world software
system
Results
• First fit the models to test the
relationships hypothesized previously.
• Use the models to predict customer
interaction for the next major
releases.
• Present full results only for two
measures.
Software Failure
Nuisance
Factors
Most important predictor
• Total Deployment time:
– Customers who installed the application early
may have detected malfunctions that are
fixed by the time later customers install their
systems
– The individuals performing the installation
and configuration may have acquired more
experience, have access to improved
documentation
– The lesson from this relationship is that
customers that are less tolerant of availability
issues should not be the first to install a
major software release
“never upgrade to dot zero
release.”
Another Important Predictor
• Operating system (software
platform, hardware configurations)
– Systems running on the proprietary OS
are 3 times less likely to experience a
software defect compared with
systems on running the open OS
(Linux)
– Systems running on the commercial
OS (Windows) are 3 times more likely
to experience a software defect
compared with systems running on the
open OS (Linux)
Customer calls
Predicting customer call
traffic
• Ports and NAPorts are not available
• know from talking to the customer call center that they have estimates of
how many calls a call representative can handle so our prediction of the
number of calls in a month can be used to plan staffing.
• The take away is that they the predictions are accurate and can be used to
plan various activities
Validations
• Accounted for data reporting
differences
– Included indicator variables in the models
to identify populations (e.g. US or
international customers)
• Independently validated the data
collection process
– Independently extracted data and
performed analyses
• Interviewed personnel to validate
findings
– Programmers
– Field technicians
Best Contributions
• Identified and quantified characteristics, like
time of deployment, that can affect customer
perceived quality by more than an order of
magnitude
• created models that can predict various
customer interactions and found that
predictors have consistent effect across
interactions
• We learned that controlled deployment may
be the key for high reliability systems