Transcript Slide 1

MetriCon 2.0
Correlating Automated Static Analysis Alert
Density to Reported Vulnerabilities in
Sendmail
Michael Gegick, Laurie Williams
North Carolina State University
7 August 2007
Introducing Security Parallels
Component – any logical part of the software system [1]
Reliability context
(well-established)
Security context
(new)
Fault-prone component
Likely to contain faults
Vulnerability-prone component
Likely to contain vulnerabilities
Failure-prone component
Likely to have failures in field
Attack-prone component
Likely to be exploited in the field
Make informed risk management decisions and prioritize
redesign, inspection, and testing efforts on components.
[1] IEEE, "ANSI/IEEE Standard Glossary of Software Engineering Terminology (IEEE Std 610.12-1990)," Los Alamitos, CA: IEEE Computer Society Press, 1990.
2
Early Reliability Metrics
•
Static analysis
– N. Nagappan and T. Ball, "Static Analysis Tools as Early Indicators of Pre-release
Defect Density," in International Conference on Software Engineering, St. Louis, MO,
2005, pp. 580-586.
– J. Zheng, L. Williams, W. Snipes, N. Nagappan, J. Hudepohl, and M. Vouk, "On the
Value of Static Analysis Tools for Fault Detection," IEEE Transactions on Software
Engineering, vol. 32, pp. 240-253, 2006.
•
Complexity metrics
– J. Munson and T. Khoshgoftaar, "The Detection of Fault-Prone Programs," IEEE
Transactions on Software Engineering, vol. 18, pp. 423-433, 1992.
– T. Khoshgoftaar and J. Munson, "Predicting Software Development Errors using Software
Complexity Metrics," IEEE Journal on Selected Areas in Communications, vol. 8, pp. 253261, 1990.
•
Historical (failure)
– N. Nagappan, T. Ball, and A. Zeller, "Mining metrics to predict component failures," in
International Conference on Software Engineering, Shanghai, China, 2006.
– T. J. Ostrand, E. J. Weyuker, and R. M. Bell, "Where the bugs are," in International
Symposium on Software Testing and Analysis, Boston, Massachusetts, 2004, pp. 86-96
•
Object-Oriented metrics
– V. Basili, L. Briand, and W. Melo, "A Validation of Object Oriented Design Metrics as Quality
Indicators," IEEE Transactions on Software Engineering, vol. 21, 1996.
– Y. Zhou and L. Hareton, "Empirical Analysis of Object-Oriented Design Metrics for
Predicting High and Low Severity Faults," IEEE Transactions on Software Engineering, vol.
32, no. 10, 2006, pp. 771-789.
3
Research Objective
Build and validate models for predicting vulnerabilityand attack-prone components based upon securitybased automated static analyzer (ASA) alerts
– Metric: ASA alert density and severity – early in the development phase
– ASA cannot find all types of security vulnerabilities
• Are ASA alerts a good predictor?
– Implementation bugs, design flaws, operational vulnerabilities
– Software engineers plug the number of security alerts into the predictive
models to determine which components are vulnerability- and attackprone.
4
Building the Initial Predictive Model
log( yˆ )  ˆ0  ˆ1 x1
Generalized linear model (data are not normally distributed)
Poisson distribution?
ˆ
y
mean number vulnerabilities in component
ˆ0
estimated intercept
ˆ1
estimated slope
x1
value of random variable – alert density of component
5
Feasibility Study
• Fortify Software’s Source Code Analyzer
(SCA)
• Scanned ten releases of Sendmail
– 8.12.2-8.12.11
– 996 total files scanned
• 21 potential vulnerabilities
– Vulnerabilities reported in RELEASE_NOTES
• Nine vulnerabilities with known exploits
6
Feasibility Study – vulnerability-prone
• Poisson distribution
– Models the response data
• Reported vulnerability
• Association between Hot alert density and number of
vulnerabilities per reported per file
– Positive slope  positive association between alerts and reported
vulnerabilities
– p-value  high significance in association
• Standard error  substantial overdispersion
– Few data points
log( yˆ )  ˆ0  ˆ1 x1
Slope
p-value
Chi-Square
/df
Goodnessoffit measure
Standard
error
294.8069
0.0016
1.1939
93.3422
7
Feasibility Study – attack-prone
• Poisson distribution
– Models the response data
• Number of known exploits (nine) for a Sendmail file
• Association between Hot alert density and number
of known exploits
– Slope  positive association between alerts and exploits
• p-value  low significance
– Standard error  substantial overdispersion
» Few data points
log( yˆ )  ˆ0  ˆ1 x1
Slope
p-value
Chi-Square
/df
Goodnessoffit measure
Standard
error
140.4334
0.4980
1.2099
207.2419
8
Questions
Thank you!
[email protected]
9