The Community for Technology Leaders
RSS Icon
Subscribe
Issue No.06 - Nov.-Dec. (2012 vol.9)
pp: 825-837
Hannes Holm , The Royal Institute of Technology, Stockholm
Mathias Ekstedt , The Royal Institute of Technology, Stockholm
Dennis Andersson , Swedish Defence Research Agency, Linköping
ABSTRACT
The Common Vulnerability Scoring System (CVSS) is a widely used and well-established standard for classifying the severity of security vulnerabilities. For instance, all vulnerabilities in the US National Vulnerability Database (NVD) are scored according to this method. As computer systems typically have multiple vulnerabilities, it is often desirable to aggregate the score of individual vulnerabilities to a system level. Several such metrics have been proposed, but their quality has not been studied. This paper presents a statistical analysis of how 18 security estimation metrics based on CVSS data correlate with the time-to-compromise of 34 successful attacks. The empirical data originates from an international cyber defense exercise involving over 100 participants and were collected by studying network traffic logs, attacker logs, observer logs, and network vulnerabilities. The results suggest that security modeling with CVSS data alone does not accurately portray the time-to-compromise of a system. However, results also show that metrics employing more CVSS data are more correlated with time-to-compromise. As a consequence, models that only use the weakest link (most severe vulnerability) to compose a metric are less promising than those that consider all vulnerabilities.
INDEX TERMS
Network security, Mathematical model, Authorization, Computer crime, Computational modeling, Telecommunication network management, Risk management, network management, Network-level security and protection, unauthorized access (hacking, phreaking), risk management
CITATION
Hannes Holm, Mathias Ekstedt, Dennis Andersson, "Empirical Analysis of System-Level Vulnerability Metrics through Actual Attacks", IEEE Transactions on Dependable and Secure Computing, vol.9, no. 6, pp. 825-837, Nov.-Dec. 2012, doi:10.1109/TDSC.2012.66
REFERENCES
[1] P. Manadhata and J. Wing, "An Attack Surface Metric," IEEE Trans. Software Eng., vol. 37, no. 3, pp. 371-386, May/June 2011.
[2] P. Mell, K. Scarfone, and S. Romanosky, "A Complete Guide to the Common Vulnerability Scoring System (CVSS)," Version 2.0, Forum of Incident Response and Security Teams, 2007.
[3] NVD, "US National Vulnerability Database," http:/nvd.nist. gov/, Feb. 2011.
[4] H. Holm, T. Sommestad, J. Almroth, and M. Persson, "A Quantitative Evaluation of Vulnerability Scanning," Information Management and Computer Security, vol. 19, no. 4, pp. 231-247, 2011.
[5] Y. Lai and P. Hsia, "Using the Vulnerability Information of Computer Systems to Improve the Network Security," Computer Comm., vol. 30, no. 9, pp. 2032-2047, 2007.
[6] M. Tupper and A. Zincir-Heywood, "VEA-Bility Security Metric: A Network Security Analysis Tool," Proc. Third Int'l Conf. Availability, Reliability and Security (ARES '08), pp. 950-957, 2008.
[7] M. McQueen, W. Boyer, M. Flynn, and G. Beitel, "Time-to-Compromise Model for Cyber Risk Reduction Estimation," Proc. Quality of Protection Workshop, pp. 49-64, 2006.
[8] E. Jonsson and T. Olovsson, "A Quantitative Model of the Security Intrusion Process Based on Attacker Behavior," IEEE Trans. Software Eng., vol. 23, no. 4, pp. 235-245, Apr. 1997.
[9] C. Wang and W. Wulf, "A Framework for Security Measurement," Proc. Nat'l Information Systems Security Conf., pp. 522-533, 1997.
[10] R. Ortalo, Y. Deswarte, and M. Kaâniche, "Experimenting with Quantitative Evaluation Tools for Monitoring Operational Security," IEEE Trans. Software Eng., vol. 25, no. 5, pp. 633-650, Sept./Oct. 2002.
[11] W. Boyer and M. Mcqueen, "Ideal Based Cyber Security Technical Metrics for Control Systems," Proc. Second Int'l Conf. Critical Information Infrastructures Security, pp. 246-260, 2008.
[12] S. Kamara, S. Fahmy, E. Schultz, F. Kerschbaum, and M. Frantzen, "Analysis of Vulnerabilities in Internet Firewalls," Computers and Security, vol. 22, no. 3, pp. 214-232, 2003.
[13] M. Hammervik, D. Andersson, and J. Hallberg, "Capturing a Cyber Defence Exercise," Proc. Symp. Technology and Methodology for Security and Crisis Management, p. 36, 2010.
[14] K. Geers, "Live Fire Exercise: Preparing for Cyber War," J. Homeland Security and Emergency Management, vol. 7, pp. 1547-7355, 2010.
[15] V. Verendel, "Quantified Security is a Weak Hypothesis: A Critical Survey of Results and Assumptions," Proc. Workshop New Security Paradigms Workshop, pp. 37-50, 2009.
[16] J. Voas, A. Ghosh, G. McGraw, F. Charron, and K. Miller, "Defining an Adaptive Software Security Metric from a Dynamic Software Failure Tolerance Measure," Proc. 11th Ann. Conf. Computer Assurance, 'Systems Integrity. Software Safety. Process Security' (COMPASS '96), pp. 250-263, 1996.
[17] R. Breu, F. Innerhofer-Oberperfler, and A. Yautsiukhin, "Quantitative Assessment of Enterprise Security System," Proc. Third Int'l Conf. Availability, Reliability and Security (ARES '08), pp. 921-928, 2008.
[18] M. Ahmed, E. Al-Shaer, and L. Khan, "A Novel Quantitative Approach for Measuring Network Security," Proc. IEEE INFOCOM, pp. 1957-1965, 2008.
[19] S. Hariri, G. Qu, T. Dharmagadda, M. Ramkishore, and C. Raghavendra, "Impact Analysis of Faults and Attacks in Large-Scale Networks," IEEE Security and Privacy, vol. 1, no. 5, pp. 49-54, Sept./Oct. 2003.
[20] D. Leversage and E. Byres, "Estimating a System's Mean Time-to-Compromise," IEEE Security and Privacy, vol. 6, no. 1, pp. 52-60, Jan./Feb. 2008.
[21] M. Sasse, S. Brostoff, and D. Weirich, "Transforming the "'Weakest Link"'a Human/Computer Interaction Approach to Usable and Effective Security," BT Technology J., vol. 19, no. 3, pp. 122-131, 2001.
[22] J. Pamula, S. Jajodia, P. Ammann, and V. Swarup, "A Weakest-Adversary Security Metric for Network Configuration Security Analysis," Proc. Second ACM Workshop Quality of Protection, pp. 31-38, 2006.
[23] R. Bohme and T. Moore, "The Iterated Weakest Link," IEEE Security and Privacy, vol. 8, no. 1, pp. 53-55, Jan.-Feb. 2010.
[24] J. Viega, G. McGraw, and G. McGraw, Building Secure Software: How to Avoid Security Problems the Right Way. Addison-Wesley Reading, 2002.
[25] I. Arce, "The Weakest Link Revisited [Information Security]," IEEE Security and Privacy, vol. 1, no. 2, pp. 72-76, Mar./Apr. 2003.
[26] B. Schneier, Secrets and Lies: Digital Security in a Networked World, John Wiley & Sons, Inc., 2000.
[27] S. Frei, M. May, U. Fiedler, and B. Plattner, "Large-Scale Vulnerability Analysis," Proc. SIGCOMM, pp. 131-138, 2006.
[28] W. Arbaugh, W. Fithen, and J. McHugh, "Windows of Vulnerability: A Case Study Analysis," Computer, vol. 33, no. 12, pp. 52-59, 2002.
[29] O. Alhazmi and Y. Malaiya, "Modeling the Vulnerability Discovery Process," Proc. IEEE 16th Int'l Symp. Software Reliability Eng. (ISSRE '05), pp. 10-138, 2005.
[30] P. Manadhata, J. Wing, M. Flynn, and M. McQueen, "Measuring the Attack Surfaces of Two FTP Daemons," Proc. Second ACM Workshop Quality of Protection, pp. 3-10, 2006.
[31] Y. Shin and L. Williams, "Is Complexity Really the Enemy of Software Security?" Proc. Fourth ACM Workshop Quality of Protection, pp. 47-50, 2008.
[32] U. Premaratne, J. Samarabandu, T. Sidhu, R. Beresh, and J. Tan, "Security Analysis and Auditing of IEC61850-Based Automated Substations," IEEE Trans. Power Delivery, vol. 25, no. 4, pp. 2346-2355, Oct. 2010.
[33] Rapid7, "Metasploit," http:/metasploit.com/, Dec. 2011.
[34] Sourcefire, "Snort," http:/www.snort.org/, Feb. 2011.
[35] T.N. Security, "Nessus," http:/www.nessus.org/, Feb. 2011.
[36] S. Furnell, P. Chiliarchaki, and P. Dowland, "Security Analysers: Administrator Assistants or Hacker Helpers?" Information Management and Computer Security, vol. 9, no. 2, pp. 93-101, 2001.
[37] IBM, "SPSS," http://www-01.ibm.com/software/analytics spss/, Dec. 2011.
[38] R. Warner, Applied Statistics: From Bivariate through Multivariate Techniques. Sage Publications, Inc, 2008.
[39] K. Scarfone and P. Mell, "An Analysis of CVSS Version 2 Vulnerability Scoring," Proc. Third Int'l Symp. Empirical Software Eng. and Measurement (ESEM '09), pp. 516-525, 2009.
[40] G. Reid and P. Mell, "Cvss-Sig Version 2 History," http://www.first.org/cvsshistory.html, June 2007.
[41] J. Wang, F. Zhang, and M. Xia, "Temporal Metrics for Software Vulnerabilities," Proc. Fourth Ann. Workshop Cyber Security and Information Intelligence Research: Developing Strategies to Meet the Cyber Security and Information Intelligence Challenges Ahead, pp. 1-3, 2008.
[42] C. Fruhwirth and T. Mannisto, "Improving CVSS-Based Vulnerability Prioritization and Response with Context Information," Proc. Third Int'l Symp. Empirical Software Eng. and Measurement, pp. 535-544, 2009.
[43] L. Gallon, "On the Impact of Environmental Metrics on CVSS Scores," Proc. IEEE Second Int'l Conf. Social Computing (SocialCom), pp. 987-992, 2010.
[44] S. Houmb and V. Franqueira, "Estimating ToE Risk Level Using CVSS," Proc. Int'l Conf. Availability, Reliability and Security, pp. 718-725, 2009.
[45] B. Schroeder and G. Gibson, "A Large-Scale Study of Failures in High-Performance Computing Systems," IEEE Trans. Dependable and Secure Computing, vol. 7, no. 4, pp. 337-351, Oct.-Dec. 2010.
[46] J. Rodgers and W. Nicewander, "Thirteen Ways to Look at the Correlation Coefficient," Am. Statistician, vol. 42, pp. 59-66, 1988.
66 ms
(Ver 2.0)

Marketing Automation Platform Marketing Automation Tool