A Survey on Systems Security Metrics

Abstract

Security metrics have received significant attention. However, they have not been systematically explored based on the understanding of attack-defense interactions, which are affected by various factors, including the degree of system vulnerabilities, the power of system defense mechanisms, attack (or threat) severity, and situations a system at risk faces. This survey particularly focuses on how a system security state can evolve as an outcome of cyber attack-defense interactions. This survey concerns how to measure system-level security by proposing a security metrics framework based on the following four sub-metrics: (1) metrics of system vulnerabilities, (2) metrics of defense power, (3) metrics of attack or threat severity, and (4) metrics of situations. To investigate the relationships among these four sub-metrics, we propose a hierarchical ontology with four sub-ontologies corresponding to the four sub-metrics and discuss how they are related to each other. Using the four sub-metrics, we discuss the state-of-art existing security metrics and their advantages and disadvantages (or limitations) to obtain lessons and insight in order to achieve an ideal goal in developing security metrics. Finally, we discuss open research questions in the security metrics research domain and we suggest key factors to enhance security metrics from a system security perspective.

References

1. M. Ahmed, E. Al-Shaer, and L. Khan. 2008. A novel quantitative approach for measuring network security. In IEEE INFOCOM’2008.
2. E. Al-Shaer, L. Khan, and M. Ahmed. 2008. A comprehensive objective network security metric framework for proactive security configuration. In Proc. CSIIRW’08. 42:1--42:3.
3. M. Albanese, S. Jajodia, and S. Noel. 2012. Time-efficient and cost-effective network hardening using attack graphs. In Proc. IEEE DSN’12. 1--12.
4. R. Albert and A. Barabasi. 2002. Statistical mechanics of complex networks. Rev. Mod. Phys. 74 (2002), 47--97.
5. P. Ammann, D. Wijesekera, and S. Kaushik. 2002. Scalable, graph-based network vulnerability analysis. In Proc. ACM CCS’02. 217--224.
6. S. Axelsson. 2009. The base-rate fallacy and its implications for the difficulty of intrusion detection. In Proc. ACM CCS’09. 1--7.
7. M. Backes and S. Nürnberger. 2014. Oxymoron: Making fine-grained memory randomization practical by allowing code sharing. In Proc. USENIX Security Symposium. 433--447.
8. S. Berinato. 2002. Finally, a Real Return on Security Spending. Retrieved from http://www.cio.com/article/2440999/metrics/finally--a-real-return-on-security-spending.html. (2002).
9. B. Biggio, G. Fumera, and F. Roli. 2014. Security evaluation of patternclassifiers under attack. IEEE Trans. Knowl. Data Eng. 26, 4 (2014), 984--996.
10. L. Bilge and T. Dumitras. 2012. Before we knew it: An empirical study of zero-day attacks in the real world. In Proc. ACM CCS’12. 833--844.
11. N. Boggs, S. Du, and S. Stolfo. 2014. Measuring drive-by download defense in depth. In Proc. RAID’14. 172--191.
12. N. Boggs and S. Stolfo. 2011. ALDR: A new metric for measuring effective layering of defenses. In Proc. Layered Assurance Workshop (LAW’11) (2011).
13. R. Böhme and F. Freiling. 2008. Dependability Metrics: Advanced Lectures, 7--13.
14. R. Böhme and T. Nowey. 2008. Dependability metrics. Chapter Economic Security Metrics, 176--187.
15. J. Bonneau. 2012a. The science of guessing: Analyzing an anonymized corpus of 70 million passwords. In Proc. IEEE Symposium on Security and Privacy. 538--552.
16. J. Bonneau. 2012b. Statistical metrics for individual password strength. In Proc. International Conference on Security Protocols. 76--86.
17. N. Burow, S. Carr, S. Brunthaler, M. Payer, J. Nash, P. Larsen, and M. Franz. 2016. Control-flow integrity: Precision, security, and performance. CoRR abs/1602.04056 (2016).
18. W. Burr, D. Dodson, and W. Polk. 2006. Electronic Authentication Guideline. NIST Publication 800-63 Version 1.0.2. Retrieved from http://csrc.nist.gov/publications/nistpubs/800-63/SP800-63V1_0_2.pdf.
19. A. Cardenas, J. Baras, and K. Seamon. 2006. A framework for the evaluation of intrusion detection systems. In Proc. IEEE 2006 Symposium on Security and Privacy.
20. L. Carin, G. Cybenko, and J. Hughes. 2008. Cybersecurity strategies: The QuERIES methodology. IEEE Comput. 41, 8 (2008), 20--26.
21. N. Carlini, A. Barresi, M. Payer, D. Wagner, and T. Gross. 2015. Control-flow bending: On the effectiveness of control-flow integrity. In 24th USENIX Security Symposium. 161--176.
22. N. Carlini and D. Wagner. 2014. ROP is still dangerous: Breaking modern defenses. In Proc. USENIX Security Symposium. 385--399.
23. X. De Carné De Carnavalet and M. Mannan. 2015. A large-scale evaluation of high-impact password strength meters. ACM TISSEC 18, 1 (May 2015), 1:1--1:32.
24. C. Castelluccia, M. Dürmuth, and D. Perito. 2012. Adaptive password-strength meters from markov models. In Proc. NDSS’12.
25. V. Chandola, A. Banerjee, and V. Kumar. 2009. Anomaly detection: A survey. ACM Comput. Surv. 41, 3 (2009), 15:1--15:58.
26. Z. Chen and C. Ji. 2007. Measuring network-aware worm spreading ability. In Proc. INFOCOM’2007. 116--124.
27. P. Cheng, L. Wang, S. Jajodia, and A. Singhal. 2012. Aggregating CVSS base scores for semantics-rich network security metrics. In Proc. IEEE SRDS’12. 31--40.
28. Y. Cheng, J. Deng, J. Li, S. DeLoach, A. Singhal, and X. Ou. 2014. Metrics of security. In Cyber Defense and Situational Awareness. Vol. 62.
29. E. Chew, M. Swanson, K. Stine, N. Bartol, A. Brown, and W. Robinson. 2008. NIST Special Publication 800-55 Revision 1: Performance Measurement Guide for Information Security.
30. J. Cho, H. Cam, and A. Oltramari. 2016. Effect of personality traits on trust and risk to phishing vulnerability: Modeling and analysis. In Proc. IEEE CogSIMA’16.
31. CIS. 2010. The CIS Security Metrics (ver 1.1.0). Retrieved from http://benchmarks.cisecurity.org/downloads/metrics/. (2010).
32. INFOSEC Research Council. 2007. Hard Problem List. Retrieved from http://www.infosec-research.org/ docs_public/20051130-IRC-HPL-FINAL.pdf. (2007).
33. G. Da, M. Xu, and S. Xu. 2014. A new approach to modeling and analyzing security of networked systems. In Proc. HotSoS’14.
34. M. Dacier, Y. Deswarte, and M. Kaâniche. 1996. Models and tools for quantitative assessment of operational security. In Proc. IFIP Security Conference. 177--186.
35. D. Dagon, G. Gu, C. Lee, and W. Lee. 2007. A taxonomy of botnet structures. In Proc. ACSAC’07. 325--339.
36. D. Dagon, C. Zou, and W. Lee. 2006. Modeling botnet propagation using time zones. In Proc. NDSS’06.
37. N. Dalvi, P. Domingos, Mausam, S. Sanghai, and D. Verma. 2004. Adversarial classification. In Proc. KDD’04. 99--108.
38. L. Davi, A. Sadeghi, D. Lehmann, and F. Monrose. 2014. Stitching the gadgets: On the ineffectiveness of coarse-grained control-flow integrity protection. In Proc. USENIX Security Symposium. 401--416.
39. M. DellAmico, P. Michiardi, and Y. Roudier. 2010. Password strength: An empirical analysis. In Proc. INFOCOM’10. 983--991.
40. Y. Desmedt and Y. Frankel. 1989. Threshold cryptosystems. In Proc. Crypto. 307--315.
41. Z. Durumeric, J. Kasten, D. Adrian, J. Halderman, M. Bailey, F. Li, N. Weaver, J. Amann, J. Beekman, M. Payer, and V. Paxson. 2014. The matter of heartbleed. In Proc. ACM IMC’14. 475--488.
42. B. Edwards, S. Hofmeyr, and S. Forrest. 2015. Hype and heavy tails: A closer look at data breaches. In Proc WEIS’15. 67--78.
43. T. Eskridge, M. Carvalho, E. Stoner, T. Toggweiler, and A. Granados. 2015. VINE: A cyber emulation environment for MTD experimentation. In Proc. ACM MTD’15. 43--47.
44. FIRST. 2015. Forum of Incident Response and Security Teams: Common Vulnerability Scoring System (CVSS) Version 3.0. Retrieved from https://www.first.org/cvss. (2015).
45. S. Frei and T. Kristensen. Feb. 2010. The Security Exposure of SOoftware Portfolios. Retrieved from https://secunia.com/gfx/pdf/Secunia_RSA_Software_Portfolio_Security_Exposure.pdf. (Feb. 2010).
46. M. Frigault and L. Wang. 2008. Measuring network security using bayesian network-based attack graphs. In Proc. IEEE CompSAC’08. 698--703.
47. M. Frigault, L. Wang, A. Singhal, and S. Jajodia. 2008. Measuring network security using dynamic bayesian network. In Proc. QoP’08. 23--30.
48. J. Gaffney Jr and J. Ulvila. 2001. Evaluation of intrusion detectors: A decision theory approach. In Proc. IEEE Symposium on Security and Privacy. 50--61.
49. E. Göktas, E. Athanasopoulos, H. Bos, and G. Portokalidis. 2014. Out of control: Overcoming control-flow integrity. In Proc. IEEE Security and Privacy. 575--589.
50. L. Gordon and M. Loeb. 2006. Budgeting process for information security expenditures. Commun. ACM 49, 1 (Jan. 2006), 121--125.
51. G. Gu, A. Cárdenas, and W. Lee. 2008. Principled reasoning and practical applications of alert fusion in intrusion detection systems. In Proc. ACM ASIACCS’08. 136--147.
52. G. Gu, P. Fogla, D. Dagon, W. Lee, and B. Skorić. 2006. Measuring intrusion detection capability: An information-theoretic approach. In Proc. AsiaCCS’06. 90--101.
53. Y. Han, W. Lu, and S. Xu. 2014. Characterizing the power of moving target defense via cyber epidemic dynamics. In Proc. HotSoS’14. 10:1--10:12.
54. S. Hardy, M. Crete-Nishihata, K. Kleemola, A. Senft, B. Sonne, G. Wiseman, P. Gill, and R. Deibert. 2014. Targeted threat index: Characterizing and quantifying politically-motivated targeted malware. In Proc. USENIX Security Symposium.
55. W. Herlands, T. Hobson, and P. Donovan. 2014. Effective entropy: Security-centric metric for memory randomization techniques. In Workshop on Cyber Security Experimentation and Test.
56. H. Holm. 2014. A large-scale study of the time required to compromise a computer system. IEEE TDSC 11, 1 (2014), 2--15.
57. J. Homer, S. Zhang, X. Ou, D. Schmidt, Y. Du, S. Rajagopalan, and A. Singhal. 2013. Aggregating vulnerability metrics in enterprise networks using attack graphs. J. Comput. Secur. 21, 4 (2013), 561--597.
58. A. Howe, I. Ray, M. Roberts, M. Urbanska, and Z. Byrne. 2012. The psychology of security for the home computer user. In IEEE Symp. on Security and Privacy. 209--223.
59. L. Huang, A. Joseph, B. Nelson, B. Rubinstein, and J. Tygar. 2011. Adversarial machine learning. In Proc. ACM AISec’11. 43--58.
60. N. Idika and B. Bhargava. 2012. Extending attack graph-based security metrics and aggregating their application. IEEE TDSC 9, 1 (Jan. 2012), 75--85.
61. W. Jansen. 2009. Directions in Security Metrics Research. Retrieved from http://csrc.nist.gov/publications/ nistir/ir7564/nistir-7564_metrics-research.pdf. (2009).
62. A. Jaquith. 2007. Security Metrics: Replacing Fear, Uncertainty, and Doubt. Addison-Wesley Professional.
63. S. Jha, O. Sheyner, and J. Wing. 2002. Two formal analys s of attack graphs. In Proc. IEEE CSF. 49--59.
64. B. Johnson, J. Chuang, J. Grossklags, and N. Christin. 2012. Metrics for measuring ISP badness: The case of spam. In Proc. FC’12. 89--97.
65. E. Jonsson and T. Olovsson. 1997. A quantitative model of the security intrusion process based on attacker behavior. IEEE Trans. SE 23, 4 (1997), 235--245.
66. P. Kelley, S. Komanduri, M. Mazurek, and R. Shay. 2012. Guess again (and again and again): Measuring password strength by simulating password-cracking algorithms. In Proc. IEEE Symposium on Security and Privacy. 523--537.
67. M. Konte, R. Perdisci, and N. Feamster. 2015. ASwatch: An as reputation system to expose bulletproof hosting ases. In Proc. ACM SIGCOMM’15. 625--638.
68. M. Kührer, C. Rossow, and T. Holz. 2014. Paint it black: Evaluating the effectiveness of malware blacklists. In Proc. RAID’14. 1--21.
69. B. Lampson. 2006. Practical Principles for Computer Security. (2006).
70. C. Landwehr, A. Bull, J. McDermott, and W. Choi. 1994. A taxonomy of computer program security flaws. ACM Comput. Surv. 26, 3 (1994), 211--254.
71. P. Larsen, A. Homescu, S. Brunthaler, and M. Franz. 2014. SoK: Automated software diversity. In Proc. 2014 IEEE Symposium on Security and Privacy. 276--291.
72. W. Lee, W. Fan, M. Miller, S. Stolfo, and E. Zadok. 2002. Toward cost-sensitive modeling for intrusion detection and response. J. Comput. Secur. 10, 1--2 (July 2002), 5--22.
73. E. LeMay, M. Ford, K. Keefe, W. Sanders, and C. Muehrcke. 2011. Model-based security metrics using adversary view security evaluation (ADVISE). In Proc. QEST’11. 191--200.
74. L. Levesque, N. Fanny, J. Fernandez, S. Chiasson, and A. Somayaji. 2013. A clinical study of risk factors related to malware infections. In Proc. ACM CCS’13. 97--108.
75. D. Levin. 2003. Lessons learned in using live red teams in IA experiments. In Proc. DISCEX-III. 110--119.
76. X. Li, P. Parker, and S. Xu. 2011. A stochastic model for quantitative security analysis of networked systems. IEEE TDSC 8, 1 (2011), 28--43.
77. R. Lippmann, J. Riordan, T. Yu, and K. Watson. 2012. Continuous Security Metrics for Prevalent Network Threats: Introduction and First Four Metrics. Technical Report IA-3. MIT Lincoln Laboratory. Retrieved from https://www.ll.mit.edu/mission/cybersec/publications/publication-files/full_papers/2012_05_22_Lippmann_TechReport_FP.pdf.
78. Y. Liu and H. Man. 2005. Network vulnerability assessment using Bayesian networks. In Data Mining, Intrusion Detection, Information Assurance, and Data Networks Security 2005, B. V. Dasarathy (Ed.), Vol. 5812. 61--71.
79. Y. Liu, A. Sarabi, J. Zhang, P. Naghizadeh, M. Karir, M. Bailey, and M. Liu. 2015. Cloudy with a chance of breach: Forecasting cyber security incidents. In USENIX Security Symposium. 1009--1024.
80. D. Lowd and C. Meek. 2005. Adversarial learning. In KDD’05. 641--647.
81. K. Lu, C. Song, B. Lee, S. Chung, T. Kim, and W. Lee. 2015. ASLR-guard: Stopping address space leakage for code reuse attacks. In Proc. ACM CCS’15. 280--291.
82. W. Lu, S. Xu, and X. Yi. 2013. Optimizing active cyber defense dynamics. In Proc. GameSec’13. 206--225.
83. B. Madan, K. Gogeva-Popstojanova, K. Vaidyanathan, and K. Trivedi. 2002. Modeling and quantification of security attributes of software systems. In Proc. DSN’02. 505--514.
84. P. Manadhata and J. Wing. 2011. An attack surface metric. IEEE Trans. Software Eng. 37, 3 (2011), 371--386.
85. W. Marczak, J. Scott-Railton, M. Marquis-Boire, and V. Paxson. 2014. When governments hack opponents: A look at actors and technology. In Proc. USENIX Security Symposium. 511--525.
86. J. McHugh. 2006. Quality of protection: Measuring the unmeasurable? In Proc QoP’06. 1--2.
87. G. Mezzour, K. Carley, and L. Carley. 2015. An empirical study of global malware encounters. In Proc. HotSoS’15. 8:1--8:11.
88. Microsoft. 2013-2014. Security Intelligence Report. Retrieved from http://www.microsoft.com/security/sir/ default.aspx. (2013-2014).
89. A. Milenkoski, M. Vieira, S. Kounev, A. Avritzer, and B. Payne. 2015. Evaluating computer intrusion detection systems: A survey of common practices. ACM Comput. Surv. 48, 1 (2015).
90. MITRE. 2014. Common Weakness Scoring System (CWSS version 1.0.1). Retrieved from https://cwe.mitre. org/cwss/cwss_v1.0.1.html. (2014).
91. A. Mohaisen and O. Alrawi. 2014. Av-meter: An evaluation of antivirus scans and labels. In Proc. DIMVA’2014. 112--131.
92. J. Morales, S. Xu, and R. Sandhu. 2012. Analyzing malware detection eciency with multiple anti-malware programs. ASE Sci. J. 1, 2 (2012), 56--66.
93. A. Nappa, R. Johnson, L. Bilge, J. Caballero, and T. Dimitras. 2015. The attack of the clones: A study of the impact of shared code on vulnerability patching. In Proc. IEEE Symposium on Security and Privacy.
94. K. Nayak, D. Marino, P. Efstathopoulos, and T. Dumitras. 2014. Some vulnerabilities are different than others. In Proc. RAID’14. 426--446.
95. A. Neupane, M. Rahman, N. Saxena, and L. Hirshfield. 2015. A multi-modal neuro-physiological study of phishing detection and malware warnings. In Proc. ACM CCS’15. 479--491.
96. D. Nicol, B. Sanders, J. Katz, B. Scherlis, T. Dumitra, L. Williams, and M. Singh. 2015. The Science of Security 5 Hard Problems. Retrieved from http://cps-vo.org/node/21590. (2015).
97. D. Nicol, W. Sanders, and K. Trivedi. 2004. Model-based evaluation: From dependability to security. IEEE TDSC 1, 1 (2004), 48--65.
98. B. Niu and G. Tan. 2015. Per-input control-flow integrity. In Proc. CCS’15. 914--926.
99. S. Noel and S. Jajodia. 2014. Metrics suite for network attack graph analytics. In Proc. CISR’14. 5--8.
100. R. Ortalo, Y. Deswarte, and M. Kaâniche. 1999. Experimenting with quantitative evaluation tools for monitoring operational security. IEEE Trans. Softw. Eng. 25, 5 (Sept. 1999), 633--650.
101. X. Ou and A. Singhal. 2011. Quantitative Security Risk Assessment of Enterprise Networks. Springer.
102. J. Pamula, S. Jajodia, P. Ammann, and V. Swarup. 2006. A weakest-adversary security metric for network configuration security analysis. In Proc. ACM QoP’06. 31--38.
103. S. Pfleeger and R. Cunningham. 2010. Why measuring security is hard. IEEE Secur. Priv. 8, 4 (July 2010), 46--54.
104. S. Pfleeger. 2009. Useful Cybersecurity Metrics. IT Profess. 11, 3 (2009), 38--45.
105. C. Phillips and L. Swiler. 1998. A graph-based system for network-vulnerability analysis. In Proc. NSPW’98. 71--79.
106. A. Prakash and M. Wellman. 2015. Empirical game-theoretic analysis for moving target defense. In Proc. ACM MTD’15. 57--65.
107. M. Rajab, F. Monrose, and A. Terzis. 2005. On the effectiveness of distributed worm monitoring. In Proc. USENIX Security Symposium.
108. M. Reiter and S. Stubblebine. 1999. Authentication metric analysis and design. ACM TISSEC 2, 2 (May 1999), 138--158.
109. R. Ritchey and P. Ammann. 2000. Using model checking to analyze network vulnerabilities. In Proc. IEEE Symposium on Security and Privacy. 156--165.
110. F. Roberts. 1979. Measurement Theory, with Applications to Decision Making, Utility and the Social Sciences. Addison-Wesley, Boston.
111. K. Roundy and B. Miller. 2013. Binary-code obfuscations in prevalent packer tools. ACM Comput. Surv. 46, 1 (July 2013), 4:1--4:32.
112. C. Sabottke, O. Suciu, and T. Dumitras. 2015. Vulnerability disclosure in the age of social media: Exploiting twitter for predicting real-world exploits. In Proc. USENIX Security Symposium. 1041--1056.
113. W. Sanders. 2014. Quantitative security metrics: Unattainable holy grail or a vital breakthrough within our reach? IEEE Secur. Priv. 12, 2 (2014), 67--69.
114. B. Schneier. 2000. Secrets 8 Lies: Digital Security in a Networked World. John Wiley 8 Sons, Inc.
115. National Science and Technology Council. 2011. Trustworthy Cyberspace: Strategic Plan for the Federal Cybersecurity Research and Development Program. Retrieved from https://www.nitrd.gov/SUBCOMMITTEE/csia/Fed_Cybersecurity_RD_Strategic_Plan_2011.pdf. (2011).
116. H. Shacham, M. Page, B. Pfaff, E. Goh, N. Modadugu, and D. Boneh. 2004. On the effectiveness of address-space randomization. In Proc. ACM CCS’04.
117. S. Sheng, M. Holbrook, P. Kumaraguru, L. Cranor, and J. Downs. 2010. Who falls for phish? A demographic analysis of phishing susceptibility and effectiveness of interventions. In Proc. CHI’10. 373--382.
118. O. Sheyner, J. Haines, S. Jha, R. Lippmann, and J. Wing. 2002. Automated generation and analysis of attack graphs. In IEEE Symp. on Security and Privacy. 273--284.
119. A. Singhal and X. Ou. 2011. Security Risk Analysis of Enterprise Networks Using Probabilistic Attack Graphs. National Institute of Standards and Technology. Retrieved from http://csrc.nist.gov/publications/nistir/ir7788/NISTIR-7788.pdf.
120. K. Snow, F. Monrose, L. Davi, A. Dmitrienko, C. Liebchen, and A. Sadeghi. 2013. Just-in-time code reuse: On the effectiveness of fine-grained address space layout randomization. In Proc. IEEE Symposium on Security and Privacy. 574--588.
121. N. Stakhanova, C. Strasburg, S. Basu, and J. Wong. 2012. Towards cost-sensitive assessment of intrusion response selection. J. Comput. Secur. 20, 2--3 (2012), 169--198.
122. S. Stevens. 1946. On the theory of scales of measurement. (1946).
123. S. Stolfo, W. Fan, W. Lee, A. Prodromidis, and P. Chan. 2000. Cost-based modeling for fraud and intrusion detection: Results from the JAM project. In Proc. DISCEX’00. 130--144.
124. B. Stone-Gross, C. Kruegel, K. Almeroth, A. Moser, and E. Kirda. 2009. FIRE: FInding rogue networks. In Proc. ACSAC’09. 231--240.
125. C. Strasburg, N. Stakhanova, S. Basu, and J. Wong. 2009. A framework for cost sensitive assessment of intrusion response selection. In Proc. COMPSAC’09. 355--360.
126. Adrian Tang, Simha Sethumadhavan, and Salvatore Stolfo. 2015. Heisenbyte: Thwarting memory disclosure attacks using destructive code reads. In Proc. ACM CCS’15. 256--267.
127. M. Tavallaee, N. Stakhanova, and A. Ghorbani. 2010. Toward credible evaluation of anomaly-based intrusion-detection methods. IEEE Trans. Syst, Man Cybernet. C 40, 5 (2010), 516--524.
128. U. Thakore. 2015. A quantitative methodology for evaluating and deploying security monitors. Retrieved from https://www.ideals.illinois.edu/handle/2142/88103. (2015).
129. X. Ugarte-Pedrero, D. Balzarotti, I. Santos, and P. Bringas. 2015. SoK: Deep packer inspection: A longitudinal study of the complexity of run-time packers. In 2015 IEEE Symposium on Security and Privacy. 659--673.
130. B. Ur, P. Kelley, S. Komanduri, J. Lee, M. Maass, M. Mazurek, T. Passaro, R. Shay, T. Vidas, L. Bauer, N. Christin, and L. Cranor. 2012. How does your password measure up? The effect of strength meters on password creation. In Proc. USENIX Security Symposium.
131. B. Ur, S. Segreti, L. Bauer, N. Christin, L. Cranor, S. Komanduri, D. Kurilova, M. Mazurek, William Melicher, and Richard Shay. 2015. Measuring real-world accuracies and biases in modeling password guessability. In Proc. 24th USENIX Security Symposium.
132. P. Velleman and L. Wilkinson. 1993. Nominal, ordinal, interval, and ratio typologies are misleading. Am. Stat. 47, 1 (1993), 65--72.
133. V. Verendel. 2009. Quantified security is a weak hypothesis: A critical survey of results and assumptions. In Proc. NSPW’09. 37--50.
134. C. Villarrubia, E. Fernandez-Medina, and M. Piattini. 2004. Towards a classification of security metrics. In Proc. ICEIS’04. 341--350.
135. Nedim Šrndic and Pavel Laskov. 2014. Practical evasion of a learning-based classifier: A case study. In Proc. IEEE Symposium on Security and Privacy. 197--211.
136. I. Wagner and D. Eckhoff. 2015. Technical Privacy Metrics: A Systematic Survey. Technical Report 1512.00327. arXiv. Retrieved from http://arxiv.org/abs/1512.00327 arXiv: 1512.00327.
137. L. Wang, T. Islam, T. Long, A. Singhal, and S. Jajodia. 2008. An attack graph-based probabilistic security metric. In Proc. IFIP Conf. on Data and App. Security. 283--296.
138. L. Wang, S. Jajodia, A. Singhal, and S. Noel. 2010. k-zero day safety: Measuring the security risk of networks against unknown attacks. In Proc. ESORICS’10. 573--587.
139. H. Wei, D. Frinke, O. Carter, and C. Ritter. 2001. Cost-benefit analysis for network intrusion detection systems. In Proceedings of the 28th Annual Computer Security Conference. Washington, D.C.
140. M. Weir, S. Aggarwal, M. Collins, and H. Stern. 2010. Testing metrics for password creation policies by attacking large sets of revealed passwords. In Proc. ACM CCS’10. 162--175.
141. L. Xu, Z. Zhan, S. Xu, and K. Ye. 2014b. An evasion and counter-evasion study in malicious websites detection. In Proc. IEEE CNS’14. 265--273.
142. M. Xu, G. Da, and S. Xu. 2015a. Cyber epidemic models with dependences. Internet Math. 11, 1 (2015), 62--92.
143. M.Xu and S. Xu. 2012. An extended stochastic model for quantitative security analysis of networked systems. Internet Math. 8, 3 (2012), 288--320.
144. S. Xu. 2014a. Cybersecurity dynamics. In Proc. HotSoS’14. 14:1--14:2.
145. S. Xu. 2014b. Emergent behavior in cybersecurity. In Proc. HotSoS’14. 13:1--13:2.
146. S. Xu, W. Lu, and H. Li. 2015b. A stochastic model of active cyber defense dynamics. Internet Math. 11, 1 (2015), 23--61.
147. S. Xu, W. Lu, and L. Xu. 2012a. Push- and pull-based epidemic spreading in arbitrary networks: Thresholds and deeper insights. ACM TAAS 7, 3 (2012), 32:1--32:26.
148. S. Xu, W. Lu, L. Xu, and Z. Zhan. 2014a. Adaptive epidemic dynamics in networks: Thresholds and control. ACM TAAS 8, 4 (2014), 19.
149. S. Xu, W. Lu, and Z. Zhan. 2012b. A stochastic model of multivirus dynamics. IEEE Trans. Depend. Secure Comput. 9, 1 (2012), 30--45.
150. D. Yardon. May 4, 2014. Symantec Develops New Attack on Cyberhacking. Retrieved from http://www.wsj. com/articles/SB10001424052702303417104579542140235850578. (May 4, 2014).
151. T. Yen, V. Heorhiadi, A. Oprea, M. Reiter, and A. Juels. 2014. An epidemiological study of malware encounters in a large enterprise. In Proc. ACM CCS’14. 1117--1130.
152. S. Yilek, E. Rescorla, H. Shacham, B. Enright, and S. Savage. 2009. When private keys are public: Results from the 2008 debian OpenSSL vulnerability. In Proc. ACM IMC’09. 15--27.
153. K. Zaffarano, J. Taylor, and S. Hamilton. 2015. A quantitative framework for moving target defense effectiveness evaluation. In Proc. ACM MTD’15. 3--10.
154. Z. Zhan, M. Xu, and S. Xu. 2014. A characterization of cybersecurity posture from network telescope data. In Proc. InTrust’14. 105--126.
155. J. Zhang, Z. Durumeric, M. Bailey, M. Liu, and M. Karir. 2014. On the mismanagement and maliciousness of networks. In Proc. NDSS’14.
156. M. Zhang, L. Wang, S. Jajodia, A. Singhal, and M. Albanese. 2016. Network diversity: A security metric for evaluating the resilience of networks against zero-day attacks. IEEE T-IFS 11, 5 (May 2016), 1071--1086.
157. S. Zhang, X. Zhang, and X. Ou. 2014. After we knew it: Empirical study and modeling of cost-effectiveness of exploiting prevalent known vulnerabilities across IaaS cloud. In Proc. ACM AsiaCCS. 317--328.
158. R. Zheng, W. Lu, and S. Xu. 2015. Active cyber defense dynamics exhibiting rich phenomena. In Proc. HotSoS’15. 2:1--2:12.

Index Terms

• General and reference

  • Cross-computing tools and techniques

    • Metrics

  • Document types

    • Surveys and overviews

• Security and privacy

  • Formal methods and theory of security