An empirical study of the robustness of MacOS applications using random testing

ABSTRACT

We report on the fourth in a series of studies on the reliability of application programs in the face of random input. Over the previous 15 years, we have studied the reliability of UNIX command line and X-Window based (GUI) applications and Windows applications. In this study, we apply our fuzz testing techniques to applications running on the Mac OS X operating system. We continue to use a simple, or even simplistic technique: unstructured black-box random testing, considering a failure to be a crash or hang. As in the previous three studies, the technique is crude but seems to be effective in locating bugs in real programs.We tested the reliability of 135 command-line UNIX utilities and thirty graphical applications on Mac OS X by feeding random input to each. We report on application failures -- crashes (dumps core) or hangs (loops indefinitely) -- and, where source code is available, we identify the causes of these failures and categorize them.Our testing crashed only 7% of the command-line utilities, a considerably lower rate of failure than observed in almost all cases of previous studies. We found the GUI-based applications to be less reliable: of the thirty that we tested, only eight did not crash or hang. Twenty others crashed, and two hung. These GUI results were noticeably worse than either of the previous Windows (Win32) or UNIX (X-Windows) studies.

REFERENCES

• 1: D. Aitel, "The Advantages of Block-Based Protocol Analysis for Security Testing", Immunity Inc., February 2002. http://www.immunitysec.com/downloads/advantages_of_block_based_analysis.html
• 2: Apple Computer, May 2006, http://developer.apple.com/documentation/Cocoa/Conceptual/Coco aFundamentals/WhatIsCocoa/chapter_2_section_6.html.
• 3: G. J. Carrette, "CRASHME: Random Input Testing", http://people.delphi.com/gjc/crashme.html, 1996.
• 4: J. W. Duran and S. C. Ntafos, "An Evaluation of Random Testing", IEEE Transactions on Software EngineeringSE-10, 4, July 1984, pp. 438--444.
• 5: J. E. Forrester and B. P. Miller, "An Empirical Study of the Robustness of Windows NT Applications Using Random Testing", 4th USENIX Windows Systems Symposium, Seattle, August 2000. Appears (in German translation) as "Empirische Studie zur Stabilität von NT-Anwendungen", iX, September 2000.
• 6: Simson Garfinkel , Gene Spafford, Practical Unix and Internet security (2nd ed.), O'Reilly & Associates, Inc., Sebastopol, CA, 1996
• 7: A. Ghosh , M. Schmid , V. Shah, Testing the Robustness of Windows NT Software, Proceedings of the The Ninth International Symposium on Software Reliability Engineering, p.231, November 04-07, 1998
• 8: A. Ghosh, V. Shah and M. Schmid, "An Approach for Analyzing the Robustness of Windows NT Software", 21st National Information Systems Security Conference, Crystal City, VA, October 1998.
• 9: Andy Hertzfeld, Revolution in The Valley (hardcover), O' Reilly & Associates, Inc., 2004
• 10: Sylvain Marquis , Thomas R. Dean , Scott Knight, SCL: a language for security testing of network applications, Proceedings of the 2005 conference of the Centre for Advanced Studies on Collaborative research, p.155-164, October 17-20, 2005, Toranto, Ontario, Canada
• 11: W. McKeeman, "Differential Testing for Software", Digital Technical Journal, Digital Equipment Corporation 10, 1, December 1998.
• 12: Glenford J. Myers, Art of Software Testing, John Wiley & Sons, Inc., New York, NY, 1979
• 13: Microsoft Corporation, "Security and Reliability Strategies", http://www.microsoft.com/whdc/driver/security/, 2006.
• 14: B. P. Miller, D. Koski, C. P. Lee, V. Maganty, R. Murthy, A. Natarajan, J. Steidl, "Fuzz Revisited: A Re-examination of the Reliability of UNIX Utilities and Services", University of Wisconsin-Madison, 1995. Appears (in German translation) as "Empirische Studie zur Zuverlasskeit von UNIX-Utilities: Nichts dazu Gerlernt", iX, September 1995. ftp://grilled.cs.wisc.edu/technical_papers/fuzz-revisted.pdf.
• 15: Barton P. Miller , Louis Fredriksen , Bryan So, An empirical study of the reliability of UNIX utilities, Communications of the ACM, v.33 n.12, p.32-44, Dec. 1990 [doi>10.1145/96267.96279]
• 16: Emin Gün Sirer , Brian N. Bershad, Using production grammars in software testing, Proceedings of the 2nd conference on Domain-specific languages, p.1-13, October 03-06, 1999, Austin, Texas, United States
• 17: David A. Wood , Garth A. Gibson , Randy H. Katz, Verifying a Multiprocessor Cache Controller Using Random Case, University of California at Berkeley, Berkeley, CA, 1989
• 18: Shu Xiao , Lijun Deng , Sheng Li , Xiangrong Wang, Integrated TCP/IP Protocol Software Testing for Vulnerability Detection, Proceedings of the 2003 International Conference on Computer Networks and Mobile Computing, p.311, October 20-23, 2003

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.5 Testing and Debugging

Additional Classification:
  D. Software
  D.4 OPERATING SYSTEMS
      D.4.0 General

          Nouns: Macintosh OS

  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Graphical user interfaces (GUI)

Keywords:
fuzz, random testing