Reducing TCB complexity for security-sensitive applications: three case studies

ABSTRACT

The large size and high complexity of security-sensitive applications and systems software is a primary cause for their poor testability and high vulnerability. One approach to alleviate this problem is to extract the security-sensitive parts of application and systems software, thereby reducing the size and complexity of software that needs to be trusted. At the system software level, we use the Nizza architecture which relies on a kernelized trusted computing base (TCB) and on the reuse of legacy code using trusted wrappers to minimize the size of the TCB. At the application level, we extract the security-sensitive portions of an already existing application into an AppCore. The AppCore is executed as a trusted process in the Nizza architecture while the rest of the application executes on a virtualized, untrusted legacy operating system. In three case studies of real-world applications (e-commerce transaction client, VPN gateway and digital signatures in an e-mail client), we achieved a considerable reduction in code size and complexity. In contrast to the few hundred thousand lines of current application software code running on millions of lines of systems software code, we have AppCores with tens of thousands of lines of code running on a hundred thousand lines of systems software code. We also show the performance penalty of AppCores to be modest (a few percent) compared to current software.

REFERENCES

• 1: Microsoft. Next-Generation Secure Computing Base. http://www.microsoft.com/resources/ngscb/default.mspx
• 2: Mozilla Foundation. Mozilla Module Owners. http://www.mozilla.org/owners.html
• 3: PeerSee Networks. MatrixSSL - Open Source Embedded SSL. http://www.matrixssl.org/
• 4: Secunia. Vulnerability Report --- Microsoft Internet Explorer 6. http://secunia.com/product/11/
• 5: Secunia. Vulnerability Report --- Mozilla Firefox 1.x. http://secunia.com/product/4227/
• 6: Secunia. Vulnerability Report --- Xll Windowing System (Xll) 6.x.http://secunia.com/product/3913/
• 7: Secunia. Vulnerability Report --- Linux Kernel 2.4.x. http://secunia.com/product/763/
• 8: Secunia. Check Point VPN-1 Products ISAKMP Buffer Overflow Vulnerability. http://secunia.com/advisories/11546/
• 9: Snapgear. Snapgear Embedded Linux. http://www.snapgear.org
• 10: Trusted Computing Group. TCG Main Specification vl.Ib, https://www.trustedcomputinggroup.org/
• 11: J. Bambenek, SANS Institute. BHO scanning tool and New Scam Targets Bank Customers. http://isc.sans.org/diary.php?date=2004-06-29.
• 12: Paul Barham , Boris Dragovic , Keir Fraser , Steven Hand , Tim Harris , Alex Ho , Rolf Neugebauer , Ian Pratt , Andrew Warfield, Xen and the art of virtualization, Proceedings of the nineteenth ACM symposium on Operating systems principles, October 19-22, 2003, Bolton Landing, NY, USA
• 13: V. Basili and D. Hutchens. An Empirical Study of a Complexity Family. In IEEE Transactions on Software Engineering, Volume 9, No. 6, November 1983, pp. 664--672.
• 14: D. Brumley, D. X. Song. Privtrans: Automatically Partitioning Programs for Privilege Separation. In Proc. USENIX Security Symposium, San Diego, USA. Aug 9--13, 2004.
• 15: Peter M. Chen , Brian D. Noble, When Virtual Is Better Than Real, Proceedings of the Eighth Workshop on Hot Topics in Operating Systems, p.133, May 20-22, 2001
• 16: N. Chou, R. Ledesma, Y. Teraguchi, D. Boneh and J. C. Mitchell, Client-side defense against web-based identity theft, In 11th Annual Network and Distributed System Security Symposium (NDSS '04), San Diego, February, 2004.
• 17: D. Engler, D. Chelf, A. Chou, and S. Hallem. Checking system rules using system specific programmer-written compiler extensions. In 4th USENIX OSDI. San Diego, Oct. 2000.
• 18: Dawson Engler , David Yu Chen , Seth Hallem , Andy Chou , Benjamin Chelf, Bugs as deviant behavior: a general approach to inferring errors in systems code, Proceedings of the eighteenth ACM symposium on Operating systems principles, October 21-24, 2001, Banff, Alberta, Canada
• 19: Norman Feske , Christian Helmuth, A Nitpicker's guide to a minimal-complexity secure GUI, Proceedings of the 21st Annual Computer Security Applications Conference, p.85-94, December 05-09, 2005 [doi>10.1109/CSAC.2005.7]
• 20: Norman E. Fenton , Niclas Ohlsson, Quantitative Analysis of Faults and Failures in a Complex Software System, IEEE Transactions on Software Engineering, v.26 n.8, p.797-814, August 2000 [doi>10.1109/32.879815 ]
• 21: Gaffney, J., Program Control Complexity and Productivity. In Proceedings of the IEEE Workshop on Quantitative Software Models, pg 179, October, 1979.
• 22: Tal Garfinkel , Ben Pfaff , Jim Chow , Mendel Rosenblum , Dan Boneh, Terra: a virtual machine-based platform for trusted computing, Proceedings of the nineteenth ACM symposium on Operating systems principles, October 19-22, 2003, Bolton Landing, NY, USA
• 23: Hermann Härtig, Security architectures revisited, Proceedings of the 10th workshop on ACM SIGOPS European workshop: beyond the PC, July 01-01, 2002, Saint-Emilion, France [doi>10.1145/1133373.1133376]
• 24: H. Härtig, M. Hohmuth, N. Feske, C. Helmuth, A. Lack-orzynski, F. Mehnert and M. Peter. The Nizza Secure-System Architecture. In IEEE CollaborateCom 2005. San Jose, USA. Dec 2005.
• 25: Hermann Härtig , Michael Hohmuth , Jochen Liedtke , Sebastian Schönberg , Jean Wolter, The performance of μ-kernel-based systems, Proceedings of the sixteenth ACM symposium on Operating systems principles, p.66-77, October 05-08, 1997, Saint Malo, France
• 26: C. Helmuth, A. Warg, and N. Feske. Mikro-SINA---Hands-on Experiences with the Nizza Security Architecture. In Proceedings of the D.A.C.H Security 2005, Darmstadt, Germany, March 2005.
• 27: A. Herzberg and A. Gbara, TrustBar: Protecting (even Naïve) Web Users from Spoofing and Phishing Attacks, Cryptology ePrint Archive, Report 2004/155. 2004.
• 28: Michael Hohmuth , Michael Peter , Hermann Härtig , Jonathan S. Shapiro, Reducing TCB size by using untrusted components: small kernels versus virtual-machine monitors, Proceedings of the 11th workshop on ACM SIGOPS European workshop: beyond the PC, September 19-22, 2004, Leuven, Belgium [doi>10.1145/1133572.1133615]
• 29: T. Jaeger, R. Sailer, and X. Zhang, Analyzing Integrity Protection in the SELinux Example Policy, in 12th USENIX Security Symposium, Washington D.C. USA, Aug. 2003.
• 30: D. Kilpatrick, Privman: A Library for Partitioning Applications. In USENIX Annual Technical Conference, FREENIX Track 2003, pp 273--284. San Antonio USA, July 2003.
• 31: David Lie , Chandramohan A. Thekkath , Mark Horowitz, Implementing an untrusted operating system on trusted hardware, Proceedings of the nineteenth ACM symposium on Operating systems principles, October 19-22, 2003, Bolton Landing, NY, USA
• 32: J. Liedtke, On micro-kernel construction, Proceedings of the fifteenth ACM symposium on Operating systems principles, p.237-250, December 03-06, 1995, Copper Mountain, Colorado, United States
• 33: Thomas J. McCabe, A complexity measure, Proceedings of the 2nd international conference on Software engineering, p.407, October 13-15, 1976, San Francisco, California, United States
• 34: Xiaohu Qie , Ruoming Pang , Larry Peterson, Defensive programming: using an annotation toolkit to build DoS-resistant software, Proceedings of the 5th symposium on Operating systems design and implementation

  Due to copyright restrictions we are not able to make the PDFs for this conference available for downloading

  , December 09-11, 2002, Boston, Massachusetts [doi>10.1145/1060289.1060295]
• 35: B. Pfitzmann, J. Riordan, C. Stüble, M. Waidner and A. Weber. The PERSEUS System Architecture. Research Report. IBM Research Division. RZ 3335. Sept. 2001.
• 36: N. Provos, M. Friedl, and P. Honeyman. Preventing privilege escalation. In 12th USENIX Security Symposium, Washington D.C, Aug. 2003.
• 37: B. Ross, C. Jackson, N. Miyake, D. Boneh and J. C. Mitchell, Stronger Password Authentication Using Browser Extensions. In 14th Usenix Security Symposium, Baltimore, USA, Aug. 2005.
• 38: JH Saltzer and MD Schroeder, The Protection of Information in Computer Systems, Proc. of the IEEE, Vol.63, No.9, Sept. 1975, pp. 1278--1308.
• 39: R. Sailer, X. Zhang, T. Jaeger, and L. V. Doorn. Design and Implementation of a TCG-based Integrity Measurement Architecture. In Proceedings of Thirteenth USENIX Security Symposium, pp 223--238, August 2004.
• 40: B. Schneier. Software Complexity and Security. Crypto-Gram Newsletter. March 2000. http://www.schneier.com/crypto-gram-0003.html
• 41: Jonathan S. Shapiro , Jonathan M. Smith , David J. Farber, EROS: a fast capability system, Proceedings of the seventeenth ACM symposium on Operating systems principles, p.170-185, December 12-15, 1999, Charleston, South Carolina, United States
• 42: J. S. Shapiro, J. Vanderburgh, E. Northup, and D. Chizmadia, Design of the EROS Trusted Window System, In Proc. USENIX Security Symposium, San Diego CA, 2004
• 43: Vincent Y. Shen , Tze-Jie Yu , Stephen M. Thebaut , Lorri R. Paulsen, Identifying error-prone software—an empirical study, IEEE Transactions on Software Engineering, v.11 n.4, p.317-324, April 1985 [doi>10.1109/TSE.1985.232222]
• 44: Martin Shepperd , Darrel Ince, Derivation and validation of software metrics, Oxford University Press, Inc., New York, NY, 1993
• 45: R. Spencer, S. Smalley, P. Loscocco, M. Hibler, D. Andersen and J. Lepreau. The Flask Security Architecture: System Support for Diverse Security Policies. In Procedings of the 8th USENLX Security Symposium, Aug. 1999.
• 46: J. D. Tygar and A. Whitten. WWW electronic commerce and Java Trojan horses. In Proc. of the 2nd USENIX Workshop on Electronic Commerce, Nov. 1996, pp. 243--250.
• 47: D. Wagner, J. Foster, E. Brewer, and A. Aiken. A first step towards automated detection of buffer overrun vulnerabilities. In Proceedings of the ISOC Symposium on Network and Distributed System Security, 2000.
• 48: D. Wheeler. SLOCCount. http://www.dwheeler.com/sloccount/
• 49: Chris Wright , Crispin Cowan , Stephen Smalley , James Morris , Greg Kroah-Hartman, Linux Security Modules: General Security Support for the Linux Kernel, Proceedings of the 11th USENIX Security Symposium, p.17-31, August 05-09, 2002
• 50: B. Yee and D. Tygar. Secure coprocessors in electronic commerce applications. In Proc. of the First USENIX Workshop on Electronic Commerce, New York, July 1995.

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
        D.2.11 Software Architectures

Additional Classification:
  D. Software
  D.4 OPERATING SYSTEMS
      D.4.6 Security and Protection

General Terms:
Reliability, Security

Keywords:
application security, trusted computing base