Application performance on the Direct Access File System

ABSTRACT

The Direct Access File System (DAFS) is a distributed file system built on top of direct-access transports (DAT). Direct-access transports are characterized by using remote direct memory access (RDMA) for data transfer and user-level networking. The motivation behind the DAT-enabled distributed file system architecture is the reduction of the CPU overhead on the I/O data path.We have created an implementation of DAFS for the FreeBSD platform. In this paper we describe the performance evaluation study of DAFS that we have performed using this software. The goal of this study is to determine whether the architecture of DAFS brings any fundamental performance benefits to applications compared to traditional distributed file systems, such as NFS. We perform comparison of DAFS to a version of NFS optimized to reduce the I/O overhead. In order to thoroughly understand the impact of DAFS on application performance, we consider a diverse range of applications workloads.We conclude that DAFS can accomplish superior performance for latency-sensitive applications, outperforming NFS by up to a factor of 2. Bandwidth-sensitive applications do equally well on both systems, unless they are CPU-intensive, in which case they perform better on DAFS. We also found that RDMA is a less restrictive mechanism to achieve copy avoidance than that used by the optimized NFS.

REFERENCES

• 1: DAFS Collaborative. Direct Access File System Protocol, Version 0.90, June 2001, http://www.dafscollaborative.org.
• 2: DAFS Collaborative. DAFSAP1, Version 0.6, June 2001, http://www.dafscollaborative.org.
• 3: Olson, M., Bostic, K., Seltzer, M. "Berkeley DB", In Proceedings of USENIX 1999 Annual Technical Conference, June 1999.
• 4: Shepler, S. et al. NFS Version 4 Protocol. RFC 3010, December 2000.
• 5: Compaq, Intel, Microsoft, Virtual Interface Specification, Version 1.0, December 16, 1997.
• 6: J. S. Chase, K. G. Yocum, and A. J. Gallatin. "End-System Optimizations for High-Speed TCP", IEEE Communications Special Issue on TCP Performance in Future Networking Environments, 39(4):68--74, April 2001.
• 7: Kostas Magoutis , Salimah Addetia , Alexandra Fedorova , Margo I. Seltzer , Jeffrey S. Chase , Andrew J. Gallatin , Richard Kisley , Rajiv Wickremesinghe , Eran Gabber, Structure and Performance of the Direct Access File System, Proceedings of the General Track: 2002 USENIX Annual Technical Conference, p.1-14, June 10-15, 2002
• 8: Addetia, S. User-level Client-side Caching for DAFS. Technical Report, Harvard University TR-14-01, March 2002.
• 9: Anderson, D., Chase, J., Gadde, S., Gallatin, A., Yocum, K. "Cheating the I/O Bottleneck: Network Storage With Trapeze/Myrinet", Proceedings of USENIX 1998 Annual Technical Conference, June 1998.
• 10: Brustoloni, J., "Interoperation of Copy Avoidance in Network and File I/O", In Proc. Of 18th IEEE Conference on Computer Communications (INFOCOM'99), New York, NY, March 1999.
• 11: Chu, J. "Zero-copy TCP in Solaris", In Proceedings of USENIX 1996 Annual Technical Conference, San Diego, CA, January 1996.
• 12: Yousef A. Khalidi , Moti N. Thadani, An Efficient Zero-Copy I/O Framework for UNIX, Sun Microsystems, Inc., Mountain View, CA, 1995
• 13: DAT Collaborative, http://www.datcollaborative.org.
• 14: Keith Arnold Smith , Margo Seltzer, Workload-specific file system benchmarks, 2001
• 15: Xiaolan Zhang , Margo Seltzer, HBench:Java: an application-specific benchmarking framework for Java virtual machines, Proceedings of the ACM 2000 conference on Java Grande, p.62-70, June 03-04, 2000, San Francisco, California, United States [doi>10.1145/337449.337474]
• 16: Margo Seltzer , David Krinsky , Keith Smith , Xiaolan Zhang, The Case for Application-Specific Benchmarking, Proceedings of the The Seventh Workshop on Hot Topics in Operating Systems, p.102, March 28-30, 1999
• 17: Yuanyuan Zhou , Angelos Bilas , Suresh Jagannathan , Cezary Dubnicki , James F. Philbin , Kai Li, Experiences with VI communication for database storage, Proceedings of the 29th annual international symposium on Computer architecture, p.257, May 25-29, 2002, Anchorage, Alaska
• 18: Scott, H., "User-level I/O for Database Management Systems", Master's Thesis, Queen's University, Kingston, Ontario, Canada, March 2001.
• 19: Transaction Processing Performance Council, "TPC Benchmark C", Standard Specification, Revision 5.0, Februrary 26, 2001, http://www.tpc.org.
• 20: Sun Microsystems, "Dbbench --- Database Benchmarking Tool --- Version 2.4 Manual", 1994.
• 21: Fedorova, A., Seltzer, M. Personal Communication on Optimizing Transactions and Data Structures in Berkeley DB, Harvard University, October 2002.
• 22: Kostas Magoutis , Salimah Addetia , Alexandra Fedorova , Margo I. Seltzer, Making the Most Out of Direct-Access Network Attached Storage, Proceedings of the 2nd USENIX Conference on File and Storage Technologies, March 31-31, 2003, San Francisco, CA
• 23: Kostas Magoutis, Design and Implementation of a Direct Access File System (DAFS) Kernel Server for FreeBSD, Proceedings of the BSDCon 2002, p.65-76, February 11-14, 2002

INDEX TERMS

Primary Classification:
  D. Software
  D.4 OPERATING SYSTEMS
      D.4.3 File Systems Management
          Subjects: Distributed file systems

Additional Classification:
  C. Computer Systems Organization
  C.4 PERFORMANCE OF SYSTEMS
      Subjects: Measurement techniques

  D. Software
  D.4 OPERATING SYSTEMS
      D.4.8 Performance
          Subjects: Modeling and prediction

General Terms:
Experimentation, Performance

Keywords:
Direct Access File System, Distributed File Systems, RDMA, performance measurement