research-article

Server-side I/O coordination for parallel file systems

Authors:

Samuel LangAuthors Info & Claims

SC '11: Proceedings of 2011 International Conference for High Performance Computing, Networking, Storage and Analysis

Article No.: 17, Pages 1 - 11

https://doi.org/10.1145/2063384.2063407

Published: 12 November 2011 Publication History

Abstract

Parallel file systems have become a common component of modern high-end computers to mask the ever-increasing gap between disk data access speed and CPU computing power. However, while working well for certain applications, current parallel file systems lack the ability to effectively handle concurrent I/O requests with data synchronization needs, whereas concurrent I/O is the norm in data-intensive applications. Recognizing that an I/O request will not complete until all involved file servers in the parallel file system have completed their parts, in this paper we propose a serverside I/O coordination scheme for parallel file systems. The basic idea is to coordinate file servers to serve one application at a time in order to reduce the completion time, and in the meantime maintain the server utilization and fairness. A window-wide coordination concept is introduced to serve our purpose. We present the proposed I/O coordination algorithm and its corresponding analysis of average completion time in this study. We also implement a prototype of the proposed scheme under the PVFS2 file system and MPI-IO environment. Experimental results demonstrate that the proposed scheme can reduce average completion time by 8% to 46%, and provide higher I/O bandwidth than that of default data access strategies adopted by PVFS2 for heavy I/O workloads. Experimental results also show that the server-side I/O coordination scheme has good scalability.

References

[1]

High-performance Storage Architecture and Scalable Cluster File System. Lustre File System White Paper, Dec. 2007.

[2]

R. H. Arpaci, A. C. Dusseau, A. M. Vahdat, L. T. Liu, T. E. Anderson, and D. A. Patterson. The Interaction of Parallel and Sequential Workloads on a Network of Workstations. In SIGMETRICS '95/PERFORMANCE '95: Proceedings of the 1995 ACM SIGMETRICS Joint International Conference on Measurement and Modeling of Computer Systems, pages 267--278, New York, NY, USA, 1995. ACM.

Digital Library

[3]

A. C. Arpaci-Dusseau. Implicit Coscheduling: Coordinated Scheduling with Implicit Information in Distributed Systems. ACM Transactions on Computer Systems (TOCS), 19(3):283--331, 2001.

Digital Library

[4]

D. D. Chambliss, G. A. Alvarez, P. Pandey, D. Jadav, J. Xu, R. Menon, and T. P. Lee. Performance Virtualization for Large-scale Storage Systems. In SRDS '03: Proceedings of the 22th International Symposium on Reliable Distributed Systems, pages 109--118, 2003.

[5]

D. G. Feitelson. Packing Schemes for Gang Scheduling. In IPPS '96: Proceedings of the Workshop on Job Scheduling Strategies for Parallel Processing, pages 89--110, London, UK, 1996. Springer-Verlag.

Digital Library

[6]

D. G. Feitelson and M. A. Jette. Improved Utilization and Responsiveness with Gang Scheduling. In IPPS '97: Proceedings of the Job Scheduling Strategies for Parallel Processing, pages 238--261, London, UK, 1997. Springer-Verlag.

Digital Library

[7]

D. G. Feitelson and L. Rudolph. Gang Scheduling Performance Benefits for Fine-Grain Synchronization. Journal of Parallel and Distributed Computing, 16:306--318, 1992.

[8]

A. Gulati, I. Ahmad, and C. A. Waldspurger. PARDA: Proportional Allocation of Resources for Distributed Storage Access. In FAST '09: Proccedings of the 7th Conference on File and Storage Technologies, pages 85--98, Berkeley, CA, USA, 2009. USENIX Association.

Digital Library

[9]

I. F. Haddad. PVFS: A Parallel Virtual File System for Linux Clusters. Linux Journal, page 5, 2000.

Digital Library

[10]

L. Huang, G. Peng, and T.-c. Chiueh. Multi-dimensional Storage Virtualization. SIGMETRICS Perform. Eval. Rev., 32(1):14--24, 2004.

Digital Library

[11]

W. B. Ligon III and R. B. Ross. Implementation and Performance of a Parallel File System for High Performance Distributed Applications. In HPDC '96: Proceedings of the 5th IEEE International Symposium on High Performance Distributed Computing, page 471, Washington, DC, USA, 1996. IEEE Computer Society.

Digital Library

[12]

M. Karlsson, C. Karamanolis, and X. Zhu. Triage: Performance Differentiation for Storage Systems Using Adaptive Control. ACM Trans. Storage, 1(4):457--480, 2005.

Digital Library

[13]

D. Kotz. Disk-directed I/O for MIMD Multiprocessors. ACM Trans. Comput. Syst., 15(1):41--74, 1997.

Digital Library

[14]

W. Lee, M. Frank, V. Lee, K. Mackenzie, and L. Rudolph. Implications of I/O for Gang Scheduled Workloads. In IPPS '97: Proceedings of the Job Scheduling Strategies for Parallel Processing, pages 215--237, London, UK, 1997. Springer-Verlag.

Digital Library

[15]

S. T. Leutenegger and M. K. Vernon. The Performance of Multiprogrammed Multiprocessor Scheduling Algorithms. In SIGMETRICS '90: Proceedings of the 1990 ACM SIGMETRICS Conference on Measurement and Modeling of Computer Systems, pages 226--236, New York, NY, USA, 1990. ACM.

Digital Library

[16]

C. L. Liu and J. W. Layland. Scheduling Algorithms for Multiprogramming in a Dard-Real-Time Environment. Journal of the ACM, 20(1):46--61, 1973.

Digital Library

[17]

J. E. Moreira, W. Chan, L. L. Fong, H. Franke, and M. A. Jette. An Infrastructure for Efficient Parallel Job Execution in Terascale Computing Environments. In Supercomputing '98: Proceedings of the 1998 ACM/IEEE conference on Supercomputing (CDROM), pages 1--14, Washington, DC, USA, 1998. IEEE Computer Society.

Digital Library

[18]

D. Nagle, D. Serenyi, and A. Matthews. The Panasas Activescale Storage Cluster: Delivering Scalable High Bandwidth Storage. In Supercomputing '04: Proceedings of the 2004 ACM/IEEE conference on Supercomputing, page 53, Washington, DC, USA, 2004. IEEE Computer Society.

Digital Library

[19]

A. Povzner, D. Sawyer, and S. Brandt. Horizon: Efficient Deadline-driven Disk I/O Management for Distributed Storage Systems. In HPDC '10: Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing, pages 1--12, 2010.

Digital Library

[20]

A. L. Narasimha Reddy and J. C. Wyllie. Disk Scheduling in a Multimedia I/O System. In MULTIMEDIA '93: Proceedings of the First ACM International Conference on Multimedia, pages 225--233, New York, NY, USA, 1993. ACM.

Digital Library

[21]

R. B. Ross and W. B. Ligon III. Server-Side Scheduling in Cluster Parallel I/O Systems. Calculateurs Parallèles Special Issue on Parallel I/O for Cluster Computing, 2001.

[22]

F. Schmuck and R. Haskin. GPFS: A Shared-Disk File System for Large Computing Clusters. In FAST '02: Proceedings of the 1st USENIX Conference on File and Storage Technologies, page 19, Berkeley, CA, USA, 2002. USENIX Association.

Digital Library

[23]

K. E. Seamons, Y. Chen, P. Jones, J. Jozwiak, and M. Winslett. Server-directed Collective I/O in Panda. In Supercomputing '95: Proceedings of the 1995 ACM/IEEE conference on Supercomputing (CDROM), page 57, New York, NY, USA, 1995. ACM.

Digital Library

[24]

A. Snavely and D. M. Tullsen. Symbiotic Job Scheduling for a Simultaneous Multithreaded Processor. In ASPLOS-IX: Proceedings of the ninth international conference on Architectural support for programming languages and operating systems, pages 234--244, New York, NY, USA, 2000. ACM.

Digital Library

[25]

P. G. Sobalvarro. Demand-Based Coscheduling of Parallel Jobs on Multiprogrammed Multiprocessors. PhD thesis, 1997. Supervisor-Weihl, William E.

Digital Library

[26]

R. Thakur, W. Gropp, and E. Lusk. Data Sieving and Collective I/O in ROMIO. In FRONTIERS '99: Proceedings of the The 7th Symposium on the Frontiers of Massively Parallel Computation, page 182, Washington, DC, USA, 1999. IEEE Computer Society.

Digital Library

[27]

F. Wang, H. Franke, M. C. Papaefthymiou, P. Pattnaik, L. Rudolph, and M. S. Squillante. A Gang Scheduling Design for Multiprogrammed Parallel Computing Environments. In IPPS '96: Proceedings of the Workshop on Job Scheduling Strategies for Parallel Processing, pages 111--125, London, UK, 1996. Springer-Verlag.

Digital Library

[28]

F. Wang, M. C. Papaefthymiou, and M. S. Squillante. Performance Evaluation of Gang scheduling for Parallel and Distributed Multiprogramming. In IPPS '97: Proceedings of the Job Scheduling Strategies for Parallel Processing, pages 277--298, London, UK, 1997. Springer-Verlag.

Digital Library

[29]

Y. Wang and A. Merchant. Proportional-share Scheduling for Distributed Storage Systems. In FAST '07: Proceedings of the 5th USENIX Conference on File and Storage Technologies, pages 4--4, Berkeley, CA, USA, 2007. USENIX Association.

Digital Library

[30]

Y. Wiseman and D. G. Feitelson. Paired Gang Scheduling. IEEE Trans. Parallel Distrib. Syst., 14(6):581--592, 2003.

Digital Library

[31]

T. M. Wong, R. A. Golding, C. Lin, and R. A. Becker-Szendy. Zygaria: Storage Performance as a Managed Resource. In In IEEE Real Time and Embedded Technology and Applications Symposium (RTAS 06, pages 125--134, 2006.

Digital Library

[32]

X. Zhang, K. Davis, and S. Jiang. IOrchestrator: Improving the Performance of Multi-node I/O Systems via Inter-Server Coordination. In Supercomputing '10: Proceedings of the 2010 ACM/IEEE conference on Supercomputing, Washington, DC, USA, 2010. IEEE Computer Society.

Digital Library

Cited By

Bez JByna SIbrahim S(2023)I/O Access Patterns in HPC Applications: A 360-Degree SurveyACM Computing Surveys10.1145/361100756:2(1-41)Online publication date: 15-Sep-2023
https://dl.acm.org/doi/10.1145/3611007
Yang BXue WZhang TLiu SMa XWang XLiu W(2023)End-to-end I/O Monitoring on Leading SupercomputersACM Transactions on Storage10.1145/356842519:1(1-35)Online publication date: 11-Jan-2023
https://dl.acm.org/doi/10.1145/3568425
Zha BShen H(2022)Adaptively Periodic I/O Scheduling for Concurrent HPC ApplicationsElectronics10.3390/electronics1109131811:9(1318)Online publication date: 21-Apr-2022
https://doi.org/10.3390/electronics11091318
Show More Cited By

Index Terms

Server-side I/O coordination for parallel file systems

Recommendations

The Design of the Expand Parallel File System

This article describes an implementation of MPI-IO using a new parallel file system, called Expand Expandable Parallel File System, which is based on NFS servers. Expand combines multiple NFS servers to create a distributed partition where files are ...
I/O Optimizations Based on Workload Characteristics for Parallel File Systems
Network and Parallel Computing
Abstract
Parallel file systems usually provide a unified storage solution, which fails to meet specific application needs. In this paper, we propose an extended file handle scheme to address this problem. It allows the file systems to specify optimizations ...
A prediction-based dynamic file assignment strategy for parallel file systems

An analysis model of file assignment and access are generalized.A load prediction model in parallel file systems is proposed.A prediction-based dynamic file assignment strategy (PDFA) is proposed.We evaluate the effectiveness of the proposed algorithms. ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SC '11: Proceedings of 2011 International Conference for High Performance Computing, Networking, Storage and Analysis

November 2011

866 pages

ISBN:9781450307710

DOI:10.1145/2063384

Conference Chair:
Scott Lathrop
University of Chicago
,
Program Chairs:
Jim Costa
Sandia National Laboratories
,
William Kramer
National Center for Supercomputing Applications

Copyright © 2011 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGARCH: ACM Special Interest Group on Computer Architecture
IEEE-CS: Computer Society

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 12 November 2011

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Conference

SC '11

Sponsor:

SIGARCH
IEEE-CS

SC '11: International Conference for High Performance Computing, Networking, Storage and Analysis

November 12 - 18, 2011

Washington, Seattle

Acceptance Rates

SC '11 Paper Acceptance Rate 74 of 352 submissions, 21%;

Overall Acceptance Rate 1,516 of 6,373 submissions, 24%

Upcoming Conference

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

37
Total Citations
View Citations
338
Total Downloads

Downloads (Last 12 months)12
Downloads (Last 6 weeks)1

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Bez JByna SIbrahim S(2023)I/O Access Patterns in HPC Applications: A 360-Degree SurveyACM Computing Surveys10.1145/361100756:2(1-41)Online publication date: 15-Sep-2023
https://dl.acm.org/doi/10.1145/3611007
Yang BXue WZhang TLiu SMa XWang XLiu W(2023)End-to-end I/O Monitoring on Leading SupercomputersACM Transactions on Storage10.1145/356842519:1(1-35)Online publication date: 11-Jan-2023
https://dl.acm.org/doi/10.1145/3568425
Zha BShen H(2022)Adaptively Periodic I/O Scheduling for Concurrent HPC ApplicationsElectronics10.3390/electronics1109131811:9(1318)Online publication date: 21-Apr-2022
https://doi.org/10.3390/electronics11091318
Pang LKant KScully-Allison CLiem RVeroneze Solorzano A(2022)Server-Side Workload Identification for HPC I/O RequestsProceedings of the 2nd Workshop on Performance EngineeRing, Modelling, Analysis, and VisualizatiOn Strategy10.1145/3526063.3535350(15-22)Online publication date: 30-Jun-2022
https://dl.acm.org/doi/10.1145/3526063.3535350
Hua YShi XHe KJin HXie WHe LChen Y(2022)LoomIO: Object-Level Coordination in Distributed File SystemsIEEE Transactions on Parallel and Distributed Systems10.1109/TPDS.2021.312626033:8(1799-1810)Online publication date: 1-Aug-2022
https://doi.org/10.1109/TPDS.2021.3126260
Patel TGarg RTiwari DNoh SWelch B(2020)GIFTProceedings of the 18th USENIX Conference on File and Storage Technologies10.5555/3386691.3386702(103-120)Online publication date: 24-Feb-2020
https://dl.acm.org/doi/10.5555/3386691.3386702
Kumaresan SShanmugam V(2020)Time-variant attribute-based multitype encryption algorithm for improved cloud data security using user profileThe Journal of Supercomputing10.1007/s11227-019-03118-8Online publication date: 15-Jan-2020
https://doi.org/10.1007/s11227-019-03118-8
Yang BJi XMa XWang XZhang TZhu XEl-Sayed NLan HYang YZhai JLiu WXue WLorch JYu M(2019)End-to-end I/O monitoring on a leading supercomputerProceedings of the 16th USENIX Conference on Networked Systems Design and Implementation10.5555/3323234.3323267(379-394)Online publication date: 26-Feb-2019
https://dl.acm.org/doi/10.5555/3323234.3323267
Huang DLiu QChoi JPodhorszki NKlasky SLogan JOstrouchov GHe XWolf M(2019)Can I/O Variability Be Reduced on QoS-Less HPC Storage Systems?IEEE Transactions on Computers10.1109/TC.2018.288170968:5(631-645)Online publication date: 1-May-2019
https://doi.org/10.1109/TC.2018.2881709
Boito FNou RPilla LBez JMehaut JCortes TNavaux P(2019)On server-side file access pattern matching2019 International Conference on High Performance Computing & Simulation (HPCS)10.1109/HPCS48598.2019.9188092(217-224)Online publication date: Jul-2019
https://doi.org/10.1109/HPCS48598.2019.9188092
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten