research-article

A distributed job scheduling and flow management system

Authors:

Norman Bobroff,

Gargi Dasgupta,

Balaji Viswanathan,

Fabio Benedetti,

Jonathan WagnerAuthors Info & Claims

ACM SIGOPS Operating Systems Review, Volume 42, Issue 1

Pages 63 - 70

https://doi.org/10.1145/1341312.1341324

Published: 01 January 2008 Publication History

Abstract

Grid computing, as a specific model of distributed systems, has sparked recent interest in managing job execution among distributed resource domains. Introduction of the meta-scheduler is a key feature in grid evolution, and the next step is to achieve collaborative interactions between meta-schedulers within and external to organizational boundaries to achieve scalability, balanced resource utilization, and location transparency to job submitters. This paper details a distributed system design that consists of a collaborative meta-scheduling framework, and an expanded resource model with schedulers and data as resources. With this framework, we also explore job scheduling and data management issues, and investigate job flow and meta-scheduling interactions for new applications that require job execution beyond simple sequential and conditional control.

References

[1]

I. Rodero, J. Corbalan, F. Guim, L. L. Fong, Y. G. Liu, and S. Masoud Sadjadi. Looking for an evolution of grid scheduling: Meta-brokering. In Proceedings of the Second CoreGRID Workshop on Middleware at ISC2007, 2007.

[2]

Douglas Thain, Todd Tannenbaum, and Miron Livny. Condor and the grid. In Grid Computing: Making the Global Infrastructure a Reality. 2002.

[3]

E. Huedo, R. S. Montero, and I. M. Llorente. A framework for adaptive execution on grids. Journal of Software -- Practice and Experience, Vol. 34, pp. 631--651, 2004.

Digital Library

[4]

Global Grid Forum. Community scheduler framework(csf). http://www.globus.org/grid_software/computation/csf.php.

[5]

I. Rodero, J. Corbalan, R. M. Badia, and J. Labarta. enanos grid resource broker. In Proceedings of European Grid Conference, 2005.

Digital Library

[6]

IBM. Tivoli workload scheduler loadleveler. http://publib.boulder.ibm.com/-infocenter/clresctr/vxrx/index.jsp, 2006.

[7]

T. Kosar and Miron Livny. Stork: Making data placement a frist class citizen in the grid. In Proceedings of 24th International Conference on Distributed Computing Systems (ICDCS), 2004.

Digital Library

[8]

Y. Zhao, M. Wilde, I. Foster, J. Voeckler, J. Dobson, E. Gilbert, T. Jordan, and E. Quigg. Virtual data grid middleware services for data-intensive science: Research articles. Concurr. Comput.: Pract. Exper., 18, 2006.

Digital Library

[9]

V. Agarwal, G. Dasgupta, K. Dasgupta, A. Purohit, and B. Viswanathan. Deco: Data replication and execution co-scheduling for utility grids. In Proceedings of Fourth International Conference on Service Oriented Computing(ICSOC), 2006.

Digital Library

[10]

M. Cai, A. Chervenak, and M. Frank. A peer-to-peer replica location service based on a distributed hash table. In Proc. of the ACM/IEEE conference on Supercomputing, 2004.

Digital Library

[11]

A. Anjomshoaa, M. Drescher, D. Fellows, A. Ly, S. McGough, D. Pulsipher, and A. Savva. Job submission description language (jsdl) specification, version 1.0, 2005.

[12]

T. Oinn, M. Addis, J. Ferris, D. Marvin, M. Senger, M. Greenwood, T. Carver, K. Glover, M. R. Pocock, A. Wipat, and P. Li. Taverna: a tool for the composition and enactment of bioinformatics workflows. Bioinformatics, 20(17):3045--3054, 2004.

Digital Library

[13]

E. Deelman, G. Singh, M. H. Su, J. Blythe, Y. Gil, C. Kesselman, G. Mehta, K. Vahi, G. B. Berriman, and J. Good. A framework for mapping complex scientific workflows onto distributed systems. Scientific Programming, 13, 2005.

Digital Library

[14]

F. Berman, A. Chien, K. Cooper, J. Dongarra, I. Foster, D. Gannon, L. Johnsson, K. Kennedy, C. Kesselman, J. Mellor-Crummey, D. Reed, L. Torczon, and R. Wolski. The grads project: Software support for high-level grid application development. International Journal of High Performance Computing Applications, 15(4):327--344, 2001.

Digital Library

[15]

I. Taylor, M. Shields, and I. Wang. Resource Management of Triana P2P Services. 2003.

[16]

Tristan Glatard, Johan Montagnat, Diane Lingrand, and Xavier Pennec. Flexible and efficient workflow deployement of data-intensive applications on grids with MOTEUR. International Journal of High Performance Computing and Applications (IJHPCA), 2007.

Digital Library

[17]

T. Andrews, F. Curbera, Siebel Systems Y. Goland H. Dholakia, J. Klein, F. Leymann, Kevin Liu, Dieter Roller, Doug Smith, Ivana Trickovic, and Sanjiva Weerawarana. Business process execution language for web services. http://www-128.ibm.com/developerworks/library/ws-bpel/, 2003.

[18]

F. Leymann. Choreography for the grid: towards fitting bpel to the resource framework: Research articles. Concurr. Comput.: Pract. Exper., 18(10), 2006.

Digital Library

[19]

V. Gucer, M. A. Lowry, and F. B. Knudsen. End-to-End Scheduling with IBM Tivoli Workload Scheduler Version. IBM Press, 2004.

[20]

W. Tan, L. Fong, and N. Bobroff. Bpel4job: a fault-handling design for job flow management. In Proceedings of Fifth International Conference on Service Oriented Computing(ICSOC), 2007.

Digital Library

[21]

O. Ezenwoye and S. M. Sadjadi. Trap/bpel: A framework for dynamic adaptation of composite services. In Proceedings of International Conference on Web Information Systems and Technologies (WEBIST), 2007.

[22]

G. Dasgupta, K. Dasgupta, and B. Viswanathan. Data-wise: Efficient management of data-intensive workflows in scheduled grid environments. In submission, 2008.

[23]

IBM. Ibm tivoli dynamic workload broker. http://www-306.ibm.com/software/tivoli/products/dynamic-workload-broker/index.html, 2006.

[24]

IBM and FIU. Latin american grid. http://www.latinamericangrid.org/, 2006.

[25]

N. Bobroff, G. Dasgupta, and et. al. Enabling Interoperability among Meta-Schedulers. In Preparation.

[26]

G. Dasgupta and B. Viswanathan. Inform: Integrated flow orchestration and meta-scheduling for managed grid systems. In Proceedings of 8th International Middleware Conference, 2007.

Digital Library

[27]

G. V. Putte and L. Gavin. Technical overview of websphere process server and websphere integration developer. www.ibm.com/redbook, 2005.

Cited By

Rodero IGuim FCorbalan JFong LSadjadi S(2010)Grid broker selection strategies using aggregated resource informationFuture Generation Computer Systems10.1016/j.future.2009.07.00926:1(72-86)Online publication date: 1-Jan-2010
https://dl.acm.org/doi/10.1016/j.future.2009.07.009
Liu YBobroff NFong LSeelam SVillegas DSadjadi SRodero IEpema DMoreira JWilliamson C(2009)An experimental system for grid meta-broker evaluationProceedings of the 1st ACM workshop on Large-Scale system and application performance10.1145/1552272.1552276(11-18)Online publication date: 10-Jun-2009
https://dl.acm.org/doi/10.1145/1552272.1552276
Rodero IGuim FCorbalan JFong LSadjadi S(2009)Broker Selection Strategies in Interoperable Grid SystemsProceedings of the 2009 International Conference on Parallel Processing10.1109/ICPP.2009.63(180-187)Online publication date: 22-Sep-2009
https://dl.acm.org/doi/10.1109/ICPP.2009.63

Index Terms

A distributed job scheduling and flow management system
1. Software and its engineering
  1. Software creation and management
    1. Designing software
  2. Software organization and properties
    1. Contextual software domains
      1. Operating systems
        Process management

Recommendations

Multi-Level Job Flow Cyclic Scheduling in Grid Virtual Organizations

Distributed environments with the decoupling of users from resource providers are generally termed as utility Grids. The paper focuses on the problems of efficient job flow distribution and scheduling in virtual organizations (VOs) of utility Grids ...
Job Ranking and Scheduling in Utility Grids VOs
Proceedings of the 13th International Conference on Parallel Computing Technologies - Volume 9251

In this work, we propose approaches to creation of a ranked jobs framework within a model of cycle scheduling in virtual organizations of utility Grids with the decoupling of users from resource providers. Two methods for job selection and scheduling ...
Using the black-box approach with machine learning methods in order to improve job scheduling in GRID environments

This article focuses on mapping jobs to resources with use of off-the-shelf machine learning methods. The machine learning methods are used in the black-box manner, having a wide variety of parameters for internal cross validation. In the article we ...

Comments

Information & Contributors

Information

Published In

cover image ACM SIGOPS Operating Systems Review

ACM SIGOPS Operating Systems Review Volume 42, Issue 1

January 2008

133 pages

ISSN:0163-5980

DOI:10.1145/1341312

Issue’s Table of Contents

Copyright © 2008 Authors.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 January 2008

Published in SIGOPS Volume 42, Issue 1

Check for updates

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
806
Total Downloads

Downloads (Last 12 months)9
Downloads (Last 6 weeks)1

Reflects downloads up to 07 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Rodero IGuim FCorbalan JFong LSadjadi S(2010)Grid broker selection strategies using aggregated resource informationFuture Generation Computer Systems10.1016/j.future.2009.07.00926:1(72-86)Online publication date: 1-Jan-2010
https://dl.acm.org/doi/10.1016/j.future.2009.07.009
Liu YBobroff NFong LSeelam SVillegas DSadjadi SRodero IEpema DMoreira JWilliamson C(2009)An experimental system for grid meta-broker evaluationProceedings of the 1st ACM workshop on Large-Scale system and application performance10.1145/1552272.1552276(11-18)Online publication date: 10-Jun-2009
https://dl.acm.org/doi/10.1145/1552272.1552276
Rodero IGuim FCorbalan JFong LSadjadi S(2009)Broker Selection Strategies in Interoperable Grid SystemsProceedings of the 2009 International Conference on Parallel Processing10.1109/ICPP.2009.63(180-187)Online publication date: 22-Sep-2009
https://dl.acm.org/doi/10.1109/ICPP.2009.63

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Issue’s Table of Contents