Mining for misconfigured machines in grid systems

ABSTRACT

Grid systems are proving increasingly useful for managing the batch computing jobs of organizations. One well-known example is Intel, whose internally developed NetBatch system manages tens of thousands of machines. The size, heterogeneity, and complexity of grid systems make them very difficult, however, to configure. This often results in misconfigured machines, which may adversely affect the entire system.We investigate a distributed data mining approach for detection of misconfigured machines. Our Grid Monitoring System (GMS) non-intrusively collects data from all sources (log files, system services, etc.) available throughout the grid system. It converts raw data to semantically meaningful data and stores this data on the machine it was obtained from, limiting incurred overhead and allowing scalability. Afterwards, when analysis is requested, a distributed outliers detection algorithm is employed to identify misconfigured machines. The algorithm itself is implemented as a recursive workflow of grid jobs. It is especially suited to grid systems, in which the machines might be unavailable most of the time and often fail altogether.

REFERENCES

• 1: Fabrizio Angiulli , Clara Pizzuti, Fast Outlier Detection in High Dimensional Spaces, Proceedings of the 6th European Conference on Principles of Data Mining and Knowledge Discovery, p.15-26, August 19-23, 2002
• 2: Jim Basney , Miron Livny, Improving Goodput by Coscheduling CPU and Network Capacity, International Journal of High Performance Computing Applications, v.13 n.3, p.220-230, August 1999 [doi>10.1177/109434209901300305]
• 3: J. W. Branch, B. Szymanski, C. Giannella, R. Wolff, and H. Kargupta. In-network outlier detection in wireless sensor networks. In Proc. of ICDCS, July 2006.
• 4: M. Cannataro, A. Massara, and P. Veltri. The OnBrowser ontology manager: Managing ontologies on the grid. In Intl. Workshop on Semantic Intelligent Middleware for the Web and the Grid, 2004.
• 5: Alice X. Zheng , Jim Lloyd , Eric Brewer, Failure Diagnosis Using Decision Trees, Proceedings of the First International Conference on Autonomic Computing (ICAC'04), p.36-43, May 17-18, 2004
• 6: Victoria Hodge , Jim Austin, A Survey of Outlier Detection Methodologies, Artificial Intelligence Review, v.22 n.2, p.85-126, October 2004 [doi>10.1023/B:AIRE.0000045502.10941.a9]
• 7: M. J. Litzkow, M. Livny, and M. W. Mutka. Condor - A hunter of idle workstations. In Proc. of ICDCS, June 1988.

INDEX TERMS

Primary Classification:
  C. Computer Systems Organization
  C.2 COMPUTER-COMMUNICATION NETWORKS
      C.2.4 Distributed Systems

Additional Classification:
  H. Information Systems
  H.2 DATABASE MANAGEMENT
      H.2.8 Database applications
          Subjects: Data mining

General Terms:
Algorithms, Human Factors, Management, Performance

Keywords:
distributed data mining, grid information system, grid systems, outliers detection, system monitoring