Disk Failure Prediction in Data Centers via Online Learning

ABSTRACT

Disk failure has become a major concern with the rapid expansion of storage systems in data centers. Based on SMART (Self-Monitoring, Analysis and Reporting Technology) attributes, many researchers derive disk failure prediction models using machine learning techniques. Despite the significant developments, the majority of works rely on offline training and thereby hinder their adaption to the continuous update of forthcoming data, suffering from the 'model aging' problem. We are therefore motivated to uncover the root cause -- the dynamic SMART distribution for 'model aging', aiming to resolve the performance degradation as to pave a comprehensive study in practice.

In this paper, we introduce a novel disk failure prediction model using Online Random Forests (ORFs). Our ORF-based model can automatically evolve with sequential arrival of data on-the-fly and thus is highly adaptive to the variance of SMART distribution over time. Moreover, it has favourable advantage against the offline counterparts in terms of superior prediction performance. Experiments on real-world datasets show that our ORF model converges rapidly to the offline random forests and achieves stable failure detection rates of 93-99% with low false alarm rates. Furthermore, we demonstrate the ability of our approach on maintaining stable prediction performance for the long-term usage in data centers.

References

1. B. Schroeder and G. A. Gibson. Disk failures in the real world: What does an MTTF of 1,000,000 hours mean to you? In Proceedings of 5th USENIX Conference on File and Storage Technologies (FAST'07), pp. 1--16, Feb. 2007.
2. I. Manousakis, S. Sankar, G. McKnight, Thu D. Nguyen, and R. Bianchini. Environmental Conditions and Disk Reliability in Free-Cooled Datacenters. In Proceedings of the 14th USENIX Conference on File and Storage Technologies (FAST'16), pp. 53--65, Feb. 2016.
3. Data center downtime costs. http://www.emerson.com/en-us/News/Pages/Net-Power-Study-Data-Center.aspx.
4. L. N. Bairavasundaram, G. R. Goodson, B. Schroeder, A. C. Arpaci-Dusseau, and R. H. Arpaci-Dusseau. An analysis of data corruption in the storage stack. In Proceedings of the 6th USENIX Conference on File and Storage Technologies (FAST'08), Feb. 2008.
5. E. Brewer, L. Ying, L. Greenfield, R. Cypher, and Theodore. Disks for Data Centers. Keynote talk for FAST'16, Technical report, Google, Feb. 2016.
6. P. M. Chen, E. K. Lee, G. A. Gibson, R. H. Katz, and D. A. Patterson. RAID: High-performance, reliable secondary storage. In ACM Computing Surveys (CSUR), vol 26, no. 2, pp. 145--185, June, 1994.
7. S.M.A.R.T. https://en.wikipedia.org/wiki/S.M.A.R.T.
8. G. Hamerly and C. Elkan. Bayesian approaches to failure prediction for disk drives. In Proceedings of the 18th International Conference on Machine Learning (ICML'01), pp. 202--209, Jun. 2001.
9. G. F. Hughes, J. F. Murray, K. Kreutz-Delgado, and C. Elkan. Improved disk-drive failure warnings. In IEEE Transactions on Reliability, vol. 51, no. 3, pp. 350--357, Sept. 2002.
10. J. F. Murray, G. F. Hughes, and K. Kreutz-Delgado. Machine learning methods for predicting failures in hard drives: A multiple instance application. In Journal of Machine Learning Research, vol. 6, pp. 783--816, May. 2005.
11. B. Zhu, G. Wang, X. Liu, D. Hu, S. Lin, and J. Ma. Proactive drive failure prediction for large scale storage systems. In Proceedings of 29th IEEE Conference on Massive Storage Systems and Technologies (MSST), pp. 1--5, May. 2013.
12. Y. Wang, Q. Miao, E. W. Ma, K.-L. Tsui, and M. G. Pecht. Online anomaly detection for hard disk drives based on mahalanobis distance. In IEEE Transactions on Reliability, vol. 62, no. 1, pp. 136--145, Mar. 2013.
13. Y. Wang, W.M. Ma, W. S. Chow, and K.-L. Tsui. A Two-Step Parametric Method for Failure Prediction in Hard Disk Drives. In IEEE Transactions on Industrial Informatics, vol. 10, no. 1, pp. 419--430, Feb. 2014.
14. B. Zhu, G. Wang, X. Ji, J. Li, and Y. Jia. Hard Drive Failure Prediction Using Classification and Regression Trees. In Proceedings of 44th IEEE Conference on Dependable Systems and Networks (DSN), pp. 383--394, Dec. 2014.
15. C. Xu, G. Wang, X. Liu, D. Guo, and T. Liu. Health Status Assessment and Failure Prediction for Hard Drives with Recurrent Neural Networks. In IEEE Transactions on Computers, vol. 65, no. 11, pp. 3502--3508, Nov. 2016.
16. J. Li, R. J. Stones, G. Wang, X. Liu, Z. Li, and M. Xu. Hard drive failure prediction using Decision Trees. In Reliability Engineering and System Safety, vol. 164, pp. 55--65, Mar. 2017.
17. J. Li, R. J. Stones, G. Wang, Z. Li, X. Liu, and X. Kang. Being accurate is not enough: New metrics for disk failure prediction. In Proceedings Symposium on Reliable Distributed Systems (SRDS), pp. 71--80, Dec. 2016.
18. O. Fontenla-Romero, B. Guijarro-Berdinas, D. Martinez-Rego, B. Perez-Sanchez, and D. Peteiro-Barral. Online machine learning. In Efficiency and Scalability Methods for Computational Intellect, pp. 27--54, 2013.
19. The backblaze hard drive data and stats. https://www.backblaze.com/b2/hard-drive-test-data.html.
20. M. Botezatu, I. Giurgiu, J. Bogojeska, and D. Wiesmann. Predicting Disk Replacement towards Reliable Data Centers. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD), pp. 39--48, Aug. 2016.
21. N. C. Oza and S. J. Russell. Online bagging and boosting. In Artificial Intelligence and Statistics, pp. 105--112, 2001.
22. A. Saffari, C. Leistner, J. Santner, M. Godec, and H. Bischof. On-line Random Forests. In Proceedings of 12th International Conference on Computer Vision Workshops (ICCV), pp. 1393--1400, 2009.
23. V. Agrawal, C. Bhattacharyya, T. Niranjan, and S. Susarla. Discovering rules from disk events for predicting hard drive failures. In Proceedings of International Conference on Machine Learning and Applications, pp. 782--786, Dec. 2009.
24. L. Breiman. Random forests. Machine Learning, vol. 45, pp. 5--32, Oct. 2001.
25. F. Mahdisoltani, I. Stefanovici, and B. Schroeder. Proactive error prediction to improve storage system reliability. In Proceedings of the 2017 USENIX Annual Technical Conference (USENIX ATC '17), pp. 391--402, Jul. 2017.
26. L. Breiman. Out-of-bag Estimation. Citeseer, 1996.
27. E. Pinheiro, W.-D. Weber, and L. A. Barroso. Failure trends in a large disk drive population. In Proceedings of 5th USENIX Conference on File and Storage Technologies (FAST'07), pp. 17--29, Feb. 2007.
28. C.-C. Chang and C.-J. Lin. LIBSVM: A library for support vector machines. In ACM Transactions on Intelligent Systems and Technology, vol. 2, no. 3, pp. 27:1--27:27, Apr. 2011.
29. Fit binary classification tree for multiclass classification. https://cn.mathworks.com/help/stats/fitctree.html.