Evolving ensemble of classifiers in random subspace

ABSTRACT

Various methods for ensemble selection and classifier combination have been designed to optimize the results of ensembles of classifiers. Genetic algorithm (GA) which uses the diversity for the ensemble selection could be very time consuming. We propose compound diversity functions as objective functions for a faster and more effective GA searching. Classifiers selected by GA are combined by a proposed pairwise confusion matrix transformation, which offer strong performance boost for EoCs.

References

1. L. I. Kuncheva and C. J. Whitaker, "Measures of Diversity in Classifier Ensembles and Their Relationship with the Ensemble Accuracy," Machine Learning, vol. 51, no. 2, pp. 181--207, 2003]]
2. T. K. Ho, "The random space method for constructing decision forests," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 20, no. 8, pp. 832--844, 1998]]
3. D. Ruta and B. Gabrys, "Analysis of the Correlation between Majority Voting Error and the Diversity Measures in Multiple Classifier Systems," In Proceedings of the 4th International Symposium on Soft Computing, 2001]]
4. G. Giacinto and F. Roli, "Design of effective neural network ensembles for image classification purposes," Image and Vision Computing, vol. 19, no. 9-10, pp. 699--707, 2001]]
5. K. Turner and J. Ghosh, "Error Correlation and Error Reduction in Ensemble Classifiers," Connection Science, vol. 8, no. 3-4, pp. 385--404, 1996]]
6. J. L. Fleiss, B. Levin, and M. C. Paik, "Statistical Methods for Rates and Proportions," Second Edition, New York: John Wiley & Sons, 2003]]
7. G. Tremblay, R. Sabourin and P. Maupin, "Optimizing Nearest Neighbour in Random Subspace using a Multi-Objective Genetic Algorithm," In Proceedings of the 17th International Conference on Pattern Recognition (ICPR 2004), pp 208--211, 2004]]
8. D. M. J. Tax, M. Van Breukelen, R. P. W. Duin and J. Kittler, "Combining Multiple Classifiers by Averaging or by Multiplying," Pattern Recognition, vol.33, no. 9, pp.1475--1485, 2000]]
9. C. A. Shipp and L.I. Kuncheva, "Relationships between combination methods and measures of diversity in combining classifiers," International Journal of Information Fusion, vol.3, no. 2, pp. 135--148, 2002]]
10. Y. S. Huang and C.Y. Suen, "A method of combining multiple experts for the recognition of unconstrained handwritten numerals," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 17, pp. 90--93, 1995]]
11. K. D. Wernecke, "A coupling procedure for discrimination of mixed data," Biometrics, vol. 48, pp. 97--506, 1992]]
12. L.I. Kuncheva, J.C. Bezdek and R.P.W. Duin, "Decision templates for multiple classifier fusion: an experimental comparison," Pattern Recognition, vol. 34, no. 2, pp. 299--314, 2001]]
13. L. I. Kuncheva, M. Skurichina and R. P. W. Duin, "An Experimental Study on Diversity for Bagging and Boosting with Linear Classifiers," International Journal of Information Fusion, vol. 3, no. 2, pp. 245--258, 2002]]
14. J. Kittler, M. Hatef, R. Duin and J. Matas, "On Combining Classifiers," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 20, no. 3, pp. 226--239, 1998]]
15. R. E. Banfield, L. O. Hall, K. W. Bowyer and W. P. Kegelmeyer, "A New Ensemble Diversity Measure Applied to Thinning Ensembles," International Workshop on Multiple Classifier Systems (MCS 2003), pp. 306 -- 316, 2003]]
16. L. Xu, A. Krzyzak and C. Y. Suen, "Methods of combining multiple classifiers and their applications to handwriting recognition," IEEE Transactions on Systems, Man and Cybernetics, vol. 22, no. 3, pp. 418--435, 1992]]
17. R. P. W. Duin, "The Combining Classifier: To Train or Not to Train?" 16th International Conference on Pattern Recognition (ICPR), vol. 2, pp. 20765, 2002]]
18. L. K. Hansen, C. Liisberg and P. Salamon, "The error-reject tradeoff," Open Systems and Information Dynamics, vol. 4, pp. 159--184, 1997]]
19. A. E. Eiben, R. Hinterding, and Z. Michalewicz, "Parameter control in evolutionary algorithms", In IEEE Transactions on Evolutionary Computation, vol.3, no. 2, pp. 124--141, 1998]]
20. L. I. Kuncheva, "A Theoretical Study on Six Classifier Fusion Strategies," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 24, no. 2, pp. 281--286, 2002]]
21. J. Milgram, M. Cheriet and R. Sabourin, "Estimating Accurate Multi-class Probabilities with Support Vector Machines," International Joint Conference on Neural Networks 2005 (IJCNN 2005), pp. 1906--1911, 2005.]]
22. S. Raudys, "Experts' boasting in trainable fusion rules," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 25, no. 9, pp. 1178 -- 1182, 2003]]
23. C. A. Shipp and L. I. Kuncheva, "Relationships Between Combination Methods and Measures of Diversity in Combining Classifiers," International Journal of Information Fusion, vol. 3, no. 2, pp. 135 -- 148, 2002]]
24. R.P.W. Duin, "Pattern Recognition Toolbox for Matlab 5.0+," available free at: ftp://ftp.ph.tn.tudelft.nl/pub/bob/prtools]]
25. D. Ruta and B. Gabrys, "Classifier Selection for Majority Voting," International Journal of Information Fusion, pp. 63--81, 2005]]
26. J. Kittler and F. M. Alkoot, "Relationship of Sum and Vote Fusion Strategies," Multiple Classifier Systems (MCS), pp. 339--348, 2001]]
27. H. Zouari, L. Heutte, Y. Lecourtier and A. Alimi, "Building Diverse Classifier Outputs to Evaluate the Behavior of Combination Methods: the Case of Two Classifiers," Multiple Classifier Systems (MCS), pp. 273--282, 2004]]

Index Terms

• Computing methodologies

  • Machine learning

    • Learning paradigms

      • Supervised learning

        • Supervised learning by classification

    • Machine learning approaches

      • Classification and regression trees