Evolving ensemble of classifiers in random subspace

ABSTRACT

Various methods for ensemble selection and classifier combination have been designed to optimize the results of ensembles of classifiers. Genetic algorithm (GA) which uses the diversity for the ensemble selection could be very time consuming. We propose compound diversity functions as objective functions for a faster and more effective GA searching. Classifiers selected by GA are combined by a proposed pairwise confusion matrix transformation, which offer strong performance boost for EoCs.

REFERENCES

• 1: Ludmila I. Kuncheva , Christopher J. Whitaker, Measures of Diversity in Classifier Ensembles and Their Relationship with the Ensemble Accuracy, Machine Learning, v.51 n.2, p.181-207, May 2003 [doi>10.1023/A:1022859003006]
• 2: Tin Kam Ho, The Random Subspace Method for Constructing Decision Forests, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.20 n.8, p.832-844, August 1998 [doi>10.1109/34.709601 ]
• 3: D. Ruta and B. Gabrys, "Analysis of the Correlation between Majority Voting Error and the Diversity Measures in Multiple Classifier Systems," In Proceedings of the 4th International Symposium on Soft Computing, 2001
• 4: G. Giacinto and F. Roli, "Design of effective neural network ensembles for image classification purposes," Image and Vision Computing, vol. 19, no. 9-10, pp. 699--707, 2001
• 5: K. Turner and J. Ghosh, "Error Correlation and Error Reduction in Ensemble Classifiers," Connection Science, vol. 8, no. 3-4, pp. 385--404, 1996
• 6: J. L. Fleiss, B. Levin, and M. C. Paik, "Statistical Methods for Rates and Proportions," Second Edition, New York: John Wiley & Sons, 2003
• 7: Guillaume Tremblay , Robert Sabourin , Patrick Maupin, Optimizing Nearest Neighbour in Random Subspaces using a Multi-Objective Genetic Algorithm, Proceedings of the Pattern Recognition, 17th International Conference on (ICPR'04) Volume 1, p.208, August 23-26, 2004 [doi>10.1109/ICPR.2004.662]
• 8: D. M. J. Tax, M. Van Breukelen, R. P. W. Duin and J. Kittler, "Combining Multiple Classifiers by Averaging or by Multiplying," Pattern Recognition, vol.33, no. 9, pp.1475--1485, 2000
• 9: C. A. Shipp and L.I. Kuncheva, "Relationships between combination methods and measures of diversity in combining classifiers," International Journal of Information Fusion, vol.3, no. 2, pp. 135--148, 2002
• 10: Y. S. Huang , C. Y. Suen, A Method of Combining Multiple Experts for the Recognition of Unconstrained Handwritten Numerals, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.17 n.1, p.90-94, January 1995 [doi>10.1109/34.368145 ]
• 11: K. D. Wernecke, "A coupling procedure for discrimination of mixed data," Biometrics, vol. 48, pp. 97--506, 1992
• 12: L.I. Kuncheva, J.C. Bezdek and R.P.W. Duin, "Decision templates for multiple classifier fusion: an experimental comparison," Pattern Recognition, vol. 34, no. 2, pp. 299--314, 2001
• 13: L. I. Kuncheva, M. Skurichina and R. P. W. Duin, "An Experimental Study on Diversity for Bagging and Boosting with Linear Classifiers," International Journal of Information Fusion, vol. 3, no. 2, pp. 245--258, 2002
• 14: Josef Kittler , Mohamad Hatef , Robert P. W. Duin , Jiri Matas, On Combining Classifiers, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.20 n.3, p.226-239, March 1998 [doi>10.1109/34.667881 ]
• 15: R. E. Banfield, L. O. Hall, K. W. Bowyer and W. P. Kegelmeyer, "A New Ensemble Diversity Measure Applied to Thinning Ensembles," International Workshop on Multiple Classifier Systems (MCS 2003), pp. 306 -- 316, 2003
• 16: L. Xu, A. Krzyzak and C. Y. Suen, "Methods of combining multiple classifiers and their applications to handwriting recognition," IEEE Transactions on Systems, Man and Cybernetics, vol. 22, no. 3, pp. 418--435, 1992
• 17: R. P. W. Duin, "The Combining Classifier: To Train or Not to Train?" 16th International Conference on Pattern Recognition (ICPR), vol. 2, pp. 20765, 2002
• 18: Lars Kai Hansen , Christian Liisberg , Peter Salamon, The Error-Reject Tradeoff, Open Systems & Information Dynamics, v.4 n.2, p.159-184, April 1997 [doi>10.1023/A:1009643503022 ]
• 19: A. E. Eiben, R. Hinterding, and Z. Michalewicz, "Parameter control in evolutionary algorithms", In IEEE Transactions on Evolutionary Computation, vol.3, no. 2, pp. 124--141, 1998
• 20: Ludmila I. Kuncheva, A Theoretical Study on Six Classifier Fusion Strategies, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.24 n.2, p.281-286, February 2002 [doi>10.1109/34.982906 ]
• 21: J. Milgram, M. Cheriet and R. Sabourin, "Estimating Accurate Multi-class Probabilities with Support Vector Machines," International Joint Conference on Neural Networks 2005 (IJCNN 2005), pp. 1906--1911, 2005.
• 22: Sarunas Raudys, Experts' Boasting in Trainable Fusion Rules, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.25 n.9, p.1178-1182, September 2003 [doi>10.1109/TPAMI.2003.1227993]
• 23: C. A. Shipp and L. I. Kuncheva, "Relationships Between Combination Methods and Measures of Diversity in Combining Classifiers," International Journal of Information Fusion, vol. 3, no. 2, pp. 135 -- 148, 2002
• 24: R.P.W. Duin, "Pattern Recognition Toolbox for Matlab 5.0+," available free at: ftp://ftp.ph.tn.tudelft.nl/pub/bob/prtools
• 25: D. Ruta and B. Gabrys, "Classifier Selection for Majority Voting," International Journal of Information Fusion, pp. 63--81, 2005
• 26: Josef Kittler , Fuad M. Alkoot, Relationship of Sum and Vote Fusion Strategies, Proceedings of the Second International Workshop on Multiple Classifier Systems, p.339-348, July 02-04, 2001
• 27: H. Zouari, L. Heutte, Y. Lecourtier and A. Alimi, "Building Diverse Classifier Outputs to Evaluate the Behavior of Combination Methods: the Case of Two Classifiers," Multiple Classifier Systems (MCS), pp. 273--282, 2004

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.5 PATTERN RECOGNITION
      I.5.2 Design Methodology
          Subjects: Classifier design and evaluation

General Terms:
Algorithms

Keywords:
combining classifiers, confusion matrix, diversity, ensemble of learning machines, fusion function, majority voting, pattern recognition