A methodology for comparing classifiers that allow the control of bias

ABSTRACT

This paper presents False Positive-Critical Classifiers Comparison a new technique for pairwise comparison of classifiers that allow the control of bias. An evaluation of Naïve Bayes, k-Nearest Neighbour and Support Vector Machine classifiers has been carried out on five datasets containing unsolicited and legitimate e-mail messages to confirm the advantage of the technique over Receiver Operating Characteristic curves. The evaluation results suggest that the technique may be useful for choosing the better classifier when the ROC curves do not show comprehensive differences, as well as to prove that the difference between two classifiers is not significant, when ROC suggests that it might be. Spam filtering is a typical application for such a comparison tool, as it requires a classifier to be biased toward negative prediction and to have some upper limit on the rate of false positives. Finally the particular evaluation summary is presented, which confirms that Support Vector Machines out-perform other methods in most cases, while the Naïve Bayes classifier works well in a narrow, but relevant range of false positive rate.

REFERENCES

• 1: I. Androutsopoulos, J. Koutsias, G. Paliouras, V. Karkaletsis, G. Sakkis, and C. Spyropoulos. Learning to filter spam email: A comparison of a naive bayesian and a memory based approach. In H. Zaragoza, P. Gallinari, and M. Rajman, editors, Procs of Workshop on Machine Learning and Textual Information Access, PKDD 2000, pages 1--13, 2000.
• 2: A. P. Bradley. The use of the area under the curve in the evaluation of machine learning algorithms. Pattern Recognition, 30(6):1145--1157, 1997.
• 3: Nello Cristianini , John Shawe-Taylor, An introduction to support Vector Machines: and other kernel-based learning methods, Cambridge University Press, New York, NY, 1999
• 4: S. J. Delany and P. Cunningham. An analysis of case-based editing in a spam filtering system. In P. Funk and P. González-Calero, editors, 7th European Conference on Case-Based Reasoning (ECCBR 2004), volume 3155 of LNAI, pages 128--141. Springer, 2004.
• 5: Thomas G. Dietterich, Approximate statistical tests for comparing supervised classification learning algorithms, Neural Computation, v.10 n.7, p.1895-1923, Oct. 1998 [doi>10.1162/089976698300017197 ]
• 6: H. Drucker, V. Vapnik, and D. Wu. Support vector machines for spam categorization. IEEE Transactions on Neural Networks, 10(5):1048--1054, 1999.
• 7: José M. Gómez Hidalgo , Manuel Maña López , Enrique Puertas Sanz, Combining text and heuristics for cost-sensitive spam filtering, Proceedings of the 2nd workshop on Learning language in logic and the 4th conference on Computational natural language learning, September 13-14, 2000, Lisbon, Portugal [doi>10.3115/1117601.1117623]
• 8: R. Kohavi, B. Becker, and D. Sommerfield. Improving simple bayes. In Procs of the 9th European Conf. on Machine Learning (ECML 97). Springer Verlag, 1997.
• 9: T. Niblett. Constructing decision trees in noisy domains. In I. Bratko and N. Lavrac, editors, Progress in Machine Learning, Procs of 2nd European Working Session on Learning (EWSL 87), pages 67--78. Sigma Press, 1987.
• 10: M. Sahami, S. Dumais, D. Heckerman, and E. Horvitz. A bayesian approach to filtering junk E-mail. In Learning for Text Categorization: Papers from the 1998 Workshop, Madison, Wisconsin, 1998. AAAI Technical Report WS-98-05.
• 11: J. Shawe-Taylor and N. Cristianini. Margin distribution and soft margin, 2000.
• 12: J. A. Swets. Measuring the accuracy of diagnostic systems. Science, (240):1285--1293, 1988.

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.5 PATTERN RECOGNITION
      I.5.2 Design Methodology
          Subjects: Classifier design and evaluation

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Information filtering

  I. Computing Methodologies
  I.5 PATTERN RECOGNITION
      I.5.2 Design Methodology
          Subjects: Pattern analysis

General Terms:
Algorithms, Design