Robustness of adaptive filtering methods in a cross-benchmark evaluation

ABSTRACT

This paper reports a cross-benchmark evaluation of regularized logistic regression (LR) and incremental Rocchio for adaptive filtering. Using four corpora from the Topic Detection and Tracking (TDT) forum and the Text Retrieval Conferences (TREC) we evaluated these methods with non-stationary topics at various granularity levels, and measured performance with different utility settings. We found that LR performs strongly and robustly in optimizing T11SU (a TREC utility function) while Rocchio is better for optimizing Ctrk (the TDT tracking cost), a high-recall oriented objective function. Using systematic cross-corpus parameter optimization with both methods, we obtained the best results ever reported on TDT5, TREC10 and TREC11. Relevance feedback on a small portion (0.05~0.2%) of the TDT5 test documents yielded significant performance improvements, measuring up to a 54% reduction in Ctrk and a 20.9% increase in T11SU (with b=0.1), compared to the results of the top-performing system in TDT2004 without relevance feedback information.

REFERENCES

• 1: James Allan, Incremental relevance feedback for information filtering, Proceedings of the 19th annual international ACM SIGIR conference on Research and development in information retrieval, p.270-278, August 18-22, 1996, Zurich, Switzerland [doi>10.1145/243199.243274]
• 2: Jamie Callan, Learning while filtering documents, Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, p.224-231, August 24-28, 1998, Melbourne, Australia [doi>10.1145/290941.290998]
• 3: Jonathan G. Fiscus , George R. Doddington, Topic detection and tracking evaluation overview, Topic detection and tracking: event-based information organization, Kluwer Academic Publishers, Norwell, MA, 2002
• 4: J. Fiscus and B. Wheatley. Overview of the TDT 2004 Evaluation and Results. In TDT-04, 2004.
• 5: T. Hastie, R. Tibshirani and J. Friedman. Elements of Statistical Learning. Springer, 2001.
• 6: S. Robertson and D. Hull. The TREC-9 filtering track final report. In TREC-9, 2000.
• 7: S. Robertson and I. Soboroff. The TREC-10 filtering track final report. In TREC-10, 2001.
• 8: S. Robertson and I. Soboroff. The TREC 2002 filtering track report. In TREC-11, 2002.
• 9: S. Robertson and S. Walker. Microsoft Cambridge at TREC-9. In TREC-9, 2000.
• 10: Robert E. Schapire , Yoram Singer , Amit Singhal, Boosting and Rocchio applied to text filtering, Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, p.215-223, August 24-28, 1998, Melbourne, Australia [doi>10.1145/290941.290996]
• 11: Yiming Yang , Bryan Kisiel, Margin-based local regression for adaptive filtering, Proceedings of the twelfth international conference on Information and knowledge management, November 03-08, 2003, New Orleans, LA, USA [doi>10.1145/956863.956902]
• 12: Yi Zhang , Jamie Callan, Maximum likelihood estimation for filtering thresholds, Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval, p.294-302, September 2001, New Orleans, Louisiana, United States [doi>10.1145/383952.384012]
• 13: Yi Zhang, Using bayesian priors to combine classifiers for adaptive filtering, Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval, July 25-29, 2004, Sheffield, United Kingdom [doi>10.1145/1008992.1009052]
• 14: Jian Zhang , Yiming Yang, Robustness of regularized linear classification methods in text categorization, Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval, July 28-August 01, 2003, Toronto, Canada [doi>10.1145/860435.860471]
• 15: Tong Zhang , Frank J. Oles, Text Categorization Based on Regularized Linear Classification Methods, Information Retrieval, v.4 n.1, p.5-31, April 2001 [doi>10.1023/A:1011441423217 ]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Information filtering

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Selection process; Retrieval models; Relevance feedback

  I. Computing Methodologies
  I.5 PATTERN RECOGNITION
      I.5.2 Design Methodology
          Subjects: Classifier design and evaluation

General Terms:
Algorithms, Experimentation, Measurement, Performance

Keywords:
adaptive filtering, cross-benchmark evaluations, logistic regression, rocchio, topic tracking