Using titles and category names from editor-driven taxonomies for automatic evaluation

ABSTRACT

Evaluation of IR systems has always been difficult because of the need for manually assessed relevance judgments. The advent of large editor-driven taxonomies on the web opens the door to a new evaluation approach. We use the ODP (Open Directory Project) taxonomy to find sets of pseudo-relevant documents via one of two assumptions: 1) taxonomy entries are relevant to a given query if their editor-entered titles exactly match the query, or 2) all entries in a leaf-level taxonomy category are relevant to a given query if the category title exactly matches the query. We compare and contrast these two methodologies by evaluating six web search engines on a sample from an America Online log of ten million web queries, using MRR measures for the first method and precision-based measures for the second. We show that this technique is stable with respect to the query set selected and correlated with a reasonably large manual evaluation.

REFERENCES

• 1: Steven M. Beitzel , Eric C. Jensen , Abdur Chowdhury , David Grossman , Ophir Frieder, Using manually-built web directories for automatic evaluation of known-item retrieval, Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval, July 28-August 01, 2003, Toronto, Canada [doi>10.1145/860435.860507]
• 2: Boyan, J., Freitag, D., and Joachims, T. A machine learning architecture for optimizing web search engines. In AAAI'96 (August, 1996) Workshop on Internet Based Information Systems. http://www.cs.cornell.edu/People/tj/publications/boyan_etal_96a.pdf
• 3: Andrei Broder, A taxonomy of web search, ACM SIGIR Forum, v.36 n.2, Fall 2002 [doi>10.1145/792550.792552]
• 4: Peter Bruza , Robert McArthur , Simon Dennis, Interactive Internet search: keyword, directory and query reformulation mechanisms compared, Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval, p.280-287, July 24-28, 2000, Athens, Greece [doi>10.1145/345508.345598]
• 5: Chris Buckley , Ellen M. Voorhees, Evaluating evaluation measure stability, Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval, p.33-40, July 24-28, 2000, Athens, Greece [doi>10.1145/345508.345543]
• 6: Buckley, C. Proposal to TREC Web Track mailing list (November, 2001). http://groups.yahoo.com/group/webir/message/760
• 7: Abdur Chowdhury , Ian Soboroff, Automatic evaluation of world wide web search services, Proceedings of the 25th annual international ACM SIGIR conference on Research and development in information retrieval, August 11-15, 2002, Tampere, Finland [doi>10.1145/564376.564474]
• 8: Craswell, N., Bailey, P., and Hawking, D. Is it fair to evaluate Web systems using TREC ad hoc methods? SIGIR'99 (Berkeley, CA, 1999) Workshop on Web Evaluation. http://pigfish.vic.cmis.csiro.au/ nickc/pubs/sigir99ws.ps.gz
• 9: Wei Ding and Gary Marchionini. Comparative study of web search service performance. In Proceedings of the ASIS 1996 Annual Conference (October 1996).
• 10: Michael Gordon , Praveen Pathak, Finding information on the World Wide Web: the retrieval effectiveness of search engines, Information Processing and Management: an International Journal, v.35 n.2, p.141-180, March 1999 [doi>10.1016/S0306-4573(98)00041-7 ]
• 11: Hawking, D., Craswell, N., and Thistlewaite P. Overview of TREC-7 Very Large Collection Track. In Proceedings of TReC7 (Gaithersburg, MD, 1998), NIST Special Publication 500-242, 91--104.
• 12: Hawking, D., Voorhees, E., Craswell, N., and Bailey, P. Overview of the TREC-8 Web Track. In Proceedings of TReC8 (Gaithersburg, MD, 1999), NIST Special Publication 500-246, 131--149.
• 13: David Hawking , Nick Craswell , Paul Thistlewaite , Donna Harman, Results and challenges in Web search evaluation, Proceeding of the eighth international conference on World Wide Web, p.1321-1330, May 1999, Toronto, Canada
• 14: Hawking, D., and Craswell, N. Overview of the TREC-2001 Web Track. In Proceedings of TReC10 (Gaithersburg, MD, 2001), NIST Special Publication 500-250, 61--67.
• 15: David Hawking , Nick Craswell , Peter Bailey , Kathleen Griffihs, Measuring Search Engine Quality, Information Retrieval, v.4 n.1, p.33-59, April 2001 [doi>10.1023/A:1011468107287 ]
• 16: Hawking, D., Craswell, N., and Griffiths, K. Which search engine is best at finding airline site home pages? CMIS Technical Report 01/45 (March, 2001). http://pigfish.vic.cmis.csiro.au/ nickc/pubs/TR01-45.pdf
• 17: Hawking, D., Craswell, N., and Griffiths, K. Which Search Engine is Best at Finding Online Services? In Proceedings of WWW10 (Hong Kong, May 2001), Posters. Actual poster available as <http://pigfish.vic.cmis.csiro.au/ nickc/pubs/www10actualposter.pdf>
• 18: Hawking, D., and Craswell, N. Overview of the TREC-2002 Web Track. To appear in Proceedings of TReC11 (Gaithersburg, MD, 2002).
• 19: Taher H. Haveliwala , Aristides Gionis , Dan Klein , Piotr Indyk, Evaluating strategies for similarity search on the web, Proceedings of the 11th international conference on World Wide Web, May 07-11, 2002, Honolulu, Hawaii, USA [doi>10.1145/511446.511502]
• 20: Bernard J. Jansen , Amanda Spink , Tefko Saracevic, Real life, real users, and real needs: a study and analysis of user queries on the web, Information Processing and Management: an International Journal, v.36 n.2, p.207-227, Jan.1.2000 [doi>10.1016/S0306-4573(99)00056-4 ]
• 21: Joachims, T. Evaluating Retrieval Performance using Clickthrough Data. SIGIR'02 (Tampere, Finland, August, 2002) Workshop on Mathematical/Formal Methods in Information Retrieval. http://www.cs.cornell.edu/People/tj/publications/joachims_02b.pdf
• 22: H. Vernon Leighton , Jaideep Srivastava, First 20 precision among World Wide Web search services (search engines), Journal of the American Society for Information Science, v.50 n.10, p.870-881, 1999 [doi>10.1002/(SICI)1097-4571(1999)50:10<870::AID-ASI4>3.3.CO;2-7]
• 23: Longzhuang Li , Yi Shang, A new method for automatic performance comparison of search engines, World Wide Web, v.3 n.4, p.241-247, December 2000 [doi>10.1023/A:1018790907285 ]
• 24: Menczer, F. Semi-Supervised Evaluation of Search Engines via Semantic Mapping. Submitted to WWW'03 (Budapest, Hungary, 2003), ACM Press. http://dollar.biz.uiowa.edu/ fil/Papers/engines.pdf
• 25: Yi Shang , Longzhuang Li, Precision Evaluation of Search Engines, World Wide Web, v.5 n.2, p.159-173, 2002 [doi>10.1023/A:1019679624079 ]
• 26: Craig Silverstein , Hannes Marais , Monika Henzinger , Michael Moricz, Analysis of a very large web search engine query log, ACM SIGIR Forum, v.33 n.1, p.6-12, Fall 1999 [doi>10.1145/331403.331405]
• 27: Amit Singhal , Marcin Kaszkiel, A case study in web search using TREC algorithms, Proceedings of the 10th international conference on World Wide Web, p.708-716, May 01-05, 2001, Hong Kong, Hong Kong [doi>10.1145/371920.372186]
• 28: Amanda Spink , Bernard J. Jansen , Dietmar Wolfram , Tefko Saracevic, From E-Sex to E-Commerce: Web Search Changes, Computer, v.35 n.3, p.107-109, March 2002 [doi>10.1109/2.989940 ]
• 29: Andrew H. Turpin , William Hersh, Why batch and user evaluations do not give the same results, Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval, p.225-231, September 2001, New Orleans, Louisiana, United States [doi>10.1145/383952.383992]
• 30: Ellen M. Voorhees, Variations in relevance judgments and the measurement of retrieval effectiveness, Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, p.315-323, August 24-28, 1998, Melbourne, Australia [doi>10.1145/290941.291017]
• 31: Ellen M. Voorhees, Evaluation by highly relevant documents, Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval, p.74-82, September 2001, New Orleans, Louisiana, United States [doi>10.1145/383952.383963]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Search process

General Terms:
Algorithms, Experimentation, Measurement

Keywords:
automatic evaluation, relevance judgments, web search