Prasenjit Majumder
Skip Abstract Section
Abstract
The aim of the Forum for Information Retrieval Evaluation (FIRE) is to create an evaluation framework in the spirit of TREC (Text REtrieval Conference), CLEF (Cross-Language Evaluation Forum), and NTCIR (NII Test Collection for IR Systems), for Indian language Information Retrieval. The first evaluation exercise conducted by FIRE was completed in 2008. This article describes the test collections used at FIRE 2008, summarizes the approaches adopted by various participants, discusses the limitations of the datasets, and outlines the tasks planned for the next iteration of FIRE.
- }}Amati, G. and Rijsbergen, C. V. 2002. Probabilistic models of information retrieval based on measuring the divergence from randomness. ACM Trans. Inform. Syst. 20, 4, 357--389. Google Scholar
Digital Library
- }}Braschler, M. and Peters, C. 2004. Cross-language evaluation forum: Objectives, results, achievements. Inform. Retriev. 7, 1/2, 7--31. Google ScholarDigital Library
- }}Dolamic, L. and Savoy, J. 2008. UniNE at FIRE 2008: Hindi, Bengali, and Marathi IR. In Working Notes from FIRE 2008 (FIRE’08).Google Scholar
- }}Harman, D. 1995. Overview of the second text retrieval conference (TREC-2). Inform. Process. Manage. 31, 3, 271--289. Google ScholarDigital Library
- }}Hiemstra, D. 2001. Using language models for information retrieval. Ph.D. thesis, University of Twente.Google Scholar
- }}Kando, N., Mitamura, T., and Sakai, T. 2008. Introduction to the NTCIR-6 Special Issue. ACM Trans. Asian Lang. Inform. Process. 7, 2, 1--3. Google ScholarDigital Library
- }}Majumder, P., Mitra, M., Parui, S., Kole, G., Mitra, P., and Datta, K. 2007. YASS: Yet another suffix stripper. ACM Trans. Inform. Syst. 25, 4, 18. Google ScholarDigital Library
- }}McNamee, P. 2008. N-gram Tokenization for Indian Language Text Retrieval. In Working Notes from FIRE 2008 (FIRE’08).Google Scholar
- }}Mitra, M. 2008. Overview of FIRE 2008. In Working Notes from FIRE 2008 (FIRE’08).Google Scholar
- }}Nakagawa, H., Mori, T., and Kando, N., Eds. 2005. ACM Trans. Asian Lang. Inform. Process. 4. Google ScholarDigital Library
- }}Oard, D. W. 2003. The surprise language exercises. ACM Trans. Asian Lang. Inform. Process. 2, 2, 79--84. Google ScholarDigital Library
- }}Ounis, I., Amati, G., Plachouras, V., He, B., Macdonald, C., and Lioma, C. 2006. Terrier: A high performance and scalable information retrieval platform. In Proceedings of the ACM Workshop on Open Source Information Retrieval (OSIR’06).Google Scholar
- }}Padariya, N., Chinnakotla, M., Nagesh, A., and Damani, O. P. 2008. Evaluation of Hindi to English, Marathi to English, and English to Hindi CLIR at FIRE 2008. In Working Notes from FIRE 2008 (FIRE’08).Google Scholar
- }}Paik, J. H. and Parui, S. K. 2008. A simple stemmer for inflectional languages. In Working Notes from FIRE 2008 (FIRE’08).Google Scholar
- }}Pal, D., Majumder, P., Mitra, M., Mitra, S., and Sen, A. 2008. Issues in searching for Indian language Web content. In Proceedings of the 2nd ACM Workshop on Improving Non English Web Searching (iNEWS’08). 93--96. Google ScholarDigital Library
- }}Peters, C. 2010. Personal communication.Google Scholar
- }}Peters, C., Jijkoun, V., Mandl, T., Müller, H., Oard, D., Peñas
, A., Petras, V., and Santos, D., Eds. 2008. In Advances in Multilingual and Multimodal Information Retrieval, the 8th Workshop of the Cross-Language Evaluation Forum (CLEF’07). Lecture Notes in Computer Science. Springer-Verlag. Google ScholarDigital Library
- }}Pingali, P., Jagarlamudi, J., and Varma, V. 2006. Webkhoj: Indian language IR from multiple character encodings. In Proceedings of the International World Wide Web Conference (WWW’06). Google ScholarDigital Library
- }}Ponte, J. and Croft, W. 1998. A language modeling approach to information retrieval. In Proceedings of the 19th Annual International ACM Conference on Research and Development in Information Retrieval (SIGIR’98). Google ScholarDigital Library
- }}Ramanathan, A. and Rao, D. 2003. A lightweight stemmer for Hindi. In Proceedings of the Conference of the European Chapter of the Association for Computational Linguistics (EACL’03).Google Scholar
- }}Rao, P. R. and Sobha, L. 2008. AU-KBC FIRE2008 submission - Cross lingual information retrieval track: Tamil-English. In Working Notes from FIRE 2008 (FIRE’08).Google Scholar
- }}Sakai, T., Kando, N., Lin, C.-J., Mitamura, T., Shima, H., Ji, D., Chen, K.-H., and Nyberg, E. 2008. Overview of the NTCIR-7 ACLIA IR4QA Task. In Proceedings of the NII Test Collection for Information Retrieval Workshop (NTCIR’08). 77--114.Google Scholar
- }}Savoy, J. 2004. Data fusion for effective European monolingual information retrieval. In Proceedings of the Cross-Language Information Retrieval and Evaluation Workshop of Cross-Language Evaluation Forum (CLEF’00). 233--244. Google ScholarDigital Library
- }}Sethuramalingam, S. and Varma, V. 2008. IIIT Hyderabad’s CLIR experiments for FIRE-2008. In Working Notes from FIRE 2008 (FIRE’08).Google Scholar
- }}Sparck Jones, K. and van Rijsbergen, C. 1976. Information retrieval test collections. J. Doc. 32, 59--75.Google ScholarCross Ref
- }}Sparck Jones, K., Walker, S., and Robertson, S. 2000. A probabilistic model of information retrieval: Development and comparative experiments. Inform. Process. Manage. 36, 6, 779--808. Google ScholarDigital Library
- }}Surve, M., Singh, S., and Bhattacharyya, P. 2004. Agro-Explorer: A meaning based multilingual search engine. In Proceedings of the International Conference on Digital Libraries (ICDL’04).Google Scholar
- }}Udupa, R., Jagarlamudi, J., and Saravanan, K. 2008. Hindi-English cross-language information retrieval. In Working Notes from FIRE 2008 (FIRE’08).Google Scholar
- }}Udupa, R., Saravanan, K., Bakalov, A., and Bhole, A. 2009. “They are out there, if you know where to look”: Mining transliterations of OOV query terms for cross-language information retrieval. In Proceedings of the European Conference on Information Retrieval (ECIR09). 437--448. Google ScholarDigital Library
- }}Voorhees, E. and Harman, D. 1997. Overview of the 5th Text Retrieval Conference. In Proceedings of the 5th Text Retrieval Conference (TREC5). 1--28.Google ScholarCross Ref
- }}Voorhees, E. M. and Harman, D. K., Eds. 2005. TREC Experiment and Evaluation in Information Retrieval. MIT Press, Cambridge, MA. Google ScholarDigital Library
- }}Zobel, J. 1998. How reliable are the results of large-scale information retrieval experiments? In Proceedings of the 19th Annual International ACM Conference on Research and Development in Information Retrieval (SIGIR’98). Google ScholarDigital Library
Index Terms
- The FIRE 2008 Evaluation Exercise
Recommendations
Current Status of the Evaluation of Information Retrieval
This is the second in the series of the articles on an application of the systems analytic approach to evaluation of information retrieval (IR). In the previous article a historical overview of IR was presented and existing terminological problems ...
A New Look at Information Retrieval Evaluation: Proposal for Solutions
This is the fourth and the final in the series of the papers on an application of the systems analytic approach to evaluation of information retrieval (IR). In the previous papers terminological and evaluation problems associated with IR were identified ...
Query clustering and IR system detection: experiments on TREC data
RIAO '07: Large Scale Semantic Access to Content (Text, Image, Video, and Sound)This paper investigates two aspects in this experiment. Linguistic techniques are used to categorize queries in a first step. This classification is then used to analyze systems performances in a TREC context. More precisely, we cluster TREC topics with ...
Comments