research-article

CLEF 15th Birthday: Past, Present, and Future

Author:
Nicola Ferro

University of Padua, Italy

University of Padua, Italy
View Profile

Authors Info & Claims

ACM SIGIR Forum Volume 48 Issue 2December 2014pp 31–55https://doi.org/10.1145/2701583.2701587

Published:23 December 2014Publication History

ACM SIGIR Forum

Abstract

2014 marks the 15th birthday for CLEF, an evaluation campaign activity which has applied the Cranfield evaluation paradigm to the testing of multilingual and multimodal information access systems in Europe. This paper provides a summary of the motivations which led to the establishment of CLEF, and a description of how it has evolved over the years, the major achievements, and what we see as the next challenges.

References

E. Agirre, O. L. de Lacalle, B. Magnini, A. Otegi, G. Rigau, and P. Vossen. SemEval-2007 Task 01: Evaluating WSD on Cross-Language Information Retrieval. In Peters et al. {207}, pages 908--917.Google Scholar
E. Agirre, G. M. Di Nunzio, N. Ferro, T. Mandl, and C. Peters. CLEF 2008: Ad Hoc Track Overview. In Peters et al. {200}, pages 15--37. Google ScholarDigital Library
E. Agirre, G. M. Di Nunzio, T. Mandl, and A. Otegi. CLEF 2009 Ad Hoc Track Overview: Robust-WSD Task. In Peters et al. {201}, pages 36--49. Google ScholarDigital Library
M. Agosti, E. Di Buccio, N. Ferro, I. Masiero, S. Peruzzo, and G. Silvello. DIRECTions: Design and Specification of an IR Evaluation Infrastructure. In Catarci et al. {53}, pages 88--99. Google ScholarDigital Library
M. Agosti, G. M. Di Nunzio, and N. Ferro. A Proposal to Extend and Enrich the Scientific Data Curation of Evaluation Campaigns. In T. Sakay, M. Sanderson, and D. K. Evans, editors, Proc. 1st International Workshop on Evaluating Information Access (EVIA 2007), pages 62--73. National Institute of Informatics, Tokyo, Japan, 2007.Google Scholar
M. Agosti, G. M. Di Nunzio, and N. Ferro. Scientific Data of an Evaluation Campaign: Do We Properly Deal With Them? In Peters et al. {198}, pages 11--20. Google ScholarDigital Library
M. Agosti, G. M. Di Nunzio, N. Ferro, D. Harman, and C. Peters. The Future of Large-scale Evaluation Campaigns for Information Retrieval in Europe. In N. Fuhr, L. Kovács, and C. Meghini, editors, Proc. 11th European Conference on Research and Advanced Technology for Digital Libraries (ECDL 2007), pages 509--512. Lecture Notes in Computer Science (LNCS) 4675, Springer, Heidelberg, Germany, 2007. Google ScholarDigital Library
M. Agosti, N. Ferro, C. Peters, M. de Rijke, and A. Smeaton, editors. Multilingual and Multimodal Information Access Evaluation. Proceedings of the International Conference of the Cross-Language Evaluation Forum (CLEF 2010). Lecture Notes in Computer Science (LNCS) 6360, Springer, Heidelberg, Germany, 2010. Google ScholarDigital Library
M. Agosti, N. Ferro, and C. Thanos. DESIRE 2011 Workshop on Data infrastructurEs for Supporting Information Retrieval Evaluation. SIGIR Forum, 46(1):51--55, June 2012. Google ScholarDigital Library
J. Allan, W. B. Croft, A. Moffat, and M. Sanderson. Frontiers, Challenges, and Opportunities for Information Retrieval -- Report from SWIRL 2012, The Second Strategic Workshop on Information Retrieval in Lorne, February 2012. SIGIR Forum, 46(1):2--32, June 2012. Google ScholarDigital Library
E. Amigó, J. Artiles, J. Gonzalo, D. Spina, B. Liu, and A. Corujo. WePS3 Evaluation Campaign: Overview of the On-line Reputation Management Task. In Braschler et al. {46}.Google Scholar
E. Amigó, J. Carrillo de Albornoz, I. Chugur, A. Corujo, J. Gonzalo, T. Martín-Wanton, E. Meij, M. de Rijke, and D. Spina. Overview of RepLab 2013: Evaluating Online Reputation Monitoring Systems. In Forner et al. {86}, pages 333--352.Google Scholar
E. Amigó, J. Carrillo de Albornoz, I. Chugur, A. Corujo, J. Gonzalo, E. Meij, M. de Rijke, and D. Spina. Overview of RepLab 2014: Author Profiling and Reputation Dimensions for Online Reputation Management. In Kanoulas et al. {129}, pages 307--322.Google Scholar
E. Amigó, A. Corujo, J. Gonzalo, E. Meij, and M. de Rijke. Overview of RepLab 2012: Evaluating Online Reputation Management Systems. In Forner et al. {85}.Google Scholar
E. Amigó, J. Gonzalo, and M. F. Verdejo. A Comparison of Evaluation Metrics for Document Filtering. In Forner et al. {84}, pages 38--49. Google ScholarDigital Library
M. Anderka and B. Stein. Overview of the 1th International Competition on Quality Flaw Prediction in Wikipedia. In Forner et al. {85}.Google Scholar
M. Angelini, N. Ferro, K. Järvelin, H. Keskustalo, A. Pirkola, G. Santucci, and G. Silvello. Cumulated Relative Position: A Metric for Ranking Evaluation. In Catarci et al. {53}, pages 112--123. Google ScholarDigital Library
M. Angelini, N. Ferro, B. Larsen, H. Müller, G. Santucci, G. Silvello, and T. Tsikrika. Measuring and Analyzing the Scholarly Impact of Experimental Evaluation Initiatives. Procedia Computer Science, (in print).Google Scholar
M. Angelini, N. Ferro, G. Santucci, and G. Silvello. Improving Ranking Evaluation Employing Visual Analytics. In Forner et al. {86}, pages 29--40.Google Scholar
S. Argamon and P. Juola. Overview of the International Authorship Identification Competition at PAN-2011. In Petras et al. {214}.Google Scholar
T. G. Armstrong, A. Moffat, W. Webber, and J. Zobel. Improvements That Don't Add Up: Ad-Hoc Retrieval Results Since 1998. In D. W.-L. Cheung, I.-Y. Song, W. W. Chu, X. Hu, and J. J. Lin, editors, Proc. 18th International Conference on Information and Knowledge Management (CIKM 2009), pages 601--610. ACM Press, New York, USA, 2009. Google ScholarDigital Library
T. Arni, P. Clough, M. Sanderson, and M. Grubinger. Overview of the ImageCLEFphoto 2008 Photographic Retrieval Task. In Peters et al. {200}, pages 500--511. Google ScholarDigital Library
J. Artiles, A. Borthwick, J. Gonzalo, S. Sekine, and E. Amigó. WePS-3 Evaluation Campaign: Overview of the Web People Search Clustering and Attribute Extraction Tasks. In Braschler et al. {46}.Google Scholar
H. Azarbonyad, A. Shakery, and H. Faili. Exploiting Multiple Translation Resources for English-Persian Cross Language Information Retrieval. In Forner et al. {86}, pages 93--99.Google Scholar
L. Azzopardi and K. Balog. Towards a Living Lab for Information Retrieval Research and Development - A Proposal for a Living Lab for Product Search Tasks. In Forner et al. {84}, pages 26--37. Google ScholarDigital Library
K. Balog, L. Azzopardi, J. Kamps, and M. de Rijke. Overview of WebCLEF 2006. In Peters et al. {198}, pages 803--819. Google ScholarDigital Library
H. Baradaran Hashemi, A. Shakery, and H. Feili. Creating a Persian-English Comparable Corpus. In Agosti et al. {8}, pages 27--39. Google ScholarDigital Library
P. Bellot, T. Bogers, S. Geva, M. A. Hall, H. C. Huurdeman, J. Kamps, G. Kazai, M. Koolen, V. Moriceau, J. Mothe, M. Preminger, E. SanJuan, R. Schenkel, M. Skov, X. Tannier, and D. Walsh. Overview of INEX 2014. In Kanoulas et al. {129}, pages 212--228.Google Scholar
P. Bellot, A. Doucet, S. Geva, S. Gurajada, J. Kamps, G. Kazai, M. Koolen, A. Mishra, V. Moriceau, J. Mothe, M. Preminger, E. SanJuan, R. Schenkel, X. Tannier, M. Theobald, M. Trappett, and Q. Wang. Overview of INEX 2013. In Forner et al. {86}, pages 269--281.Google Scholar
A. Beloborodov, P. Braslavski, and M. Driker. Towards Automatic Evaluation of Health-Related CQA Data. In Kanoulas et al. {129}, pages 7--18.Google Scholar
I. Bensalem, P. Rosso, and S. Chikhi. A New Corpus for the Evaluation of Arabic Intrinsic Plagiarism Detection. In Forner et al. {86}, pages 53--58.Google Scholar
R. Berendsen, M. Tsagkias, M. de Rijke, and E. Meij. Generating Pseudo Test Collections for Learning to Rank Scientific Articles. In Catarci et al. {53}, pages 42--53. Google ScholarDigital Library
R. Berlanga Llavori, A. Jimeno-Yepes, M. Pérez Catalán, and D. Rebholz-Schuhmann. Context-Dependent Semantic Annotation in Cross-Lingual Biomedical Resources. In Forner et al. {86}, pages 120--123.Google Scholar
R. Berlanga Llavori, M. Pérez Catalán, L. Museros Cabedo, and R. Forcada. Semantic Discovery of Resources in Cloud-Based PACS/RIS Systems. In Forner et al. {86}, pages 167--178.Google Scholar
R. Besançon, S. Chaudiron, D. Mostefa, O. Hamon, Timimi. I., and K. Choukri. Overview of CLEF 2008 INFILE Pilot Track. In Peters et al. {200}, pages 939--946. Google ScholarDigital Library
R. Besançon, S. Chaudiron, D. Mostefa, Timimi. I., K. Choukri, and M. Laïb. Information Filtering Evaluation: Overview of CLEF 2009 INFILE Track. In Peters et al. {201}, pages 342--353. Google ScholarDigital Library
P. Bhaskar and S. Bandyopadhyay. Language Independent Query Focused Snippet Generation. In Catarci et al. {53}, pages 138--140. Google ScholarDigital Library
F. Borri, A. Nardi, C. Peters, and N. Ferro, editors. CLEF 2008 Working Notes. CEUR Workshop Proceedings (CEUR-WS.org), ISSN 1613-0073, http://ceur-ws.org/Vol-1174/, 2008.Google Scholar
F. Borri, A. Nardi, C. Peters, and N. Ferro, editors. CLEF 2009 Working Notes. CEUR Workshop Proceedings (CEUR-WS.org), ISSN 1613-0073, http://ceur-ws.org/Vol-1175/, 2009.Google Scholar
F. Borri, C. Peters, and N. Ferro, editors. CLEF 2004 Working Notes. CEUR Workshop Proceedings (CEUR-WS.org), ISSN 1613-0073, http://ceur-ws.org/Vol-1170/, 2004.Google Scholar
M. Braschler. CLEF 2000 -- Overview of Results. In Peters {193}, pages 89--101. Google ScholarDigital Library
M. Braschler. CLEF 2001 -- Overview of Results. In Peters et al. {195}, pages 9--26. Google ScholarDigital Library
M. Braschler. CLEF 2002 -- Overview of Results. In Peters et al. {196}, pages 9--27.Google Scholar
M. Braschler. CLEF 2003 -- Overview of Results. In Peters et al. {197}, pages 44--63.Google Scholar
M. Braschler, G. M. Di Nunzio, N. Ferro, and C. Peters. CLEF 2004: Ad Hoc Track Overview and Results Analysis. In Peters et al. {199}, pages 10--26. Google ScholarDigital Library
M. Braschler, D. K. Harman, E. Pianta, and N. Ferro, editors. CLEF 2010 Working Notes. CEUR Workshop Proceedings (CEUR-WS.org), ISSN 1613-0073, http://ceur-ws.org/Vol-1176/, 2010.Google Scholar
M. Braschler and C. Peters. Cross-Language Evaluation Forum: Objectives, Results, Achievements. Information Retrieval, 7(1-2):7--31, 2004. Google ScholarDigital Library
G. Cabanac, G. Hubert, M. Boughanem, and C. Chrisment. Tie-Breaking Bias: Effect of an Uncontrolled Parameter on Information Retrieval Evaluation. In Agosti et al. {8}, pages 112--123. Google ScholarDigital Library
L. Cappellato, N. Ferro, M. Halvey, and W. Kraaij, editors. CLEF 2014 Working Notes. CEUR Workshop Proceedings (CEUR-WS.org), ISSN 1613-0073, http://ceur-ws.org/Vol-1180/, 2014.Google Scholar
B. Caputo, H. Müller, J. Martínez-Gómez, M. Villegas, B. Acar, N. Patricia, N. Barzegar Marvasti, S. Üsküdarli, R. Paredes, M. Cazorla, I. García-Varea, and V. Morell. ImageCLEF 2014: Overview and Analysis of the Results. In Kanoulas et al. {129}, pages 192--211.Google Scholar
B. Caputo, H. Müller, B. Thomee, M. Villegas, R. Paredes, D. Zellhöfer, H. Goëau, A. Joly, P. Bonnet, J. Martínez-Gómez, I. García-Varea, and M. Cazorla. ImageCLEF 2013: The Vision, the Data and the Open Challenges. In Forner et al. {86}, pages 250--268.Google Scholar
T. Cassidy, H. Ji, H. Deng, J. Zheng, and J. Han. Analysis and Refinement of Cross-Lingual Entity Linking. In Catarci et al. {53}, pages 1--12. Google ScholarDigital Library
T. Catarci, P. Forner, D. Hiemstra, A. Peñas, and G. Santucci, editors. Information Access Evaluation. Multilinguality, Multimodality, and Visual Analytics. Proceedings of the Third International Conference of the CLEF Initiative (CLEF 2012). Lecture Notes in Computer Science (LNCS) 7488, Springer, Heidelberg, Germany, 2012. Google ScholarDigital Library
T. Chappell and S. Geva. Overview of the INEX 2012 Relevance Feedback Track. In Forner et al. {85}.Google Scholar
P. Cimiano, V. Lopez, C. Unger, E. Cabrio, A.-C. Ngonga Ngomo, and S. Walter. Multilingual Question Answering over Linked Data (QALD-3): Lab Overview. In Forner et al. {86}, pages 321--332.Google Scholar
P. Clough and P. Goodale. Selecting Success Criteria: Experiences with an Academic Library Catalogue. In Forner et al. {86}, pages 59--70.Google Scholar
P. Clough, M. Grubinger, T. Deselaers, A. Hanbury, and H. Müller. Overview of the ImageCLEF 2006 Photographic Retrieval and Object Annotation Tasks. In Peters et al. {198}, pages 223--256. Google ScholarDigital Library
P. Clough, H. Müller, T. Deselaers, M. Grubinger, T. M. Lehmann, J. R. Jensen, and W. R. Hersh. The CLEF 2005 Cross-Language Image Retrieval Track. In Peters et al. {206}, pages 535--557. Google ScholarDigital Library
P. Clough, H. Müller, and M. Sanderson. The CLEF 2004 Cross-Language Image Retrieval Track. In Peters et al. {199}, pages 597--613. Google ScholarDigital Library
P. Clough and M. Sanderson. The CLEF 2003 Cross Language Image Retrieval Track. In Peters et al. {197}, pages 581--593.Google Scholar
R. Corezola Pereira, V. Pereira Moreira, and R. Galante. A New Approach for Cross-Language Plagiarism Analysis. In Agosti et al. {8}, pages 15--26. Google ScholarDigital Library
S. de L. Pertile and V. Pereira Moreira. A Test Collection to Evaluate Plagiarism by Missing or Incorrect References. In Catarci et al. {53}, pages 141--143. Google ScholarDigital Library
S. de L. Pertile, P. Rosso, and V. Pereira Moreira. Counting Co-occurrences in Citations to Identify Plagiarised Text Fragments. In Forner et al. {86}, pages 150--154.Google Scholar
M. de Rijke, K. Balog, T. Bogers, and A. van den Bosch. On the Evaluation of Entity Profiles. In Agosti et al. {8}, pages 94--99. Google ScholarDigital Library
T. Deselaers and T. M. Deserno. Medical Image Annotation in ImageCLEF 2008. In Peters et al. {200}, pages 523--530. Google ScholarDigital Library
T. Deselaers and A. Hanbury. The Visual Concept Detection Task in ImageCLEF 2008. In Peters et al. {200}, pages 531--538. Google ScholarDigital Library
T. Deselaers, A. Hanbury, V. Viitaniemi, A. A. Benczúr, M. Brendel, B. Daróczy, H. J. Escalante Balderas, T. Gevers, C. A. Hernández-Gracidas, S. C. H. Hoi, J. Laaksonen, M. Li, H. M. Marín Castro, H. Ney, X. Rui, N. Sebe, J. Stöttinger, and L. Wu. Overview of the ImageCLEF 2007 Object Retrieval Task. In Peters et al. {207}, pages 445--471. Google ScholarDigital Library
E. Di Buccio, M. Dussin, N. Ferro, I. Masiero, G. Santucci, and G. Tino. To Re-rank or to Re-query: Can Visual Analytics Solve This Dilemma? In Forner et al. {84}, pages 119--130. Google ScholarDigital Library
G. M. Di Nunzio, N. Ferro, G. J. F. Jones, and C. Peters. CLEF 2005: Ad Hoc Track Overview. In Peters et al. {206}, pages 11--36. Google ScholarDigital Library
G. M. Di Nunzio, N. Ferro, T. Mandl, and C. Peters. CLEF 2006: Ad Hoc Track Overview. In Peters et al. {198}, pages 21--34. Google ScholarDigital Library
G. M. Di Nunzio, N. Ferro, T. Mandl, and C. Peters. CLEF 2007: Ad Hoc Track Overview. In Peters et al. {207}, pages 13--32.Google Scholar
G. M. Di Nunzio, J. Leveling, and T. Mandl. LogCLEF 2011 Multilingual Log File Analysis: Language Identification, Query Classification, and Success of a Query. In Petras et al. {214}.Google Scholar
M. Dussin and N. Ferro. Managing the Knowledge Creation Process of Large-Scale Evaluation Campaigns. In M. Agosti, J. Borbinha, S. Kapidakis, C. Papatheodorou, and G. Tsakonas, editors, Proc. 13th European Conference on Research and Advanced Technology for Digital Libraries (ECDL 2009), pages 63--74. Lecture Notes in Computer Science (LNCS) 5714, Springer, Heidelberg, Germany, 2009. Google ScholarDigital Library
F. Esuli and F. Sebastiani. Evaluating Information Extraction. In Agosti et al. {8}, pages 100--111. Google ScholarDigital Library
A. M. Ezzeldin, M. H. Kholief, and Y. El-Sonbaty. ALQASIM: Arabic Language Question Answer Selection in Machines. In Forner et al. {86}, pages 100--103.Google Scholar
M. Federico, N. Bertoldi, G.-A. Levow, and G. J. F. Jones. CLEF 2004 Cross-Language Spoken Document Retrieval Track. In Peters et al. {199}, pages 816--820. Google ScholarDigital Library
M. Federico and G. J. F. Jones. The CLEF 2003 Cross-Language Spoken Document Retrieval Track. In Peters et al. {197}, page 646.Google Scholar
M. Ferrante, N. Ferro, and M. Maistro. Rethinking How to Extend Average Precision to Graded Relevance. In Kanoulas et al. {129}, pages 19--30.Google Scholar
N. Ferro, R. Berendsen, A. Hanbury, M. Lupu, V. Petras, M. de Rijke, and G. Silvello. PROMISE Retreat Report -- Prospects and Opportunities for Information Access Evaluation. SIGIR Forum, 46(2):60--84, December 2012. Google ScholarDigital Library
N. Ferro and D. Harman. CLEF 2009: Grid@CLEF Pilot Track Overview. In Peters et al. {201}, pages 552--565. Google ScholarDigital Library
N. Ferro and C. Peters. CLEF 2009 Ad Hoc Track Overview: TEL & Persian Tasks. In Peters et al. {201}, pages 13--35. Google ScholarDigital Library
N. Ferro and G. Silvello. CLEF 15th Birthday: What Can We Learn From Ad Hoc Retrieval? In Kanoulas et al. {129}, pages 31--43.Google Scholar
N. Ferro and G. Silvello. Making it Easier to Discover, Re-Use and Understand Search Engine Experimental Evaluation Data. ERCIM News, 96:26--27, January 2014.Google Scholar
P. Forner, J. Gonzalo, J. Kekäläinen, M. Lalmas, and M. de Rijke, editors. Multilingual and Multimodal Information Access Evaluation. Proceedings of the Second International Conference of the Cross-Language Evaluation Forum (CLEF 2011). Lecture Notes in Computer Science (LNCS) 6941, Springer, Heidelberg, Germany, 2011. Google ScholarDigital Library
P. Forner, J. Karlgren, C. Womser-Hacker, and N. Ferro, editors. CLEF 2012 Working Notes. CEUR Workshop Proceedings (CEUR-WS.org), ISSN 1613-0073, http://ceur-ws.org/Vol-1178/, 2012.Google Scholar
P. Forner, H. Müller, R. Paredes, P. Rosso, and B. Stein, editors. Information Access Evaluation meets Multilinguality, Multimodality, and Visualization. Proceedings of the Fourth International Conference of the CLEF Initiative (CLEF 2013). Lecture Notes in Computer Science (LNCS) 8138, Springer, Heidelberg, Germany, 2013.Google Scholar
P. Forner, R. Navigli, D. Tufis, and N. Ferro, editors. CLEF 2013 Working Notes. CEUR Workshop Proceedings (CEUR-WS.org), ISSN 1613-0073, http://ceur-ws.org/Vol-1179/, 2013.Google Scholar
P. Forner, A. Peñas, E. Agirre, I. Alegria, C. Forascu, N. Moreau, P. Osenova, P. Prokopidis, P. Rocha, B. Sacaleanu, R. F. E. Sutcliffe, and E. F. T. K. Sang. Overview of the Clef 2008 Multilingual Question Answering Track. In Peters et al. {200}, pages 262--295. Google ScholarDigital Library
M. Gäde, N. Ferro, and M. Lestari Paramita. CHiC 2011 -- Cultural Heritage in CLEF: From Use Cases to Evaluation in Practice for Multilingual Information Access to Cultural Heritage. In Petras et al. {214}.Google Scholar
M. Gäde, J. Stiller, and V. Petras. Which Log for Which Information? Gathering Multilingual Data from Different Log File Types. In Agosti et al. {8}, pages 70--81. Google ScholarDigital Library
P. Galuscáková, P. Pecina, and J. Hajic. Penalty Functions for Evaluation Measures of Unsegmented Speech Retrieval. In Catarci et al. {53}, pages 100--111. Google ScholarDigital Library
D. Ganguly, J. Leveling, and G. J. F. Jones. Simulation of Within-Session Query Variations Using a Text Segmentation Approach. In Forner et al. {84}, pages 89--94. Google ScholarDigital Library
D. Ganguly, J. Leveling, and G. J. F. Jones. A Case Study in Decompounding for Bengali Information Retrieval. In Forner et al. {86}, pages 108--119.Google Scholar
F. Gey, R. Larson, M. Sanderson, K. Bischoff, T. Mandl, K. Womser-Hacker, D. Santos, P. Rocha, G. M. Di Nunzio, and N. Ferro. GeoCLEF 2006: the CLEF 2006 Cross-Language Geographic Information Retrieval Track Overview. In Peters et al. {198}, pages 852--876. Google ScholarDigital Library
F. C. Gey, R. R. Larson, M. Sanderson, H. Joho, P. Clough, and V. Petras. GeoCLEF: The CLEF 2005 Cross-Language Geographic Information Retrieval Track Overview. In Peters et al. {206}, pages 908--919. Google ScholarDigital Library
D. Giampiccolo, P. Forner, J. Herrera, A. Peñas, C. Ayache, C. Forascu, V. Jijkoun, P. Osenova, P. Rocha, B. Sacaleanu, and R. F. E. Sutcliffe. Overview of the CLEF 2007 Multilingual Question Answering Track. In Peters et al. {207}, pages 200--236.Google Scholar
D. G. Glinos. Discovering Similar Passages within Large Text Documents. In Kanoulas et al. {129}, pages 98--109.Google Scholar
H. Goëau, P. Bonnet, A. Joly, N. Boujemaa, D. Barthelemy, J.-F. Molino, P. Birnbaum, E. Mouysset, and M. Picard. The CLEF 2011 Plant Images Classification Task. In Petras et al. {214}.Google Scholar
H. Goëau, P. Bonnet, A. Joly, I. Yahiaoui, D. Barthelemy, N. Boujemaa, and J.-F. Molino. The ImageCLEF 2012 Plant Identification Task. In Forner et al. {85}.Google Scholar
T. Gollub, M. Potthast, A. Beyer, M. Busse, F. Rangel Pardo, P. Rosso, E. Stamatatos, and B. Stenno. Recent Trends in Digital Text Forensics and Its Evaluation - Plagiarism Detection, Author Identification, and Author Profiling. In Forner et al. {86}, pages 282--302.Google Scholar
J. Gonzalo, P. Clough, and J. Karlgren. Overview of iCLEF 2008: Search Log Analysis for Multilingual Image Retrieval. In Peters et al. {200}, pages 227--235. Google ScholarDigital Library
J. Gonzalo, P. Clough, and A. Vallin. Overview of the CLEF 2005 Interactive Track. In Peters et al. {206}, pages 251--262. Google ScholarDigital Library
J. Gonzalo and D. W. Oard. The CLEF 2002 Interactive Track. In Peters et al. {196}, pages 372--382.Google Scholar
J. Gonzalo and D. W. Oard. iCLEF 2004 Track Overview: Pilot Experiments in Interactive Cross-Language Question Answering. In Peters et al. {199}, pages 310--322. Google ScholarDigital Library
J. Gonzalo, V. Peinado, P. Clough, and J. Karlgren. Overview of iCLEF 2009: Exploring Search Behaviour in a Multilingual Folksonomy Environment. In Peters et al. {209}, pages 13--20. Google ScholarDigital Library
T. Goodwin and S. M. Harabagiu. The Impact of Belief Values on the Identification of Patient Cohorts. In Forner et al. {86}, pages 155--166.Google Scholar
M. Grubinger, P. Clough, A. Hanbury, and H. Müller. Overview of the ImageCLEFphoto 2007 Photographic Retrieval Task. In Peters et al. {207}, pages 433--444.Google Scholar
P. Gupta, A. Barrón-Cedeño, and P. Rosso. Cross-Language High Similarity Search Using a Conceptual Thesaurus. In Catarci et al. {53}, pages 67--75. Google ScholarDigital Library
M. Hagen and C. Glimm. Supporting More-Like-This Information Needs: Finding Similar Web Content in Different Scenarios. In Kanoulas et al. {129}, pages 50--61.Google Scholar
M. Hall and E. Toms. Building a Common Framework for IIR Evaluation. In Forner et al. {86}, pages 17--28.Google Scholar
H. Hammarström. Automatic Annotation of Bibliographical References for Descriptive Language Materials. In Forner et al. {84}, pages 62--73. Google ScholarDigital Library
A. Hanbury and H. Müller. Automated Component-Level Evaluation: Present and Future. In Agosti et al. {8}, pages 124--135. Google ScholarDigital Library
A. Hanbury, H. Müller, G. Langs, M.-A. Weber, B. H. Menze, and T. Salas Fernandez. Bringing the Algorithms to the Data: Cloud-Based Benchmarking for Medical Image Analysis. In Catarci et al. {53}, pages 24--29. Google ScholarDigital Library
D. K. Harman. Information Retrieval Evaluation. Morgan & Claypool Publishers, USA, 2011. Google ScholarDigital Library
D. K. Harman and E. M. Voorhees, editors. TREC. Experiment and Evaluation in Information Retrieval. MIT Press, Cambridge (MA), USA, 2005. Google ScholarDigital Library
C. G. Harris and T. Xu. The Importance of Visual Context Clues in Multimedia Translation. In Forner et al. {84}, pages 107--118. Google ScholarDigital Library
D. Hiemstra and C. Hauff. MapReduce for Information Retrieval Evaluation: "Let's Quickly Test This on 12 TB of Data". In Agosti et al. {8}, pages 64--69. Google ScholarDigital Library
F. Hopfgartner, B. Kille, A. Lommatzsch, T. Plumbaum, T. Brodt, and T. Heintz. Benchmarking News Recommendations in a Living Lab. In Kanoulas et al. {129}, pages 250--267.Google Scholar
D. A. Hull and D. W. Oard. Cross-Language Text and Speech Retrieval -- Papers from the AAAI Spring Symposium. Association for the Advancement of Artificial Intelligence (AAAI), Technical Report SS-97-05, http://www.aaai.org/Press/Reports/Symposia/Spring/ss-97-05.php, 1997.Google Scholar
B. Huurnink, K. Hofmann, M. de Rijke, and M. Bron. Validating Query Simulators: An Experiment Using Commercial Searches and Purchases. In Agosti et al. {8}, pages 40--51. Google ScholarDigital Library
G. Inches and F. Crestani. Overview of the International Sexual Predator Identification Competition at PAN-2012. In Forner et al. {85}.Google Scholar
V. Jijkoun and M. de Rijke. Overview of WebCLEF 2007. In Peters et al. {207}, pages 725--731.Google Scholar
V. Jijkoun and M. de Rijke. Overview of WebCLEF 2008. In Peters et al. {200}, pages 787--793. Google ScholarDigital Library
A. Joly, H. Goëau, H. Glotin, C. Spampinato, P. Bonnet, W.-P. Vellinga, R. Planqué, A. Rauber, R. B. Fisher, and H. Müller. LifeCLEF 2014: Multimedia Life Species Identification Challenges. In Kanoulas et al. {129}, pages 229--249.Google Scholar
G. J. F. Jones and M. Federico. CLEF 2002 Cross-Language Spoken Document Retrieval Pilot Track Report. In Peters et al. {196}, pages 446--457.Google Scholar
P. Juola. An Overview of the Traditional Authorship Attribution Subtask. In Forner et al. {85}.Google Scholar
J. Jürgens, P. Hansen, and C. Womser-Hacker. Going beyond CLEF-IP: The 'Reality' for Patent Searchers. In Catarci et al. {53}, pages 30--35. Google ScholarDigital Library
J. Kalpathy-Cramer, H. Müller, S. Bedrick, I. Eggel, A. Garcia Seco de Herrera, and T. Tsikrika. Overview of the CLEF 2011 Medical Image Classification and Retrieval Tasks. In Petras et al. {214}.Google Scholar
E. Kanoulas, M. Lupu, P. Clough, M. Sanderson, M. Hall, A. Hanbury, and E. Toms, editors. Information Access Evaluation -- Multilinguality, Multimodality, and Interaction. Proceedings of the Fifth International Conference of the CLEF Initiative (CLEF 2014). Lecture Notes in Computer Science (LNCS) 8685, Springer, Heidelberg, Germany, 2014.Google Scholar
J. Karlgren, J. Gonzalo, and P. Clough. iCLEF 2006 Overview: Searching the Flickr WWW Photo-Sharing Repository. In Peters et al. {198}, pages 186--194. Google ScholarDigital Library
L. Kelly, L. Goeuriot, H. Suominen, T. Schreck, G. Leroy, D. L. Mowery, S. Velupillai, W. Webber Chapman, D. Martínez, G. Zuccon, and J. R. M. Palotti. Overview of the ShARe/CLEF eHealth Evaluation Lab 2014. In Kanoulas et al. {129}, pages 172--191.Google Scholar
A. Keszler, L. Kovács, and T. Szirányi. The Appearance of the Giant Component in Descriptor Graphs and Its Application for Descriptor Selection. In Catarci et al. {53}, pages 76--81. Google ScholarDigital Library
S.-J. Kim and J.-H. Lee. Subtopic Mining Based on Head-Modifier Relation and Co-occurrence of Intents Using Web Documents. In Forner et al. {86}, pages 179--191.Google Scholar
M. Kluck. The Domain-Specific Track in CLEF 2004: Overview of the Results and Remarks on the Assessment Process. In Peters et al. {199}, pages 260--270. Google ScholarDigital Library
M. Kluck and F. C. Gey. The Domain-Specific Task of CLEF -- Specific Evaluation Strategies in Cross-Language Information Retrieval. In Peters {193}, pages 48--56. Google ScholarDigital Library
M. Kluck and M. Stempfhuber. Domain-Specific Track CLEF 2005: Overview of Results and Approaches, Remarks on the Assessment Analysis. In Peters et al. {206}, pages 212--221. Google ScholarDigital Library
M. Koolen, G. Kazai, J. Kamps, M. Preminger, A. Doucet, and M. Landoni. Overview of the INEX 2012 Social Book Search Track. In Forner et al. {85}.Google Scholar
A. Kosmopoulos, G. Paliouras, and I. Androutsopoulos. The Effect of Dimensionality Reduction on Large Scale Hierarchical Classification. In Kanoulas et al. {129}, pages 160--171.Google Scholar
N. K. Kumar, G. S. K. Santosh, and V. Varma. A Language-Independent Approach to Identify the Named Entities in Under-Resourced Languages and Clustering Multilingual Documents. In Forner et al. {84}, pages 74--82. Google ScholarDigital Library
M. Kurimo, M. Creutz, and M. Varjokallio. Morpho Challenge Evaluation Using a Linguistic Gold Standard. In Peters et al. {207}, pages 864--872.Google Scholar
M. Kurimo, V. T. Turunen, and M. Varjokallio. Overview of Morpho Challenge 2008. In Peters et al. {200}, pages 951--966. Google ScholarDigital Library
M. Kurimo, S. Virpioja, V. T. Turunen, G. W. Blackwood, and W. Byrne. Overview and Results of Morpho Challenge 2009. In Peters et al. {201}, pages 587--597. Google ScholarDigital Library
J. Kürsten and M. Eibl. Comparing IR System Components Using Beanplots. In Catarci et al. {53}, pages 136--137. Google ScholarDigital Library
M. Kvist and S. Velupillai. SCAN: A Swedish Clinical Abbreviation Normalizer - Further Development and Adaptation to Radiology. In Kanoulas et al. {129}, pages 62--73.Google Scholar
M. Larson, E. Newman, and G. J. F. Jones. Overview of VideoCLEF 2008: Automatic Generation of Topic-Based Feeds for Dual Language Audio-Visual Content. In Peters et al. {200}, pages 906--917. Google ScholarDigital Library
M. Larson, E. Newman, and G. J. F. Jones. Overview of VideoCLEF 2009: New Perspectives on Speech-Based Multimedia Content Enrichment. In Peters et al. {209}, pages 354--368. Google ScholarDigital Library
L. A. Leiva, M. Villegas, and R. Paredes. Relevant Clouds: Leveraging Relevance Feedback to Build Tag Clouds for Image Search. In Forner et al. {86}, pages 143--149.Google Scholar
C. W. Leong, S. Hassan, M. E. Ruiz, and M. Rada. Improving Query Expansion for Image Retrieval via Saliency and Picturability. In Forner et al. {84}, pages 137--142. Google ScholarDigital Library
M. Lestari Paramita, M. Sanderson, and P. Clough. Diversity in Photo Retrieval: Overview of the Image-CLEFPhoto Task 2009. In Peters et al. {209}, pages 45--59. Google ScholarDigital Library
A. Lipani, F. Piroi, L. Andersson, and A. Hanbury. An Information Retrieval Ontology for Information Retrieval Nanopublications. In Kanoulas et al. {129}, pages 44--49.Google Scholar
A. Loponen and K. Järvelin. A Dictionary- and Corpus-Independent Statistical Lemmatizer for Information Retrieval in Low Resource Languages. In Agosti et al. {8}, pages 3--14. Google ScholarDigital Library
S. Mackie, R. McCreadie, C. Macdonald, and I. Ounis. Comparing Algorithms for Microblog Summarisation. In Kanoulas et al. {129}, pages 153--159.Google Scholar
W. Magdy and G. J. F. Jones. Examining the Robustness of Evaluation Metrics for Patent Retrieval with Incomplete Relevance Judgements. In Agosti et al. {8}, pages 82--93. Google ScholarDigital Library
B. Magnini, D. Giampiccolo, P. Forner, C. Ayache, V. Jijkoun, P. Osenova, A. Peñas, P. Rocha, B. Sacaleanu, and R. F. E. Sutcliffe. Overview of the CLEF 2006 Multilingual Question Answering Track. In Peters et al. {198}, pages 223--256. Google ScholarDigital Library
B. Magnini, S. Romagnoli, A. Vallin, J. Herrera, A. Peñas, V. Peinado, M. F. Verdejo, and M. de Rijke. The Multiple Language Question Answering Track at CLEF 2003. In Peters et al. {197}, pages 471--486.Google Scholar
B. Magnini, A. Vallin, C. Ayache, G. Erbach, A. Peñas, M. de Rijke, P. Rocha, K. I. Simov, and R. F. E. Sutcliffe. Overview of the CLEF 2004 Multilingual Question Answering Track. In Peters et al. {199}, pages 371--391. Google ScholarDigital Library
T. Mandl, M. Agosti, G. M. Di Nunzio, A. S. Yeh, I. Mani, C. Doran, and J. M. Schulz. LogCLEF 2009: The CLEF 2009 Multilingual Logfile Analysis Track Overview. In Peters et al. {201}, pages 508--517. Google ScholarDigital Library
T. Mandl, P. Carvalho, G. M. Di Nunzio, F. Gey, R. Larson, D. Santos, and C. Womser-Hacker. GeoCLEF 2008: The CLEF 2008 Cross-Language Geographic Information Retrieval Track Overview. In Peters et al. {200}, pages 808--821. Google ScholarDigital Library
T. Mandl, G. M. Di Nunzio, and J. M. Schulz. LogCLEF 2010: the CLEF 2010 Multilingual Logfile Analysis Track Overview. In Braschler et al. {46}.Google Scholar
T. Mandl, F. Gey, G. M. Di Nunzio, N. Ferro, R. Larson, M. Sanderson, D. Santos, C. Womser-Hacker, and X. Xie. GeoCLEF 2007: the CLEF 2007 Cross-Language Geographic Information Retrieval Track Overview. In Peters et al. {207}, pages 745--772.Google Scholar
J. Martnez-Gómez, I. García-Varea, and B. Caputo. Overview of the ImageCLEF 2012 Robot Vision Task. In Forner et al. {85}.Google Scholar
J. Mayfield, D. Lawrie, P. McNamee, and D. W. Oard. Building a Cross-Language Entity Linking Collection in Twenty-One Languages. In Forner et al. {84}, pages 3--13. Google ScholarDigital Library
R. McCreadie, C. Macdonald, I. Ounis, and J. Brassey. A Study of Personalised Medical Literature Search. In Kanoulas et al. {129}, pages 74--85.Google Scholar
M. R. Mirsarraf and N. Dehghani. A Dependency-Inspired Semantic Evaluation of Machine Translation Systems. In Forner et al. {86}, pages 71--74.Google Scholar
A. Molina, E. SanJuan, and J.-M. Torres-Moreno. A Turing Test to Evaluate a Complex Summarization Task. In Forner et al. {86}, pages 75--80.Google Scholar
R. Morante and W. Daelemans. Overview of the QA4MRE Pilot Task: Annotating Modality and Negation for a Machine Reading Evaluation. In Petras et al. {214}.Google Scholar
H. Müller, P. Clough, T. Deselaers, and B. Caputo, editors. ImageCLEF -- Experimental Evaluation in Visual Information Retrieval. Springer-Verlag, Heidelberg, Germany, 2010. Google ScholarDigital Library
H. Müller, T. Deselaers, T. M. Deserno, P. Clough, E. Kim, and W. R. Hersh. Overview of the Image- CLEFmed 2006 Medical Retrieval and Medical Annotation Tasks. In Peters et al. {198}, pages 595--608. Google ScholarDigital Library
H. Müller, T. Deselaers, T. M. Deserno, J. Kalpathy-Cramer, E. Kim, and W. R. Hersh. Overview of the ImageCLEFmed 2007 Medical Retrieval and Medical Annotation Tasks. In Peters et al. {207}, pages 472--491.Google Scholar
H. Müller, A. Garcia Seco de Herrera, J. Kalpathy-Cramer, D. Demner-Fushman, S. Antani, and I. Eggel. Overview of the ImageCLEF 2012 Medical Image Retrieval and Classification Tasks. In Forner et al. {85}.Google Scholar
H. Müller, J. Kalpathy-Cramer, I. Eggel, S. Bedrick, S. Radhouani, B. Bakke, C. E. Khan Jr., and W. R. Hersh. Overview of the CLEF 2009 Medical Image Retrieval Track. In Peters et al. {209}, pages 72--84. Google ScholarDigital Library
H. Müller, J. Kalpathy-Cramer, I. Eggel, S. Bedrick, J. Reisetter, C. E. Khan Jr., and W. R. Hersh. Overview of the CLEF 2010 Medical Image Retrieval Track. In Braschler et al. {46}.Google Scholar
H. Müller, J. Kalpathy-Cramer, C. E. Kahn, W. Hatt, S. Bedrick, and W. Hersh. Overview of the Image- CLEFmed 2008 Medical image Retrieval Task. In Peters et al. {200}, pages 512--522. Google ScholarDigital Library
A. Nardi, C. Peters, and N. Ferro, editors. CLEF 2007 Working Notes. CEUR Workshop Proceedings (CEUR-WS.org), ISSN 1613-0073, http://ceur-ws.org/Vol-1173/, 2007.Google Scholar
A. Nardi, C. Peters, J. L. Vicedo, and N. Ferro, editors. CLEF 2006 Working Notes. CEUR Workshop Proceedings (CEUR-WS.org), ISSN 1613-0073, http://ceur-ws.org/Vol-1172/, 2006.Google Scholar
R. Nordlie and N. Pharo. Seven Years of INEX Interactive Retrieval Experiments - Lessons and Challenges. In Catarci et al. {53}, pages 13--23. Google ScholarDigital Library
S. Nowak and P. Dunker. Overview of the CLEF 2009 Large-Scale Visual Concept Detection and Annotation Task. In Peters et al. {209}, pages 94--109. Google ScholarDigital Library
S. Nowak and M. J. Huiskes. New Strategies for Image Annotation: Overview of the Photo Annotation Task at ImageCLEF 2010. In Braschler et al. {46}.Google Scholar
S. Nowak, K. Nagel, and J. Liebetrau. The CLEF 2011 Photo Annotation and Concept-based Retrieval Tasks. In Petras et al. {214}.Google Scholar
D. W. Oard and J. Gonzalo. The CLEF 2001 Interactive Track. In Peters et al. {195}, pages 308--319. Google ScholarDigital Library
D. W. Oard and J. Gonzalo. The CLEF 2003 Interactive Track. In Peters et al. {197}, pages 425--434.Google Scholar
D. W. Oard, J. Wang, G. J. F. Jones, R. W. White, P. Pecina, D. Soergel, X. Huang, and I. Shafran. Overview of the CLEF-2006 Cross-Language Speech Retrieval Track. In Peters et al. {198}, pages 744--758. Google ScholarDigital Library
M.-D. Olvera-Lobo and J. Gutiérrez-Artacho. Multilingual Question-Answering System in Biomedical Domain on the Web: An Evaluation. In Forner et al. {84}, pages 83--88. Google ScholarDigital Library
N. Orio, C. C. S. Liem, G. Peeters, and M. Schedl. MusiClef: Multimodal Music Tagging Task. In Catarci et al. {53}, pages 36--41 Google ScholarDigital Library
N. Orio and D. Rizo. Overview of MusiCLEF 2011. In Petras et al. {214}.Google Scholar
P. Pecina, P. Hoffmannová, G. J. F. Jones, Y. Zhang, and D. W. Oard. Overview of the CLEF-2007 Cross-Language Speech Retrieval Track. In Peters et al. {207}, pages 674--686.Google Scholar
A. Peñas, P. Forner, A. Rodrigo, R. F. E. Sutcliffe, C. Forascu, and C. Mota. Overview of ResPubliQA 2010: Question Answering Evaluation over European Legislation. In Braschler et al. {46}.Google Scholar
A. Peñas, P. Forner, R. F. E. Sutcliffe, A. Rodrigo, C. Forascu, I. Alegria, D. Giampiccolo, N. Moreau, and P. Osenova. Overview of ResPubliQA 2009: Question Answering Evaluation over European Legislation. In Peters et al. {201}, pages 174--196. Google ScholarDigital Library
A. Peñas, E. H. Hovy, P. Forner, A. Rodrigo, R. F. E. Sutcliffe, C. Forascu, and C. Sporleder. Overview of QA4MRE at CLEF 2011: Question Answering for Machine Reading Evaluation. In Petras et al. {214}.Google Scholar
A. Peñas, E. H. Hovy, P. Forner, A. Rodrigo, R. F. E. Sutcliffe, and R. Morante. QA4MRE 2011-2013: Overview of Question Answering for Machine Reading Evaluation. In Forner et al. {86}, pages 303--320.Google Scholar
A. Peñas, E. H. Hovy, P. Forner, A. Rodrigo, R. F. E. Sutcliffe, C. Sporleder, C. Forascu, Y. Benajiba, and P. Osenova. Overview of QA4MRE at CLEF 2012: Question Answering for Machine Reading Evaluation. In Forner et al. {85}.Google Scholar
A. Peñas, C. Unger, and A.-C. and Ngonga Ngomo. Overview of CLEF Question Answering Track 2014. In Kanoulas et al. {129}, pages 300--306.Google Scholar
C. Peters, editor. Cross-Language Information Retrieval and Evaluation: Workshop of Cross-Language Evaluation Forum (CLEF 2000). Lecture Notes in Computer Science (LNCS) 2069, Springer, Heidelberg, Germany, 2001.Google Scholar
C. Peters, M. Braschler, and P. Clough. Multilingual Information Retrieval. Springer-Verlag, Heidelberg, Germany, 2011.Google Scholar
C. Peters, M. Braschler, J. Gonzalo, and M. Kluck, editors. Evaluation of Cross-Language Information Retrieval Systems: Second Workshop of the Cross--Language Evaluation Forum (CLEF 2001) Revised Papers. Lecture Notes in Computer Science (LNCS) 2406, Springer, Heidelberg, Germany, 2002.Google ScholarCross Ref
C. Peters, M. Braschler, J. Gonzalo, and M. Kluck, editors. Advances in Cross-Language Information Retrieval: Third Workshop of the Cross--Language Evaluation Forum (CLEF 2002) Revised Papers. Lecture Notes in Computer Science (LNCS) 2785, Springer, Heidelberg, Germany, 2003.Google ScholarCross Ref
C. Peters, M. Braschler, J. Gonzalo, and M. Kluck, editors. Comparative Evaluation of Multilingual Information Access Systems: Fourth Workshop of the Cross-Language Evaluation Forum (CLEF 2003) Revised Selected Papers. Lecture Notes in Computer Science (LNCS) 3237, Springer, Heidelberg, Germany, 2004. Google ScholarDigital Library
C. Peters, P. Clough, F. C. Gey, J. Karlgren, B. Magnini, D. W. Oard, M. de Rijke, and M. Stempfhuber, editors. Evaluation of Multilingual and Multi-modal Information Retrieval : Seventh Workshop of the Cross-- Language Evaluation Forum (CLEF 2006). Revised Selected Papers. Lecture Notes in Computer Science (LNCS) 4730, Springer, Heidelberg, Germany, 2007. Google ScholarDigital Library
C. Peters, P. Clough, J. Gonzalo, G. J. F. Jones, M. Kluck, and B. Magnini, editors. Multilingual Information Access for Text, Speech and Images: Fifth Workshop of the Cross-Language Evaluation Forum (CLEF 2004) Revised Selected Papers. Lecture Notes in Computer Science (LNCS) 3491, Springer, Heidelberg, Germany, 2005. Google ScholarDigital Library
C. Peters, T. Deselaers, N. Ferro, J. Gonzalo, G. J. F. Jones, M. Kurimo, T. Mandl, and A. Peñas, editors. Evaluating Systems for Multilingual and Multimodal Information Access: Ninth Workshop of the Cross-Language Evaluation Forum (CLEF 2008). Revised Selected Papers. Lecture Notes in Computer Science (LNCS) 5706, Springer, Heidelberg, Germany, 2009. Google ScholarDigital Library
C. Peters, G. M. Di Nunzio, M. Kurimo, T. Mandl, D. Mostefa, A. Peñas, and G. Roda, editors. Multilingual Information Access Evaluation Vol. I Text Retrieval Experiments -- Tenth Workshop of the Cross--Language Evaluation Forum (CLEF 2009). Revised Selected Papers. Lecture Notes in Computer Science (LNCS) 6241, Springer, Heidelberg, Germany, 2010. Google ScholarDigital Library
C. Peters and N. Ferro, editors. CLEF 2000 Working Notes. CEURWorkshop Proceedings (CEUR-WS.org), ISSN 1613-0073, http://ceur-ws.org/Vol-1166/, 2000.Google Scholar
C. Peters and N. Ferro, editors. CLEF 2001 Working Notes. CEURWorkshop Proceedings (CEUR-WS.org), ISSN 1613-0073, http://ceur-ws.org/Vol-1167/, 2001.Google Scholar
C. Peters and N. Ferro, editors. CLEF 2002 Working Notes. CEURWorkshop Proceedings (CEUR-WS.org), ISSN 1613-0073, http://ceur-ws.org/Vol-1168/, 2002.Google ScholarCross Ref
C. Peters and N. Ferro, editors. CLEF 2003 Working Notes. CEURWorkshop Proceedings (CEUR-WS.org), ISSN 1613-0073, http://ceur-ws.org/Vol-1169/, 2003.Google Scholar
C. Peters, F. C. Gey, J. Gonzalo, G. J. F. Jones, M. Kluck, B. Magnini, H. Müller, and M. de Rijke, editors. Accessing Multilingual Information Repositories: Sixth Workshop of the Cross-Language Evaluation Forum (CLEF 2005). Revised Selected Papers. Lecture Notes in Computer Science (LNCS) 4022, Springer, Heidelberg, Germany, 2006. Google ScholarDigital Library
C. Peters, V. Jijkoun, T. Mandl, H. Müller, D. W. Oard, A. Peñas, V. Petras, and D. Santos, editors. Advances in Multilingual and Multimodal Information Retrieval: Eighth Workshop of the Cross-Language Evaluation Forum (CLEF 2007). Revised Selected Papers. Lecture Notes in Computer Science (LNCS) 5152, Springer, Heidelberg, Germany, 2008. Google ScholarDigital Library
C. Peters, V. Quochi, and N. Ferro, editors. CLEF 2005 Working Notes. CEUR Workshop Proceedings (CEUR-WS.org), ISSN 1613-0073, http://ceur-ws.org/Vol-1171/, 2005.Google Scholar
C. Peters, T. Tsikrika, H. Müller, J. Kalpathy-Cramer, G. J. F. Jones, J. Gonzalo, and B. Caputo, editors. Multilingual Information Access Evaluation Vol. II Multimedia Experiments -- Tenth Workshop of the Cross-Language Evaluation Forum (CLEF 2009). Revised Selected Papers. Lecture Notes in Computer Science (LNCS), Springer, Heidelberg, Germany, 2010. Google ScholarDigital Library
V. Petras and S. Baerisch. The Domain-Specific Track at CLEF 2008. In Peters et al. {200}, pages 186--198. Google ScholarDigital Library
V. Petras, S. Baerisch, and M. Stempfhuber. The Domain-Specific Track at CLEF 2007. In Peters et al. {207}, pages 160--173.Google Scholar
V. Petras, T. Bogers, M. Hall, J. Savoy, P. Malak, A. Pawlowski, N. Ferro, and I. Masiero. Cultural Heritage in CLEF (CHiC) 2013. In Forner et al. {86}, pages 192--211.Google Scholar
V. Petras, N. Ferro, M. Gäde, A. Isaac, M. Kleineberg, I. Masiero, M. Nicchio, and J. Stiller. Cultural Heritage in CLEF (CHiC) Overview 2012. In Forner et al. {85}.Google Scholar
V. Petras, P. Forner, P. Clough, and N. Ferro, editors. CLEF 2011 Working Notes. CEUR Workshop Proceedings (CEUR-WS.org), ISSN 1613-0073, http://ceur-ws.org/Vol-1177/, 2011.Google Scholar
F. Piroi. CLEF-IP 2010: Retrieval Experiments in the Intellectual Property Domain. In Braschler et al. {46}.Google Scholar
F. Piroi, M. Lupu, and A. Hanbury. Effects of Language and Topic Size in Patent IR: An Empirical Study. In Catarci et al. {53}, pages 54--66. Google ScholarDigital Library
F. Piroi, M. Lupu, and A. Hanbury. Overview of CLEF-IP 2013 Lab - Information Retrieval in the Patent Domain. In Forner et al. {86}, pages 232--249.Google Scholar
F. Piroi, M. Lupu, A. Hanbury, A. P. Sexton, W. Magdy, and I. V. Filippov. CLEF-IP 2012: Retrieval Experiments in the Intellectual Property Domain. In Forner et al. {85}.Google Scholar
F. Piroi, M. Lupu, A. Hanbury, and V. Zenz. CLEF-IP 2011: Retrieval in the Intellectual Property Domain. In Petras et al. {214}.Google Scholar
A. Popescu, T. Tsikrika, and J. Kludas. Overview of the Wikipedia Retrieval Task at ImageCLEF 2010. In Braschler et al. {46}.Google Scholar
M. Potthast, A. Barrón-Cedeño, A. Eiselt, B. Stein, and P. Rosso. Overview of the 2nd International Competition on Plagiarism Detection. In Braschler et al. {46}.Google Scholar
M. Potthast, A. Eiselt, A. Barrón-Cedeño, B. Stein, and P. Rosso. Overview of the 3rd International Competition on Plagiarism Detection. In Petras et al. {214}.Google Scholar
M. Potthast, T. Gollub, M. Hagen, J. Kiesel, M. Michel, A. Oberländer, M. Tippmann, A. Barrón-Cedeño, P. Gupta, P. Rosso, and B. Stein. Overview of the 4th International Competition on Plagiarism Detection. In Forner et al. {85}.Google Scholar
M. Potthast, T. Gollub, F. Rangel Pardo, P. Rosso, E. Stamatatos, and B. Stein. Improving the Reproducibility of PAN's Shared Tasks: Plagiarism Detection, Author Identification, and Author Profiling. In Kanoulas et al. {129}, pages 268--299.Google Scholar
M. Potthast and T. Holfeld. Overview of the 2nd International Competition on Wikipedia Vandalism Detection. In Petras et al. {214}.Google Scholar
M. Potthast, B. Stein, and T. Holfeld. Overview of the 1st International Competition on Wikipedia Vandalism Detection. In Braschler et al. {46}.Google Scholar
A. Pronobis, M. Fornoni, H. I. Christensen, and B. Caputo. The Robot Vision Track at ImageCLEF 2010. In Braschler et al. {46}.Google Scholar
A. Pronobis, L. Xing, and B. Caputo. Overview of the CLEF 2009 Robot Vision Track. In Peters et al. {209}, pages 110--119. Google ScholarDigital Library
D. Rebholz-Schuhmann, S. Clematide, F. Rinaldi, S. Kafkas, E. M. van Mulligen, Q.-C. Bui, J. Hellrich, J. Lewin, D. Milward, M. Poprat, A. Jimeno-Yepes, U. Hahn, and J. A. Kors. Entity Recognition in Parallel Multi-lingual Biomedical Corpora: The CLEF-ER Laboratory Overview. In Forner et al. {86}, pages 353--367.Google Scholar
N. Rekabsaz and M. Lupu. A Real-World Framework for Translator as Expert Retrieval. In Kanoulas et al. {129}, pages 141--152.Google Scholar
G. Roda, J. Tait, F. Piroi, and V. Zenz. CLEF-IP 2009: Retrieval Experiments in the Intellectual Property Domain. In Peters et al. {201}, pages 385--409. Google ScholarDigital Library
A. Rodrigo, A. Peñas, and M. F. Verdejo. Overview of the Answer Validation Exercise 2008. In Peters et al. {200}, pages 296--313. Google ScholarDigital Library
R. Roller and M. Stevenson. Self-supervised Relation Extraction Using UMLS. In Kanoulas et al. {129}, pages 116--127.Google Scholar
B. R. Rowe, D. W. Wood, A. L. Link, and D. A. Simoni. Economic Impact Assessment of NIST's Text REtrieval Conference (TREC) Program. RTI Project Number 0211875, RTI International, USA. http://trec.nist.gov/pubs/2010.economic.impact.pdf, July 2010.Google Scholar
S. Sabetghadam, R. Bierig, and A. Rauber. A Hybrid Approach for Multi-faceted IR in Multimodal Domain. In Kanoulas et al. {129}, pages 86--97.Google Scholar
E. SanJuan, V. Moriceau, X. Tannier, P. Bellot, and J. Mothe. Overview of the INEX 2012 Tweet Contextualization Track. In Forner et al. {85}.Google Scholar
D. Santos and L. M. Cabral. GikiCLEF: Expectations and Lessons Learned. In Peters et al. {201}, pages 212--222. Google ScholarDigital Library
D. Savenkov, P. Braslavski, and M. Lebedev. Search Snippet Evaluation at Yandex: Lessons Learned and Future Directions. In Forner et al. {84}, pages 14--25. Google ScholarDigital Library
P. Schaer. Better than Their Reputation? On the Reliability of Relevance Assessments with Students. In Catarci et al. {53}, pages 124--135. Google ScholarDigital Library
P. Schaüble and P. Sheridan. Cross-Language Information Retrieval (CLIR) Track Overview. In E. M. Voorhees and D. K. Harman, editors, The Sixth Text REtrieval Conference (TREC-6), pages 31--44. National Institute of Standards and Technology (NIST), Special Publication 500-240, Washington, USA., 1997.Google Scholar
A. Schuth and M. Marx. Evaluation Methods for Rankings of Facetvalues for Faceted Search. In Forner et al. {84}, pages 131--136. Google ScholarDigital Library
B. Sigurbjörnsson, J. Kamps, and M. de Rijke. Overview of WebCLEF 2005. In Peters et al. {206}, pages 810--824. Google ScholarDigital Library
P. Sorg, P. Cimiano, A. Schultz, and S. Sizov. Overview of the Cross-lingual Expert Search (CriES) Pilot Challenge. In Braschler et al. {46}.Google Scholar
D. Spina, E. Amigó, and J. Gonzalo. Filter Keywords and Majority Class Strategies for Company Name Disambiguation in Twitter. In Forner et al. {84}, pages 38--49. Google ScholarDigital Library
V. Stefanov, A. Sachs, M. Kritz, M. Samwald, M. Gschwandtner, and A. Hanbury. A Formative Evaluation of a Comprehensive Search System for Medical Professionals. In Forner et al. {86}, pages 81--92.Google Scholar
M. Stempfhuber and S. Baerisch. The Domain-Specific Track at CLEF 2006: Overview of Approaches, Results and Assessment. In Peters et al. {198}, pages 163--169. Google ScholarDigital Library
H. Suominen. CLEFeHealth2012 - The CLEF 2012 Workshop on Cross-Language Evaluation of Methods, Applications, and Resources for eHealth Document Analysis. In Forner et al. {85}.Google Scholar
H. Suominen, S. Salanterä, S. Velupillai, W. Webber Chapman, G. K. Savova, N. Elhadad, S. Pradhan, B. R. South, D. L. Mowery, G. J. F. Jones, J. Leveling, L. Kelly, L. Goeuriot, D. Martínez, and G. Zuccon. Overview of the ShARe/CLEF eHealth Evaluation Lab 2013. In Forner et al. {86}, pages 212--231.Google Scholar
W. Tannebaum and A. Rauber. Mining Query Logs of USPTO Patent Examiners. In Forner et al. {86}, pages 136--142.Google Scholar
B. Thomee and A. Popescu. Overview of the ImageCLEF 2012 Flickr Photo Annotation and Retrieval Task. In Forner et al. {85}.Google Scholar
C. V. Thornley, A. C. Johnson, A. F. Smeaton, and H. Lee. The Scholarly Impact of TRECVid (2003-2009). Journal of the American Society for Information Science and Technology (JASIST), 62(4):613--627, April 2011. Google ScholarDigital Library
T. Tommasi, B. Caputo, P. Welter, M. O. Güld, and T. M. Deserno. Overview of the CLEF 2009 Medical Image Annotation Track. In Peters et al. {209}, pages 85--93. Google ScholarDigital Library
M. Trappett, S. Geva, A. Trotman, F. Scholer, and M. Sanderson. Overview of the INEX 2012 Snippet Retrieval Track. In Forner et al. {85}.Google Scholar
T. Tsikrika, A. Garcia Seco de Herrera, and H. Müller. Assessing the Scholarly Impact of ImageCLEF. In Forner et al. {84}, pages 95--106. Google ScholarDigital Library
T. Tsikrika and J. Kludas. Overview of the WikipediaMM Task at ImageCLEF 2008. In Peters et al. {200}, pages 539--550. Google ScholarDigital Library
T. Tsikrika and J. Kludas. Overview of the WikipediaMM Task at ImageCLEF 2009. In Peters et al. {209}, pages 60--71 Google ScholarDigital Library
T. Tsikrika, B. Larsen, H. Müller, S. Endrullis, and E. Rahm. The Scholarly Impact of CLEF (2000--2009). In Forner et al. {86}, pages 1--12.Google Scholar
T. Tsikrika, A. Popescu, and J. Kludas. Overview of the Wikipedia Image Retrieval Task at ImageCLEF 2011. In Petras et al. {214}.Google Scholar
M. Turchi, J. Steinberger, M. Alexandrov Kabadjov, and R. Steinberger. Using Parallel Corpora for Multilingual (Multi-document) Summarisation Evaluation. In Agosti et al. {8}, pages 52--63. Google ScholarDigital Library
J. Turmo, P. Comas, S. Rosset, O. Galibert, N. Moreau, D. Mostefa, P. Rosso, and D. Buscaldi. Overview of QAST 2009. In Peters et al. {201}, pages 197--211. Google ScholarDigital Library
J. Turmo, P. Comas, S. Rosset, L. Lamel, N. Moreau, and D. Mostefa. Overview of QAST 2008. In Peters et al. {200}, pages 296--313. Google ScholarDigital Library
A. Vallin, B. Magnini, D. Giampiccolo, L. Aunimo, C. Ayache, P. Osenova, A. Peñas, M. de Rijke, B. Sacaleanu, D. Santos, and R. F. E. Sutcliffe. Overview of the CLEF 2005 Multilingual Question Answering Track. In Peters et al. {206}, pages 307--331. Google ScholarDigital Library
F. Valverde-Albacete, J. Carrillo de Albornoz, and C. Peláez-Moreno. A Proposal for New Evaluation Metrics and Result Visualization Technique for Sentiment Analysis Tasks. In Forner et al. {86}, pages 41--42.Google Scholar
M. Villegas and R. Paredes. Overview of the ImageCLEF 2012 Scalable Web Image Annotation Task. In Forner et al. {85}.Google Scholar
E. M. Voorhees. TREC: Continuing Information Retrieval's Tradition of Experimentation. Communications of the ACM (CACM), 50(11):51--54, November 2007. Google ScholarDigital Library
A. Walker, A. Starkey, J. Z. Pan, and A. Siddharthan. Making Test Corpora for Question Answering More Representative. In Kanoulas et al. {129}, pages 1--6.Google Scholar
Q. Wang, J. Kamps, G. Ramírez Camps, M. Marx, A. Schuth, M. Theobald, S. Gurajada, and A. Mishra. Overview of the INEX 2012 Linked Data Track. In Forner et al. {85}.Google Scholar
X. Wang, Wang. X., and Q. Zhang. A Web-Based CLIR System with Cross-Lingual Topical Pseudo Relevance Feedback. In Forner et al. {86}, pages 104--107.Google Scholar
R. W. White, D. W. Oard, G. J. F. Jones, D. Soergel, and X. Huang. Overview of the CLEF-2005 Cross- Language Speech Retrieval Track. In Peters et al. {206}, pages 744--759. Google ScholarDigital Library
T. Wilhelm-Stein and M. Eibl. A Quantitative Look at the CLEF Working Notes. In Forner et al. {86}, pages 13--16.Google Scholar
T. Wilhelm-Stein, R. Herms, M. Ritter, and M. Eibl. Improving Transcript-Based Video Retrieval Using Unsupervised Language Model Adaptation. In Kanoulas et al. {129}, pages 110--115.Google Scholar
X. Yan, G. Gao, X. Su, H. Wei, X. Zhang, and Q. Lu. Hidden Markov Model for Term Weighting in Verbose Queries. In Catarci et al. {53}, pages 82--87. Google ScholarDigital Library
H. Zamani, H. N. Esfahani, P. Babaie, S. Abnar, M. Dehghani, and A. Shakery. Authorship Identification Using Dynamic Selection of Features from Probabilistic Feature Set. In Kanoulas et al. {129}, pages 128--140.Google Scholar
D. Zellhöfer. Overview of the Personal Photo Retrieval Pilot Task at ImageCLEF 2012. In Forner et al. {85}.Google Scholar
L. Zhang, A. Rettinger, M. Färber, and M. Tadic. A Comparative Evaluation of Cross-Lingual Text Annotation Techniques. In Forner et al. {86}, pages 124--135.Google Scholar

Recommendations

Dublin city university at CLEF 2005: experiments with the ImageCLEF st andrew’s collection
CLEF'05: Proceedings of the 6th international conference on Cross-Language Evalution Forum: accessing Multilingual Information Repositories

The aim of the Dublin City University’s participation in the CLEF 2005 ImageCLEF St Andrew’s Collection task was to explore an alternative approach to exploiting text annotation and content-based retrieval in a novel combined way for pseudo relevance ...
Read More
Model Fusion Experiments for the CLSR Task at CLEF 2007
Advances in Multilingual and Multimodal Information Retrieval

This paper presents the participation of the University of Ottawa group in the Cross-Language Speech Retrieval (CL-SR) task at CLEF 2007. We present the results of the submitted runs for the English collection. We have used two Information Retrieval ...
Read More
Some results using different approaches to merge visual and text-based features in CLEF'08 photo collection
CLEF'08: Proceedings of the 9th Cross-language evaluation forum conference on Evaluating systems for multilingual and multimodal information access

This paper describes the participation of the MIRACLE team at the ImageCLEF Photographic Retrieval task of CLEF 2008. We succeeded in submitting 41 runs. Obtained results from text-based retrieval are better than content-based as previous experiments in ...
Read More

Comments

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

Published in
ACM SIGIR Forum Volume 48, Issue 2
December 2014
107 pages
ISSN:0163-5840
DOI:10.1145/2701583
Editors:
Ben Carterette
University of Delaware, Newark, DE, USA
,
Craig Macdonald
University of Glasgow, Glasgow, UK
Issue’s Table of Contents
Copyright © 2014 Author
Sponsors
In-Cooperation
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
- Published: 23 December 2014
Check for updates
Qualifiers
- research-article
Conference
Funding Sources
Other Metrics
View Article Metrics

Article Metrics
- 15
  Total Citations
  View Citations
- 122
  Total Downloads
- Downloads (Last 12 months)5
- Downloads (Last 6 weeks)0
Other Metrics
View Author Metrics
Cited By
View all

PDF Format

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

CLEF 15th Birthday: Past, Present, and Future

ACM SIGIR Forum

Abstract

References

Cited By

Recommendations

Dublin city university at CLEF 2005: experiments with the ImageCLEF st andrew’s collection

Model Fusion Experiments for the CLSR Task at CLEF 2007

Some results using different approaches to merge visual and text-based features in CLEF'08 photo collection