Optimizing search engines using clickthrough data

Subscribe (Full Service)


	Search: The ACM Digital Library The Guide

	Feedback

Optimizing search engines using clickthrough data

Full text

Pdf (954 KB)

Source

Conference on Knowledge Discovery in Data archive
Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining table of contents

Edmonton, Alberta, Canada

SESSION: Web search and navigation table of contents

Pages: 133 - 142

Year of Publication: 2002

ISBN:1-58113-567-X

Author

Thorsten Joachims

Cornell University, Ithaca, NY

Sponsors

SIGKDD: ACM Special Interest Group on Knowledge Discovery in Data
SIGMOD: ACM Special Interest Group on Management of Data
: AAAI

Publisher

ACM New York, NY, USA

Bibliometrics

Downloads (6 Weeks): 56, Downloads (12 Months): 577, Citation Count: 122

Additional Information:

abstract references cited by index terms collaborative colleagues peer to peer

Tools and Actions:	Review this Article Save this Article to a Binder Display Formats: BibTex EndNote ACM Ref

DOI Bookmark:	Use this link to bookmark this Article: http://doi.acm.org/10.1145/775047.775067 What is a DOI?

ABSTRACT

This paper presents an approach to automatically optimizing the retrieval quality of search engines using clickthrough data. Intuitively, a good information retrieval system should present relevant documents high in the ranking, with less relevant documents following below. While previous approaches to learning retrieval functions from examples exist, they typically require training data generated from relevance judgments by experts. This makes them difficult and expensive to apply. The goal of this paper is to develop a method that utilizes clickthrough data for training, namely the query-log of the search engine in connection with the log of links the users clicked on in the presented ranking. Such clickthrough data is available in abundance and can be recorded at very low cost. Taking a Support Vector Machine (SVM) approach, this paper presents a method for learning retrieval functions. From a theoretical perspective, this method is shown to be well-founded in a risk minimization framework. Furthermore, it is shown to be feasible even for large sets of queries and features. The theoretical results are verified in a controlled experiment. It shows that the method can effectively adapt the retrieval function of a meta-search engine to a particular group of users, outperforming Google in terms of retrieval quality after only a couple of hundred training examples.

REFERENCES

Note: OCR errors may be found in this Reference List extracted from the full text article. ACM has opted to expose the complete List rather than only correct and linked references.

	1	Ricardo A. Baeza-Yates , Berthier Ribeiro-Neto, Modern Information Retrieval, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1999
	2	Brian T. Bartell , Garrison W. Cottrell , Richard K. Belew, Automatic combination of multiple ranked retrieval systems, Proceedings of the 17th annual international ACM SIGIR conference on Research and development in information retrieval, p.173-181, July 03-06, 1994, Dublin, Ireland
	3	Doug Beeferman , Adam Berger, Agglomerative clustering of a search engine query log, Proceedings of the sixth ACM SIGKDD international conference on Knowledge discovery and data mining, p.407-416, August 20-23, 2000, Boston, Massachusetts, United States [doi>10.1145/347090.347176]
	4	Bernhard E. Boser , Isabelle M. Guyon , Vladimir N. Vapnik, A training algorithm for optimal margin classifiers, Proceedings of the fifth annual workshop on Computational learning theory, p.144-152, July 27-29, 1992, Pittsburgh, Pennsylvania, United States [doi>10.1145/130385.130401]
	5	J. Boyan, D. Freitag, and T. Joachims. A machine learning architecture for optimizing web search engines. In AAAI Workshop on Internet Based Information Systems, August 1996.
	6	W. Cohen, R. Shapire, and Y. Singer. Learning to order things. Journal of Artificial Intelligence Research, 10, 1999.
	7	Corinna Cortes , Vladimir Vapnik, Support-Vector Networks, Machine Learning, v.20 n.3, p.273-297, Sept. 1995 [doi>10.1023/A:1022627411411]
	8	K. Crammer and Y. Singer. Pranking with ranking. In Advances in Neural Information Processing Systems (NIPS), 2001.
	9	Yoav Freund , Raj D. Iyer , Robert E. Schapire , Yoram Singer, An Efficient Boosting Algorithm for Combining Preferences, Proceedings of the Fifteenth International Conference on Machine Learning, p.170-178, July 24-27, 1998
	10	Norbert Fuhr, Optimum polynomial retrieval functions based on the probability ranking principle, ACM Transactions on Information Systems (TOIS), v.7 n.3, p.183-204, July 1989 [doi>10.1145/65943.65944]
	11	N. Fuhr, S. Hartmann, G. Lustig, M. Schwantner, K. Tzeras, and G. Knorz. Air/x - a rule-based multistage indexing system for large subject fields. In RIAO, pages 606--623, 1991.
	12	R. Herbrich, T. Graepel, and K. Obermayer. Large margin rank boundaries for ordinal regression. In Advances in Large Margin Classifiers, pages 115--132. MIT Press, Cambridge, MA, 2000.
	13	Klaus-U. Höffgen , Hans-U. Simon , Kevin S. van Horn, Robust trainability of single neurons, Journal of Computer and System Sciences, v.50 n.1, p.114-125, Feb. 1995 [doi>10.1006/jcss.1995.1011 ]
	14	Thorsten Joachims, Making large-scale support vector machine learning practical, Advances in kernel methods: support vector learning, MIT Press, Cambridge, MA, 1999
	15	Thorsten Joachims, Learning to Classify Text Using Support Vector Machines: Methods, Theory and Algorithms, Kluwer Academic Publishers, Norwell, MA, 2002
	16	T. Joachims. Unbiased evaluation of retrieval quality using clickthrough data. Technical report, Cornell University, Department of Computer Science, 2002. http://www.joachims.org.
	17	T. Joachims, D. Freitag, and T. Mitchell. WebWatcher: a tour guide for the world wide web. In Proceedings of International Joint Conference on Artificial Intelligence (IJCAI), volume 1, pages 770--777. Morgan Kaufmann, 1997.
	18	J. Kemeny and L. Snell. Mathematical Models in the Social Sciences. Ginn & Co, 1962.
	19	M. Kendall. Rank Correlation Methods. Hafner, 1955.
	20	H. Lieberman. Letizia: An agent that assists Web browsing. In Proceedings of the Fifteenth International Joint Conference on Artificial Intelligence (IJCAI '95), Montreal, Canada, 1995. Morgan Kaufmann.
	21	A. Mood, F. Graybill, and D. Boes. Introduction to the Theory of Statistics. McGraw-Hill, 3 edition, 1974.
	22	L. Page and S. Brin. Pagerank, an eigenvector based ranking approach for hypertext. In 2lst Annual ACM/SIGIR International Conference on Research and Development in Information Retrieval, 1998.
	23	Gerard Salton , Christopher Buckley, Term-weighting approaches in automatic text retrieval, Information Processing and Management: an International Journal, v.24 n.5, p.513-523, 1988 [doi>10.1016/0306-4573(88)90021-0 ]
	24	C. Silverstein, M. Henzinger, H. Marais, and M. Moricz. Analysis of a very large altavista query log. Technical Report SRC 1998-014, Digital Systems Research Center, 1998.
	25	V. Vapnik. Statistical Learning Theory. Wiley, Chichester, GB, 1998.
	26	Y. Y. Yao, Measuring retrieval effectiveness based on user preference of documents, Journal of the American Society for Information Science, v.46 n.2, p.133-145, March 1995 [doi>10.1002/(SICI)1097-4571(199503)46:2<133::AID-ASI6>3.0.CO;2-Z]

CITED BY 122

	Rosie Jones , Daniel C. Fain, Query word deletion prediction, Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval, July 28-August 01, 2003, Toronto, Canada
	Jean-David Ruvini, Adapting to the user's internet search strategy on small devices, Proceedings of the 8th international conference on Intelligent user interfaces, January 12-15, 2003, Miami, Florida, USA
	Laura A. Granka , Thorsten Joachims , Geri Gay, Eye-tracking analysis of user behavior in WWW search, Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval, July 25-29, 2004, Sheffield, United Kingdom
	Xuehua Shen , Bin Tan , ChengXiang Zhai, UCAIR: a personalized search toolbar, Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, August 15-19, 2005, Salvador, Brazil
	Ciya Liao , Thomas Chang, Applying ranking SVM in query relaxation, Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, July 23-27, 2007, Amsterdam, The Netherlands
	Bin Tan , Xuehua Shen , ChengXiang Zhai, Mining long-term search history to improve search accuracy, Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, August 20-23, 2006, Philadelphia, PA, USA
	Shane Ahern , Simon King , Mor Naaman , Rahul Nair, Summarization of online image collections via implicit feedback, Proceedings of the 16th international conference on World Wide Web, May 08-12, 2007, Banff, Alberta, Canada
	Omid Madani , Dennis DeCoste, Contextual recommender problems [extended abstract], Proceedings of the 1st international workshop on Utility-based data mining, p.86-89, August 21-21, 2005, Chicago, Illinois
	Christophe Foussette , Daniel Hakenjos , Martin Scholz, KDD-Cup 2004: protein homology task, ACM SIGKDD Explorations Newsletter, v.6 n.2, December 2004
	Toshihiro Kamishima, Nantonac collaborative filtering: recommendation based on order responses, Proceedings of the ninth ACM SIGKDD international conference on Knowledge discovery and data mining, August 24-27, 2003, Washington, D.C.
	Sara Cohen , Carmel Domshlak , Naama Zwerdling, On ranking techniques for desktop search, Proceedings of the 16th international conference on World Wide Web, May 08-12, 2007, Banff, Alberta, Canada
	Rong Jin , Hamed Valizadegan , Hang Li, Ranking refinement and its application to information retrieval, Proceeding of the 17th international conference on World Wide Web, April 21-25, 2008, Beijing, China
	Aaron Archer Waterman , Jessie Burger , Mark Strohmaier, Application of learned user context to improve web search results, Journal of Computing Sciences in Colleges, v.19 n.5, p.321-323, May 2004
	Xin Fu, Evaluating sources of implicit feedback in web searches, Proceedings of the 2007 ACM conference on Recommender systems, October 19-20, 2007, Minneapolis, MN, USA
	Nick Craswell , Onno Zoeter , Michael Taylor , Bill Ramsey, An experimental comparison of click position-bias models, Proceedings of the international conference on Web search and web data mining, February 11-12, 2008, Palo Alto, California, USA
	Jingfang Xu , Xing Li, Learning to rank collections, Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, July 23-27, 2007, Amsterdam, The Netherlands
	Zhiwei Li , Shuming Shi , Lei Zhang, Improving relevance judgment of web search results with image excerpts, Proceeding of the 17th international conference on World Wide Web, April 21-25, 2008, Beijing, China
	Xuehua Shen , ChengXiang Zhai, Active feedback in ad hoc information retrieval, Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, August 15-19, 2005, Salvador, Brazil
	Donald Metzler , W. Bruce Croft, A Markov random field model for term dependencies, Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, August 15-19, 2005, Salvador, Brazil
	Qiankun Zhao , Tie-Yan Liu , Sourav S. Bhowmick , Wei-Ying Ma, Event detection from evolution of click-through data, Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, August 20-23, 2006, Philadelphia, PA, USA
	Nick Craswell , Martin Szummer, Random walks on the click graph, Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, July 23-27, 2007, Amsterdam, The Netherlands
	Hiranmay Ghosh , P. Poornachander , Anupama Mallik , Santanu Chaudhury, Learning ontology for personalized video retrieval, Workshop on multimedia information retrieval on The many faces of multimedia semantics, September 28-28, 2007, Augsburg, Bavaria, Germany
	Edson J. R. Justino , Flávio Bortolozzi , Robert Sabourin, A comparison of SVM and HMM classifiers in the off-line signature verification, Pattern Recognition Letters, v.26 n.9, p.1377-1385, 1 July 2005
	Regina Barzilay , Mirella Lapata, Modeling local coherence: an entity-based approach, Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics, p.141-148, June 25-30, 2005, Ann Arbor, Michigan
	Steve Fox , Kuldeep Karnawat , Mark Mydland , Susan Dumais , Thomas White, Evaluating implicit measures to improve web search, ACM Transactions on Information Systems (TOIS), v.23 n.2, p.147-168, April 2005
	Pedram Ghodsnia , Ali Mohammad Zareh Bidoki , Nasser Yazdani, A punishment/reward based approach to ranking, Proceedings of the 2nd international conference on Scalable information systems, June 06-08, 2007, Suzhou, China
	Philip Zigoris , Yi Zhang, Bayesian adaptive user profiling with explicit & implicit feedback, Proceedings of the 15th ACM international conference on Information and knowledge management, November 06-11, 2006, Arlington, Virginia, USA
	Rafael Alonso , Jeffrey A. Bloom , Hua Li , Chumki Basu, An adaptive nearest neighbor search for a parts acquisition ePortal, Proceedings of the ninth ACM SIGKDD international conference on Knowledge discovery and data mining, August 24-27, 2003, Washington, D.C.
	Ingo Mierswa , Katharina Morik, Method trees: building blocks for self-organizable representations of value series: how to evolve representations for classifying audio data, Proceedings of the 2005 workshops on Genetic and evolutionary computation, June 25-26, 2005, Washington, D.C.
	Jun Xu , Yunbo Cao , Hang Li , Min Zhao, Ranking definitions with supervised learning methods, Special interest tracks and posters of the 14th international conference on World Wide Web, May 10-14, 2005, Chiba, Japan
	Qiankun Zhao , Steven C. H. Hoi , Tie-Yan Liu , Sourav S. Bhowmick , Michael R. Lyu , Wei-Ying Ma, Time-dependent semantic similarity measure of queries using historical click-through data, Proceedings of the 15th international conference on World Wide Web, May 23-26, 2006, Edinburgh, Scotland
	Eugene Agichtein , Eric Brill , Susan Dumais , Robert Ragno, Learning user interaction models for predicting web search result preferences, Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, August 06-11, 2006, Seattle, Washington, USA
	Dong Xin , Xuehua Shen , Qiaozhu Mei , Jiawei Han, Discovering interesting patterns through user's interactive feedback, Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, August 20-23, 2006, Philadelphia, PA, USA
	Kyung-Soon Lee , Kyo Kageura, Virtual relevant documents in text categorization with support vector machines, Information Processing and Management: an International Journal, v.43 n.4, p.902-913, July, 2007
	Namhee Kwon , Eduard Hovy, Information acquisition using multiple classifications, Proceedings of the 4th international conference on Knowledge capture, October 28-31, 2007, Whistler, BC, Canada
	Melinda T. Gervasio , Michael D. Moffitt , Martha E. Pollack , Joseph M. Taylor , Tomas E. Uribe, Active preference learning for personalized calendar scheduling assistance, Proceedings of the 10th international conference on Intelligent user interfaces, January 10-13, 2005, San Diego, California, USA
	Xuehua Shen , Bin Tan , ChengXiang Zhai, Implicit user modeling for personalized search, Proceedings of the 14th ACM international conference on Information and knowledge management, October 31-November 05, 2005, Bremen, Germany
	Hong Wu , Hanqing Lu , Songde Ma, A practical SVM-based algorithm for ordinal regression in image retrieval, Proceedings of the eleventh ACM international conference on Multimedia, November 02-08, 2003, Berkeley, CA, USA
	Lei Zhang , Le Chen , Feng Jing , Kefeng Deng , Wei-Ying Ma, EnjoyPhoto: a vertical image search engine for enjoying high-quality photos, Proceedings of the 14th annual ACM international conference on Multimedia, October 23-27, 2006, Santa Barbara, CA, USA
	Hwanjo Yu , Seung-won Hwang , Kevin Chen-Chuan Chang, Enabling soft queries for data retrieval, Information Systems, v.32 n.4, p.560-574, June, 2007
	Xuanhui Wang , ChengXiang Zhai, Learn from web search logs to organize search results, Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, July 23-27, 2007, Amsterdam, The Netherlands
	Vanessa Murdock , Massimiliano Ciaramita , Vassilis Plachouras, A noisy-channel approach to contextual advertising, Proceedings of the 1st international workshop on Data mining and audience intelligence for advertising, p.21-27, August 12-12, 2007, San Jose, California
	Ingo Mierswa , Katharina Morik, Automatic Feature Extraction for Classifying Audio Data, Machine Learning, v.58 n.2-3, p.127-149, February 2005
	Sandeep Pandey , Sourashis Roy , Christopher Olston , Junghoo Cho , Soumen Chakrabarti, Shuffling a stacked deck: the case for partially randomized ranking of search engine results, Proceedings of the 31st international conference on Very large data bases, August 30-September 02, 2005, Trondheim, Norway
	Gui-Rong Xue , Hua-Jun Zeng , Zheng Chen , Yong Yu , Wei-Ying Ma , WenSi Xi , WeiGuo Fan, Optimizing web search using web click-through data, Proceedings of the thirteenth ACM international conference on Information and knowledge management, November 08-13, 2004, Washington, D.C., USA
	Hwanjo Yu, SVM selective sampling for ranking with application to data retrieval, Proceeding of the eleventh ACM SIGKDD international conference on Knowledge discovery in data mining, August 21-24, 2005, Chicago, Illinois, USA
	Xuehua Shen , Bin Tan , ChengXiang Zhai, Context-sensitive information retrieval using implicit feedback, Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, August 15-19, 2005, Salvador, Brazil
	Eugene Agichtein , Zijian Zheng, Identifying "best bet" web search results by mining past user behavior, Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, August 20-23, 2006, Philadelphia, PA, USA
	Toshihiro Kamishima , Shotaro Akaho, Supervised ordering by regression combined with Thurstone's model, Artificial Intelligence Review, v.25 n.3, p.231-246, May 2006
	Vincent Ng, Machine learning for coreference resolution: from local classification to global ranking, Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics, p.157-164, June 25-30, 2005, Ann Arbor, Michigan
	Yunbo Cao , Jun Xu , Tie-Yan Liu , Hang Li , Yalou Huang , Hsiao-Wuen Hon, Adapting ranking SVM to document retrieval, Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, August 06-11, 2006, Seattle, Washington, USA
	Eugene Agichtein , Eric Brill , Susan Dumais, Improving web search ranking by incorporating user behavior information, Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, August 06-11, 2006, Seattle, Washington, USA
	Tao Qin , Xu-Dong Zhang , Tie-Yan Liu , De-Sheng Wang , Wei-Ying Ma , Hong-Jiang Zhang, An active feedback framework for image retrieval, Pattern Recognition Letters, v.29 n.5, p.637-646, April, 2008
	Donald Metzler , W. Bruce Croft, Linear feature-based models for information retrieval, Information Retrieval, v.10 n.3, p.257-274, June 2007
	Michael Taylor , John Guiver , Stephen Robertson , Tom Minka, SoftRank: optimizing non-smooth rank metrics, Proceedings of the international conference on Web search and web data mining, February 11-12, 2008, Palo Alto, California, USA
	Hideki Isozaki , Hideto Kazawa , Tsutomu Hirao, A deterministic word dependency analyzer enhanced with preference learning, Proceedings of the 20th international conference on Computational Linguistics, p.275-es, August 23-27, 2004, Geneva, Switzerland
	Lori Lorigo , Bing Pan , Helene Hembrooke , Thorsten Joachims , Laura Granka , Geri Gay, The influence of task and gender on search and evaluation behavior using Google, Information Processing and Management: an International Journal, v.42 n.4, p.1123-1131, July 2006
	Thorsten Joachims, Training linear SVMs in linear time, Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, August 20-23, 2006, Philadelphia, PA, USA
	Dmitri Roussinov , Weiguo Fan, Discretization based learning approach to information retrieval, Proceedings of the conference on Human Language Technology and Empirical Methods in Natural Language Processing, p.153-160, October 06-08, 2005, Vancouver, British Columbia, Canada
	Ming-Feng Tsai , Tie-Yan Liu , Tao Qin , Hsin-Hsi Chen , Wei-Ying Ma, FRank: a ranking method with fidelity loss, Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, July 23-27, 2007, Amsterdam, The Netherlands
	Wilfred Ng , Lin Deng , Dik Lun Lee, Mining User preference using Spy voting for search engine personalization, ACM Transactions on Internet Technology (TOIT), v.7 n.4, p.19-es, October 2007
	Jiang Bian , Yandong Liu , Eugene Agichtein , Hongyuan Zha, Finding the right facts in the crowd: factoid question answering over social media, Proceeding of the 17th international conference on World Wide Web, April 21-25, 2008, Beijing, China
	Antonio Bahamonde , Gustavo F. Bayón , Jorge Díez , José Ramón Quevedo , Oscar Luaces , Juan José del Coz , Jaime Alonso , Félix Goyache, Feature subset selection for learning preferences: a case study, Proceedings of the twenty-first international conference on Machine learning, p.7, July 04-08, 2004, Banff, Alberta, Canada
	Sandeep Pandey , Christopher Olston, User-centric Web crawling, Proceedings of the 14th international conference on World Wide Web, May 10-14, 2005, Chiba, Japan
	Elad Yom-Tov , Shai Fine , David Carmel , Adam Darlow, Learning to estimate query difficulty: including applications to missing content detection and distributed information retrieval, Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, August 15-19, 2005, Salvador, Brazil
	Yuanqing Li , Cuntai Guan, Joint feature re-extraction and classification using an iterative semi-supervised support vector machine algorithm, Machine Learning, v.71 n.1, p.33-53, April 2008
	Zhaohui Zheng , Keke Chen , Gordon Sun , Hongyuan Zha, A regression framework for learning ranking functions using relative relevance judgments, Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, July 23-27, 2007, Amsterdam, The Netherlands
	Qingqing Gan , Josh Attenberg , Alexander Markowetz , Torsten Suel, Analysis of geographic queries in a search engine log, Proceedings of the first international workshop on Location and the web, p.49-56, April 22-22, 2008, Beijing, China
	Huyen-Trang Vu , Patrick Gallinari, A machine learning based approach to evaluating retrieval systems, Proceedings of the main conference on Human Language Technology Conference of the North American Chapter of the Association of Computational Linguistics, p.399-406, June 04-09, 2006, New York, New York
	Amir Hosein Keyhanipour , Behzad Moshiri , Majid Kazemian , Maryam Piroozmand , Caro Lucas, Aggregation of web search engines based on users' preferences in WebFusion, Knowledge-Based Systems, v.20 n.4, p.321-328, May, 2007
	Ruifang Ge Raymond , J. Mooney, Discriminative reranking for semantic parsing, Proceedings of the COLING/ACL on Main conference poster sessions, p.263-270, July 17-18, 2006, Sydney, Australia
	Zhe Cao , Tao Qin , Tie-Yan Liu , Ming-Feng Tsai , Hang Li, Learning to rank: from pairwise approach to listwise approach, Proceedings of the 24th international conference on Machine learning, p.129-136, June 20-24, 2007, Corvalis, Oregon
	Xiubo Geng , Tie-Yan Liu , Tao Qin , Hang Li, Feature selection for ranking, Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, July 23-27, 2007, Amsterdam, The Netherlands
	Massimiliano Ciaramita , Vanessa Murdock , Vassilis Plachouras, Online learning from click data for sponsored search, Proceeding of the 17th international conference on World Wide Web, April 21-25, 2008, Beijing, China
	Rodrigo B. Almeida , Virgilio A. F. Almeida, A community-aware search engine, Proceedings of the 13th international conference on World Wide Web, May 17-20, 2004, New York, NY, USA
	Thorsten Joachims , Laura Granka , Bing Pan , Helene Hembrooke , Geri Gay, Accurately interpreting clickthrough data as implicit feedback, Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, August 15-19, 2005, Salvador, Brazil
	Jian-Tao Sun , Hua-Jun Zeng , Huan Liu , Yuchang Lu , Zheng Chen, CubeSVD: a novel approach to personalized Web search, Proceedings of the 14th international conference on World Wide Web, May 10-14, 2005, Chiba, Japan
	Ghim-Eng Yap , Ah-Hwee Tan , Hwee-Hwa Pang, Dynamically-optimized context in recommender systems, Proceedings of the 6th international conference on Mobile data management, May 09-13, 2005, Ayia Napa, Cyprus
	Jianfeng Gao , Haoliang Qi , Xinsong Xia , Jian-Yun Nie, Linear discriminant model for information retrieval, Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, August 15-19, 2005, Salvador, Brazil
	Wilfred Ng , Ho Lam Lau, A co-training framework for searching XML documents, Information Systems, v.32 n.3, p.477-503, May, 2007
	Joachim Diederich , Aqeel Al-Ajmi , Peter Yellowlees, Ex-ray: Data mining and mental health, Applied Soft Computing, v.7 n.3, p.923-928, June, 2007
	D. Higgins , J. Burstein , Y. Attali, Identifying off-topic student essays without topic-specific training data, Natural Language Engineering, v.12 n.2, p.145-159, June 2006
	Qiaozhu Mei , Kenneth Church, Entropy of search logs: how hard is search? with personalization? with backoff?, Proceedings of the international conference on Web search and web data mining, February 11-12, 2008, Palo Alto, California, USA
	Filip Radlinski , Thorsten Joachims, Query chains: learning to rank from implicit feedback, Proceeding of the eleventh ACM SIGKDD international conference on Knowledge discovery in data mining, August 21-24, 2005, Chicago, Illinois, USA
	Sriram Venkatapathy , Aravind K. Joshi, Measuring the relative compositionality of verb-noun (V-N) collocations by integrating features, Proceedings of the conference on Human Language Technology and Empirical Methods in Natural Language Processing, p.899-906, October 06-08, 2005, Vancouver, British Columbia, Canada
	Seikyung Jung , Jonathan L. Herlocker , Janet Webster, Click data as implicit relevance feedback in web search, Information Processing and Management: an International Journal, v.43 n.3, p.791-807, May, 2007
	Wei Gao , Cheng Niu , Jian-Yun Nie , Ming Zhou , Jian Hu , Kam-Fai Wong , Hsiao-Wuen Hon, Cross-lingual query suggestion using query logs of different languages, Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, July 23-27, 2007, Amsterdam, The Netherlands
	Alekh Agarwal , Soumen Chakrabarti , Sunny Aggarwal, Learning to rank networked entities, Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, August 20-23, 2006, Philadelphia, PA, USA
	Tao Qin , Xu-Dong Zhang , Ming-Feng Tsai , De-Sheng Wang , Tie-Yan Liu , Hang Li, Query-level loss functions for information retrieval, Information Processing and Management: an International Journal, v.44 n.2, p.838-855, March, 2008
	Alekh Agarwal , Soumen Chakrabarti, Learning random walks to rank nodes in graphs, Proceedings of the 24th international conference on Machine learning, p.9-16, June 20-24, 2007, Corvalis, Oregon
	Shai Shalev-Shwartz , Yoram Singer, Efficient Learning of Label Ranking by Soft Projections onto Polyhedra, The Journal of Machine Learning Research, 7, p.1567-1599, 12/1/2006
	Jun Xu , Hang Li, AdaRank: a boosting algorithm for information retrieval, Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, July 23-27, 2007, Amsterdam, The Netherlands
	Nenad Stojanovic , Jorge Gonzalez , Ljiljana Stojanovic, ONTOLOGER: a system for usage-driven management of ontology-based information portals, Proceedings of the 2nd international conference on Knowledge capture, October 23-25, 2003, Sanibel Island, FL, USA
	Le Chen , Lei Zhang , Feng Jing , Ke-Feng Deng , Wei-Ying Ma, Ranking web objects from multiple communities, Proceedings of the 15th ACM international conference on Information and knowledge management, November 06-11, 2006, Arlington, Virginia, USA
	Mikhail Bilenko , Ryen W. White, Mining the search trails of surfing crowds: identifying relevant websites from user activity, Proceeding of the 17th international conference on World Wide Web, April 21-25, 2008, Beijing, China
	Zhiyuan Chen , Tao Li, Addressing diverse user preferences in SQL-query-result navigation, Proceedings of the 2007 ACM SIGMOD international conference on Management of data, June 11-14, 2007, Beijing, China
	Tao Qin , Tie-Yan Liu , Xu-Dong Zhang , De-Sheng Wang , Wen-Ying Xiong , Hang Li, Learning to rank relational objects and its application to web search, Proceeding of the 17th international conference on World Wide Web, April 21-25, 2008, Beijing, China
	Crystal Nakatsu , Michael White, Learning to say it well: reranking realizations by predicted synthesis quality, Proceedings of the 21st International Conference on Computational Linguistics and the 44th annual meeting of the ACL, p.1113-1120, July 17-18, 2006, Sydney, Australia
	Namhee Kwon , Stuart W. Shulman , Eduard Hovy, Multidimensional text analysis for eRulemaking, Proceedings of the 2006 international conference on Digital government research, May 21-24, 2006, San Diego, California
	Soumen Chakrabarti , Kriti Puniyani , Sujatha Das, Optimizing scoring functions and indexes for proximity search in type-annotated corpora, Proceedings of the 15th international conference on World Wide Web, May 23-26, 2006, Edinburgh, Scotland
	Pavel A. Dmitriev , Nadav Eiron , Marcus Fontoura , Eugene Shekita, Using annotations in enterprise search, Proceedings of the 15th international conference on World Wide Web, May 23-26, 2006, Edinburgh, Scotland
	Qiaozhu Mei , Dong Xin , Hong Cheng , Jiawei Han , Chengxiang Zhai, Semantic annotation of frequent patterns, ACM Transactions on Knowledge Discovery from Data (TKDD), v.1 n.3, p.11-es, December 2007
	Matthew Richardson , Amit Prakash , Eric Brill, Beyond PageRank: machine learning for static ranking, Proceedings of the 15th international conference on World Wide Web, May 23-26, 2006, Edinburgh, Scotland
	Hongyuan Zha , Zhaohui Zheng , Haoying Fu , Gordon Sun, Incorporating query difference for learning retrieval functions in world wide web search, Proceedings of the 15th ACM international conference on Information and knowledge management, November 06-11, 2006, Arlington, Virginia, USA
	Filip Radlinski , Thorsten Joachims, Active exploration for learning rankings from clickthrough data, Proceedings of the 13th ACM SIGKDD international conference on Knowledge discovery and data mining, August 12-15, 2007, San Jose, California, USA
	Ronen I. Brafman , Carmel Domshlak, Graphically structured value-function compilation, Artificial Intelligence, v.172 n.2-3, p.325-349, February, 2008
	Changhu Wang , Feng Jing , Lei Zhang , Hong-Jiang Zhang, Learning query-biased web page summarization, Proceedings of the sixteenth ACM conference on Conference on information and knowledge management, November 06-10, 2007, Lisbon, Portugal
	Lida Li , Michael J. Muller , Werner Geyer , Casey Dugan , Beth Brownholtz , David R. Millen, Predicting individual priorities of shared activities using support vector machines, Proceedings of the sixteenth ACM conference on Conference on information and knowledge management, November 06-10, 2007, Lisbon, Portugal
	Jun Xu , Yun-Bo Cao , Hang Li , Min Zhao , Ya-Lou Huang, A supervised learning approach to search of definitions, Journal of Computer Science and Technology, v.21 n.3, p.439-449, May 2006
	Jonathan L. Elsas , Vitor R. Carvalho , Jaime G. Carbonell, Fast learning of document ranking functions with the committee perceptron, Proceedings of the international conference on Web search and web data mining, February 11-12, 2008, Palo Alto, California, USA
	Jorge Díez , Juan José del Coz , Oscar Luaces , Antonio Bahamonde, Clustering people according to their preference criteria, Expert Systems with Applications: An International Journal, v.34 n.2, p.1274-1284, February, 2008
	Harris Wu , Michael Gordon , Kurtis DeMaagd , Weiguo Fan, Mining web navigations for intelligence, Decision Support Systems, v.41 n.3, p.574-591, March 2006
	Tao Qin , Xu-Dong Zhang , De-Sheng Wang , Tie-Yan Liu , Wei Lai , Hang Li, Ranking with multiple hyperplanes, Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, July 23-27, 2007, Amsterdam, The Netherlands
	Massimo Melucci , Ryen W. White, Utilizing a geometry of context for enhanced implicit feedback, Proceedings of the sixteenth ACM conference on Conference on information and knowledge management, November 06-10, 2007, Lisbon, Portugal
	Baoning Wu , Brian D. Davison, Detecting semantic cloaking on the web, Proceedings of the 15th international conference on World Wide Web, May 23-26, 2006, Edinburgh, Scotland
	Thorsten Joachims , Laura Granka , Bing Pan , Helene Hembrooke , Filip Radlinski , Geri Gay, Evaluating the accuracy of implicit feedback from clicks and query reformulations in Web search, ACM Transactions on Information Systems (TOIS), v.25 n.2, p.7-es, April 2007
	Regina Barzilay , Mirella Lapata, Modeling local coherence: An entity-based approach, Computational Linguistics, v.34 n.1, p.1-34, March 2008
	Shenghua Bao , Guirong Xue , Xiaoyuan Wu , Yong Yu , Ben Fei , Zhong Su, Optimizing web search using social annotations, Proceedings of the 16th international conference on World Wide Web, May 08-12, 2007, Banff, Alberta, Canada
	Kalervo Järvelin, An analysis of two approaches in information retrieval: From frameworks to study designs, Journal of the American Society for Information Science and Technology, v.58 n.7, p.971-986, May 2007
	Jaroslaw Baliński , Czeslaw Daniłowicz, Re-ranking method based on inter-document distances, Information Processing and Management: an International Journal, v.41 n.4, p.759-775, July 2005
	Sébastien Gadat , Laurent Younes, A Stochastic Algorithm for Feature Selection in Pattern Recognition, The Journal of Machine Learning Research, 8, p.509-547, 5/1/2007
	Yiping Ke , Lin Deng , Wilfred Ng , Dik-Lun Lee, Web dynamics and their ramifications for the development of web search engines, Computer Networks: The International Journal of Computer and Telecommunications Networking, v.50 n.10, p.1430-1447, 14 July 2006