Learning a Deep Listwise Context Model for Ranking Refinement

ABSTRACT

Learning to rank has been intensively studied and widely applied in information retrieval. Typically, a global ranking function is learned from a set of labeled data, which can achieve good performance on average but may be suboptimal for individual queries by ignoring the fact that relevant documents for different queries may have different distributions in the feature space. Inspired by the idea of pseudo relevance feedback where top ranked documents, which we refer as the local ranking context, can provide important information about the query's characteristics, we propose to use the inherent feature distributions of the top results to learn a Deep Listwise Context Model that helps us fine tune the initial ranked list. Specifically, we employ a recurrent neural network to sequentially encode the top results using their feature vectors, learn a local context model and use it to re-rank the top results. There are three merits with our model: (1) Our model can capture the local ranking context based on the complex interactions between top results using a deep neural network; (2) Our model can be built upon existing learning-to-rank methods by directly using their extracted feature vectors; (3) Our model is trained with an attention-based loss function, which is more effective and efficient than many existing listwise methods. Experimental results show that the proposed model can significantly improve the state-of-the-art learning to rank methods on benchmark retrieval corpora.

References

1. Qingyao Ai, Keping Bi, Cheng Luo, Jiafeng Guo, and W. Bruce Croft . 2018. Unbiased Learning to Rank with Unbiased Propensity Estimation Proceedings of the 41st ACM SIGIR. ACM.
2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio . 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473 (2014).
3. Chris Burges, Tal Shaked, Erin Renshaw, Ari Lazier, Matt Deeds, Nicole Hamilton, and Greg Hullender . 2005. Learning to rank using gradient descent. In Proceedings of the 22nd ICML. ACM, 89--96.
4. Christopher JC Burges . 2010. From ranknet to lambdarank to lambdamart: An overview. Learning Vol. 11 (2010), 23--581.
5. Ethem F Can, W Bruce Croft, and R Manmatha . 2014. Incorporating query-specific feedback into learning-to-rank models Proceedings of the 37th ACM SIGIR. ACM, 1035--1038.
6. Zhe Cao, Tao Qin, Tie-Yan Liu, Ming-Feng Tsai, and Hang Li . 2007. Learning to rank: from pairwise approach to listwise approach Proceedings of the 24th ICML. ACM, 129--136.
7. Olivier Chapelle and Yi Chang . 2011. Yahoo! Learning to Rank Challenge Overview.. In Yahoo! Learning to Rank Challenge. 1--24.
8. Olivier Chapelle, Donald Metlzer, Ya Zhang, and Pierre Grinspan . 2009. Expected reciprocal rank for graded relevance. In Proceedings of the 18th ACM CIKM. ACM, 621--630.
9. Heng-Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, et almbox. . 2016. Wide & Deep Learning for Recommender Systems. In Proceedings of the 1st Workshop on Deep Learning for Recommender Systems. ACM, 7--10.
10. Kyunghyun Cho, Bart Van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio . 2014 a. On the properties of neural machine translation: Encoder-decoder approaches. arXiv preprint arXiv:1409.1259 (2014).
11. Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio . 2014 b. Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078 (2014).
12. Mostafa Dehghani, Hamed Zamani, Aliaksei Severyn, Jaap Kamps, and W Bruce Croft . 2017. Neural Ranking Models with Weak Supervision. arXiv preprint arXiv:1704.08803 (2017).
13. Fernando Diaz . 2007. Regularizing query-based retrieval scores. Information Retrieval Vol. 10, 6 (2007), 531--562.
14. Yajuan Duan, Long Jiang, Tao Qin, Ming Zhou, and Heung-Yeung Shum . 2010. An empirical study on learning to rank of tweets. In Proceedings of the 23rd International Conference on Computational Linguistics. Association for Computational Linguistics, 295--303.
15. Jerome H Friedman . 2001. Greedy function approximation: a gradient boosting machine. Annals of statistics (2001), 1189--1232.
16. Xiubo Geng, Tie-Yan Liu, Tao Qin, Andrew Arnold, Hang Li, and Heung-Yeung Shum . 2008. Query dependent ranking using k-nearest neighbor. In Proceedings of the 31st ACM SIGIR. ACM, 115--122.
17. Jiafeng Guo, Yixing Fan, Qingyao Ai, and W Bruce Croft . 2016. A deep relevance matching model for ad-hoc retrieval Proceedings of the 25th ACM CIKM. ACM, 55--64.
18. Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry Heck . 2013. Learning deep structured semantic models for web search using clickthrough data Proceedings of the 22nd ACM CIKM. ACM, 2333--2338.
19. Kalervo J"arvelin and Jaana Kek"al"ainen . 2002. Cumulated gain-based evaluation of IR techniques. ACM Transactions on Information Systems Vol. 20, 4 (2002), 422--446.
20. Thorsten Joachims . 2002. Optimizing search engines using clickthrough data. In Proceedings of the eighth ACM SIGKDD. ACM, 133--142.
21. Thorsten Joachims . 2006. Training linear SVMs in linear time. In Proceedings of the 12th ACM SIGKDD. ACM, 217--226.
22. Victor Lavrenko and W Bruce Croft . 2001. Relevance based language models. In Proceedings of the 24th ACM SIGIR. ACM, 120--127.
23. Tie-Yan Liu . 2009. Learning to rank for information retrieval. Foundations and Trends in Information Retrieval Vol. 3, 3 (2009), 225--331.
24. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher . 2016. Pointer Sentinel Mixture Models. arXiv preprint arXiv:1609.07843 (2016).
25. Tao Qin and Tie-Yan Liu . 2013. Introducing LETOR 4.0 Datasets. CoRR Vol. abs/1306.2597 (2013). deftempurl%http://arxiv.org/abs/1306.2597 tempurl
26. Tao Qin, Tie-Yan Liu, Xu-Dong Zhang, De-Sheng Wang, and Hang Li . 2008. Global ranking of documents using continuous conditional random fields. Technical Report. Technical Report MSR-TR-2008--156, Microsoft Corporation.
27. C Quoc and Viet Le . 2007. Learning to rank with nonsmooth cost functions. Advances in Neural Information Processing Systems Vol. 19 (2007), 193--200.
28. Stephen E Robertson and K Sparck Jones . 1976. Relevance weighting of search terms. Journal of American Society for Information science Vol. 27, 3 (1976), 129--146.
29. Gerard Salton and Chris Buckley . 1997. Improving retrieval performance by relevance feedback. Readings in information retrieval Vol. 24, 5 (1997), 355--363.
30. Mike Schuster and Kuldip K Paliwal . 1997. Bidirectional recurrent neural networks. IEEE Transactions on Signal Processing Vol. 45, 11 (1997), 2673--2681.
31. Mark D Smucker, James Allan, and Ben Carterette . 2007. A comparison of statistical significance tests for information retrieval evaluation Proceedings of the sixteenth ACM CIKM. ACM, 623--632.
32. Richard Socher, Danqi Chen, Christopher D Manning, and Andrew Ng . 2013. Reasoning with neural tensor networks for knowledge base completion Advances in Neural Information Processing Systems. 926--934.
33. Ilya Sutskever, Oriol Vinyals, and Quoc V Le . 2014. Sequence to sequence learning with neural networks Advances in neural information processing systems. 3104--3112.
34. Michael Taylor, John Guiver, Stephen Robertson, and Tom Minka . 2008. Softrank: optimizing non-smooth rank metrics. In Proceedings of WSDM'08. ACM, 77--86.
35. Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly . 2015. Pointer networks. In Advances in Neural Information Processing Systems. 2692--2700.
36. Fen Xia, Tie-Yan Liu, Jue Wang, Wensheng Zhang, and Hang Li . 2008. Listwise approach to learning to rank: theory and algorithm Proceedings of the 25th ICML. ACM, 1192--1199.
37. Liu Yang, Qingyao Ai, Damiano Spina, Ruey-Cheng Chen, Liang Pang, W Bruce Croft, Jiafeng Guo, and Falk Scholer . 2016. Beyond Factoid QA: Effective Methods for Non-factoid Answer Sentence Retrieval ECIR. Springer, 115--128.
38. Chengxiang Zhai and John Lafferty . 2001. Model-based feedback in the language modeling approach to information retrieval Proceedings of the 10th ACM CIKM. ACM, 403--410.
39. Chengxiang Zhai and John Lafferty . 2004. A study of smoothing methods for language models applied to information retrieval. ACM Transactions on Information Systems Vol. 22, 2 (2004), 179--214.

Index Terms

• Information systems

  • Information retrieval

    • Retrieval models and ranking

      • Learning to rank