Learning deep structured semantic models for web search using clickthrough data

ABSTRACT

Latent semantic models, such as LSA, intend to map a query to its relevant documents at the semantic level where keyword-based matching often fails. In this study we strive to develop a series of new latent semantic models with a deep structure that project queries and documents into a common low-dimensional space where the relevance of a document given a query is readily computed as the distance between them. The proposed deep structured semantic models are discriminatively trained by maximizing the conditional likelihood of the clicked documents given a query using the clickthrough data. To make our models applicable to large-scale Web search applications, we also use a technique called word hashing, which is shown to effectively scale up our semantic models to handle large vocabularies which are common in such tasks. The new models are evaluated on a Web document ranking task using a real-world data set. Results show that our best model significantly outperforms other latent semantic models, which were considered state-of-the-art in the performance prior to the work presented in this paper.

References

1. Bengio, Y., 2009. "Learning deep architectures for AI." Foundumental Trends Machine Learning, vol. 2.
2. Blei, D. M., Ng, A. Y., and Jordan, M. J. 2003. "Latent Dirichlet allocation." In JMLR, vol. 3.
3. Burges, C., Shaked, T., Renshaw, E., Lazier, A., Deeds, M., Hamilton, and Hullender, G. 2005. "Learning to rank using gradient descent." In ICML.
4. Collobert, R., Weston, J., Bottou, L., Karlen, M., Kavukcuoglu, K., and Kuksa, P., 2011. "Natural language processing (almost) from scratch." in JMLR, vol. 12.
5. Dahl, G., Yu, D., Deng, L., and Acero, A., 2012. "Context-dependent pre-trained deep neural networks for large vocabulary speech recognition." in IEEE Transactions on Audio, Speech, and Language Processing.
6. Deerwester, S., Dumais, S. T., Furnas, G. W., Landauer, T., and Harshman, R. 1990. "Indexing by latent semantic analysis." J. American Society for Information Science, 41(6): 391--407
7. Deng, L., He, X., and Gao, J., 2013. "Deep stacking networks for information retrieval." In ICASSP
8. Dumais, S. T., Letsche, T. A., Littman, M. L., and Landauer, T. K. 1997. "Automatic cross-linguistic information retrieval using latent semantic indexing." In AAAI-97 Spring Sympo-sium Series: Cross-Language Text and Speech Retrieval.
9. Gao, J., He, X., and Nie, J-Y. 2010. "Clickthrough-based translation models for web search: from word models to phrase models." In CIKM.
10. Gao, J., Toutanova, K., Yih., W-T. 2011. "Clickthrough-based latent semantic models for web search." In SIGIR.
11. Gao, J., Yuan, W., Li, X., Deng, K., and Nie, J-Y. 2009. "Smoothing clickthrough data for web search ranking." In SIGIR.
12. He, X., Deng, L., and Chou, W., 2008. "Discriminative learning in sequential pattern recognition," Sept. IEEE Sig. Proc. Mag.
13. Heck, L., Konig, Y., Sonmez, M. K., and Weintraub, M. 2000. "Robustness to telephone handset distortion in speaker recognition by discriminative feature design." In Speech Communication.
14. Hinton, G., Deng, L., Yu, D., Dahl, G., Mohamed, A., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T., and Kingsbury, B., 2012. "Deep neural networks for acoustic modeling in speech recognition," IEEE Sig. Proc. Mag.
15. Hofmann, T. 1999. "Probabilistic latent semantic indexing." In SIGIR.
16. Hutchinson, B., Deng, L., and Yu, D., 2013. "Tensor deep stacking networks." In IEEE T-PAMI, vol. 35.
17. Jarvelin, K. and Kekalainen, J. 2000. "IR evaluation methods for retrieving highly relevant documents." In SIGIR.
18. Konig, Y., Heck, L., Weintraub, M., and Sonmez, M. K. 1998. "Nonlinear discriminant feature extraction for robust text-independent speaker recognition." in RLA2C.
19. Mesnil, G., He, X., Deng, L., and Bengio, Y., 2013. "Investigation of recurrent-neural-network architectures and learning methods for spoken language understanding." In Interspeech.
20. Montavon, G., Orr, G., Müller, K., 2012. Neural Networks: Tricks of the Trade (Second edition). Springer.
21. Platt, J., Toutanova, K., and Yih, W. 2010. "Translingual doc-ument representations from discriminative projections." In EMNLP.
22. Salakhutdinov R., and Hinton, G., 2007 "Semantic hashing." in Proc. SIGIR Workshop Information Retrieval and Applications of Graphical Models.
23. Socher, R., Huval, B., Manning, C., Ng, A., 2012. "Semantic compositionality through recursive matrix-vector spaces." In EMNLP.
24. Svore, K., and Burges, C. 2009. "A machine learning approach for improved BM25 retrieval." In CIKM.
25. Tur, G., Deng, L., Hakkani-Tur, D., and He, X., 2012. "Towards deeper understanding deep convex networks for semantic utterance classification." In ICASSP.
26. Yih, W., Toutanova, K., Platt, J., and Meek, C. 2011. "Learning discriminative projections for text similarity measures." In CoNLL.

Index Terms

• Information systems

  • Information retrieval