The Importance of Prior Probabilities for Entry Page Search

ABSTRACT

An important class of searches on the world-wide-web has the goal to find an entry page (homepage) of an organisation. Entry page search is quite different from Ad Hoc search. Indeed a plain Ad Hoc system performs disappointingly. We explored three non-content features of web pages: page length, number of incoming links and URL form. Especially the URL form proved to be a good predictor. Using URL form priors we found over 70% of all entry pages at rank 1, and up to 89% in the top 10. Non-content features can easily be embedded in a language model framework as a prior probability.

REFERENCES

• 1: Brian Amento , Loren Terveen , Will Hill, Does “authority” mean quality? predicting expert quality ratings of Web documents, Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval, p.296-303, July 24-28, 2000, Athens, Greece [doi>10.1145/345508.345603]
• 2: Peter Bailey , Nick Craswell , David Hawking, Engineering a multi-purpose test collection for web retrieval experiments, Information Processing and Management: an International Journal, v.39 n.6, p.853-871, November 2003 [doi>10.1016/S0306-4573(02)00084-5]
• 3: Krishna Bharat , Monika R. Henzinger, Improved algorithms for topic distillation in a hyperlinked environment, Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, p.104-111, August 24-28, 1998, Melbourne, Australia [doi>10.1145/290941.290972]
• 4: Sergey Brin , Lawrence Page, The anatomy of a large-scale hypertextual Web search engine, Computer Networks and ISDN Systems, v.30 n.1-7, p.107-117, April 1, 1998
• 5: David Cohn , Huan Chang, Learning to Probabilistically Identify Authoritative Documents, Proceedings of the Seventeenth International Conference on Machine Learning, p.167-174, June 29-July 02, 2000
• 6: Nick Craswell , David Hawking , Stephen Robertson, Effective site finding using link anchor information, Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval, p.250-257, September 2001, New Orleans, Louisiana, United States [doi>10.1145/383952.383999]
• 7: N. Craswell, D. Hawking, R. Wilkinson, and M. Wu. TREC10 web and interactive tracks at CSIRO. In Voorhees and Harman TREC10, pages 261--268.
• 8: F. Crivellari and M. Melucci. Web document retrieval using passage retrieval, connectivity information, and automatic link weighting - TREC-9 report. In Voorhees and Harman, pages 611--620.
• 9: W. B. Croft, D. J. Harper, D. H. Kraft, and J. Zobel, editors. Proceedings of the 23rd Annual International Conference on Research and Development in Information Retrieval, 2000.
• 10: Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, August 1998
• 11: Brian D. Davison, Topical locality in the Web, Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval, p.272-279, July 24-28, 2000, Athens, Greece [doi>10.1145/345508.345597]
• 12: C. Gurrin and A. F. Smeaton. Dublin city university experiments in connectivity analysis for TREC-9. In Voorhees and Harman TREC9, pages 179--188.
• 13: D. Hawking. Overview of the TREC-9 web track. In Voorhees and Harman TREC9, pages 87--102.
• 14: D. Hawking and N. Craswell. Overview of the TREC-2001 web track. In Voorhees and Harman TREC10, pages 25--31.
• 15: D. Hawking, E. voorhees, N. Craswell, and P. Bailey. Overview of the TREC-8 web track. In Voorhees and Harman TREC8, pages 131--148.
• 16: D. Hiemstra. Using language models for information retrieval. PhD thesis, Centre for Telematics and Information Technology, University of Twente, 2001.
• 17: D. Hiemstra and W. Kraaij. Twenty-One at TREC-7: Ad-hoc and cross-language track. In Voorhees and Harman TREC7, pages 227--238.
• 18: Jon M. Kleinberg, Authoritative sources in a hyperlinked environment, Journal of the ACM (JACM), v.46 n.5, p.604-632, Sept. 1999 [doi>10.1145/324133.324140]
• 19: W. Kraaij, M. Spitters, and M. van der Heijden. Combining a mixture language model and naive bayes for multi-document summarisation. In Working notes of the DUC2001 workshop (SIGIR2001), New Orleans, 2001.
• 20: W. Kraaij and T. Westerveld. TNO/UT at TREC-9: How different are web documents? In Voorhees and Harman TREC9, pages 665--671.
• 21: J. Lafferty and C. Zhai. Probabilistic IR models based on document and query generation. In J. Callan, B. Croft, and J. Lafferty, editors, Proceedings of the workshop on Language Modeling and Informati on Retrieval, 2001.
• 22: David D. Lewis, Naive (Bayes) at Forty: The Independence Assumption in Information Retrieval, Proceedings of the 10th European Conference on Machine Learning, p.4-15, April 21-23, 1998
• 23: D. R. H. Miller, T. Leek, and R. M. Schwartz. BBN at TREC-7: using hidden markov models for information retrieval. In Voorhees and Harman TREC7, pages 133--142.
• 24: Thomas M. Mitchell, Machine Learning, McGraw-Hill Higher Education, 1997
• 25: Andrew Y. Ng , Alice X. Zheng , Michael I. Jordan, Stable algorithms for link analysis, Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval, p.258-266, September 2001, New Orleans, Louisiana, United States [doi>10.1145/383952.384003]
• 26: Jay M. Ponte , W. Bruce Croft, A language modeling approach to information retrieval, Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, p.275-281, August 24-28, 1998, Melbourne, Australia [doi>10.1145/290941.291008]
• 27: D.-Y. Ra, E.-K. Park, and J.-S. Jang. Yonsei/etri at TREC-10: Utilizing web document properties. In Voorhees and Harman TREC10, pages 643--650.
• 28: S. Robertson and K. Sparck Jones. Relevance weighting of search terms. Journal of the American Society for Information Science, 27(3):129--146, 1976.
• 29: S. E. Robertson , S. Walker, Some simple effective approximations to the 2-Poisson model for probabilistic weighted retrieval, Proceedings of the 17th annual international ACM SIGIR conference on Research and development in information retrieval, p.232-241, July 03-06, 1994, Dublin, Ireland
• 30: Gerard Salton , Christopher Buckley, Term-weighting approaches in automatic text retrieval, Information Processing and Management: an International Journal, v.24 n.5, p.513-523, 1988 [doi>10.1016/0306-4573(88)90021-0 ]
• 31: J. Savoy and Y. Rasolofo. Report on the TREC-9 experiment: Link-based retrieval and distributed collections. In Voorhees and Harman TREC9, pages 579--588.
• 32: J. Savoy and Y. Rasolofo. Report on the TREC-10 experiment: Distributed collections and entrypage searching. In Voorhees and Harman TREC10, pages 578--590.
• 33: Amit Singhal , Chris Buckley , Mandar Mitra, Pivoted document length normalization, Proceedings of the 19th annual international ACM SIGIR conference on Research and development in information retrieval, p.21-29, August 18-22, 1996, Zurich, Switzerland [doi>10.1145/243199.243206]
• 34: E. M. Voorhees and D. K. Harman, editors. The Seventh Text Retrieval Conference (TREC7), volume 7. National Institute of Standards and Technology, NIST, 1999.
• 35: E. M. Voorhees and D. K. Harman, editors. The Eighth Text Retrieval Conference (TREC8), volume 8. National Institute of Standards and Technology, NIST, 2000.
• 36: E. M. Voorhees and D. K. Harman, editors. The Ninth Text Retrieval Conference (TREC9), volume 9. National Institute of Standards and Technology, NIST, 2001.
• 37: E. M. Voorhees and D. K. Harman, editors. The Tenth Text Retrieval Conference (TREC-2001), volume 10. National Institute of Standards and Technology, NIST, 2002.
• 38: T. Westerveld, W. Kraaij, and D. Hiemstra. Retrieving web pages using content, links, URL's and anchors. In Voorhees and Harman TREC10, pages 52--61.
• 39: W. Xi and E. A. Fox. Machine learning approaches for homepage finding tasks at TREC-10. In Voorhees and Harman TREC10, pages 633--642.
• 40: Justin Zobel , Alistair Moffat, Exploring the similarity space, ACM SIGIR Forum, v.32 n.1, p.18-34, Spring 1998 [doi>10.1145/281250.281256]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval

General Terms:
Experimentation

Keywords:
URLs, entry page search, language models, links, parameter estimation, prior probabilities