skip to main content
10.1145/1008992.1009058acmconferencesArticle/Chapter ViewAbstractPublication PagesirConference Proceedingsconference-collections
Article

Human versus machine in the topic distillation task

Published: 25 July 2004 Publication History

Abstract

This paper reports on and discusses a set of user experiments using the TREC 2003 Web interactive track protocol. The focus is on comparing humans and machine algorithms in terms of performance in a topic distillation task. We also investigated the effect of the search results layout in supporting the users' effort.We have demonstrated that machines can perform nearly as well as people on the topic distillation task. Given a system tailored to the task there is significant performance improvement and finally, given a presentation that supports the task, there is strong user satisfaction.

References

[1]
Belkin, N. J., Oddy R. N., and Brooks, H. M., ASK for information retrieval: Part I. Background and theory. Journal of Documentation, 38(2):61--71, 1982.
[2]
Belkin, N. J., Kelly, D., Lee, H. J., Li, Y. L., Muresan, G. etc. Rutgers' HARD and Web Interactive Track Experiments. In TREC12 Proceedings, 2003.
[3]
Belkin, N. J., Marchetti, P and Cool, C. BRAQUE: Design of an interface to support user interaction in information retrieval. Information Processing & Management, Vol. 29 (3), pp.325--344, 1993.
[4]
Borlund, P. Experimental components for the evaluation of interactive information retrieval systems. Journal of Documentation 56(1), pp. 71--90, 2000.
[5]
Bharat, K. and Henzinger, M. Improved algorithms for topic distillation in hyperlinked environments. In Proceedings of the 21st Annual ACM SIGIR Conference on Research and Development in Information Retrieval. pp.104--111, 1998.
[6]
Brajnik, G., Mizzaro, S. and Tasso, C. Evaluating user interfaces to information retrieval systems: a case study on user support. In 19th International Conference on Research and Developments in Information Retrieval (SIGIR'96). Zurich, Switzerland, August 1996, pp. 128--136.
[7]
Chakrabarti, S., Dom, B., Gibson, D., Kleinberg, J., Raghavan, P. and Rajagopalan, S., Automatic resource compilation by analyzing hyperlink structure and associated text. Proceedings of the 7th World-Wide Web conference, 1998.
[8]
Chakrabarti, S., Joshi, M. and Tawde, V., Enhanced topic distillation using text, markup tags, and hyperlinks. In Proceedings of SIGIR'01, Sept. 9-12, 2001, New Orleans, Louisiana, USA.
[9]
Craswell, N. & Hawking, D. Overview of the TREC 2002 Web track. In E. M. Voorhees & D. M. Harman (Eds.). The 2002 Text Retrieval Conference. Washington, D. C.
[10]
Craswell, N., Hawking, D., Thoms, J., Trystan, U., Wilkinson, R. and Wu, M. TREC11 Web and interactive tracks at CSIRO. The 2002 Text Retrieval Conference (TREC2002). Washington, D. C.
[11]
Craswell, N., Hawking, D., Wilkinson, R. and Wu, M. TREC12 Web track overview. In TREC12 Proceedings, 2004.
[12]
Craswell, N., Hawking, D., Wilkinson, R. and Wu, M. Task Descriptions: Web Track 2003. In TREC12 Proceedings, 2004.
[13]
Craswell, N., Hawking, D., Upstill, T., McLean, A., Wilkinson, R. and Wu, M. TREC12 Web track and interactive tracks at CSIRO. In TREC12 Proceedings, 2004.
[14]
Dumais, S. T., Cutrell, E. and Chen, H. Bring order to the Web: Optimizing search by showing results in context. In Proceedings of CHI'01, Human Factors in Computing Systems, pp. 277--283, April 2001.
[15]
Hawking, D., Craswell, N., and Thistlewaite, P. ACSys TREC-7 Experiments. In Proceedings of Seventh Text Retrieval Conference. November 1998. Gaithersburg, USA.
[16]
Hearst, M. and Pedersen, J. O. Reexamining the cluster hypothesis: Scatter/Gather on retrieval results. In Proceeding of the 19th Annual ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 76--84, 1996.
[17]
Hearst, M. and Karadi, C. Cat-a-Cone: An interactive interface of specifying searches and viewing retrieval results using a large category hierarchy. In Proceeding of the 20th Annual ACM SIGIR Conference on Research and Development in Information Retrieval, 1997.
[18]
Hersh, W., Turpin, A., Price, S., Kraemer, D., Chan, B., Sacherek, L., Olson, D., Do batch and user evaluations give the same results? In Proceedings of the 23rd Annual ACM SIGIR Conference on Research and Development in Information Retrieval, pp.17--24, 2000.
[19]
Kleinberg. J., Authoritative sources in a hyperlinked environment. Proc. ACM-SIAM Symposium on Discrete Algorithms, 1998.
[20]
Olsen, K. A., Korfhage, R. R., Sochats, K. M., Spring, M. B. and Williams, J. G. Visualization of a document collection: the VIBE systems. Information Processing and Management, vol.(29), No.1, pp.69--81, 1993.
[21]
Veerasamy, A. and Belkin, N. J. Evaluation of a tool for visualization of information retrieval results. In Proceeding of the 19th Annual ACM SIGIR Conference on Research and Development in Information Retrieval, pp.85--92, 1997.
[22]
Wu, M., Fuller, M. and Wilkinson, R. Using clustering and classification approaches in interactive retrieval. Information Processing and Management, vol. 37, pp.459--484, 2001.
[23]
Wu, M., Fuller, M. and Wilkinson, R. Searcher performance in question answering. In Proceedings of 24th Annual ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 375--381, 2001.

Cited By

View all
  • (2016)A study of the use of simulated work task situations in interactive information retrieval evaluationsJournal of Documentation10.1108/JD-06-2015-006872:3(394-413)Online publication date: 9-May-2016
  • (2012)Human question answering performance using an interactive document retrieval systemProceedings of the 4th Information Interaction in Context Symposium10.1145/2362724.2362735(35-44)Online publication date: 21-Aug-2012
  • (2010)Presenting query aspects to support exploratory searchProceedings of the Eleventh Australasian Conference on User Interface - Volume 10610.5555/1862280.1862286(23-32)Online publication date: 1-Jan-2010
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
SIGIR '04: Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval
July 2004
624 pages
ISBN:1581138814
DOI:10.1145/1008992
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 25 July 2004

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. evaluation
  2. interactive information retrieval
  3. search result organization
  4. topic distillation

Qualifiers

  • Article

Conference

SIGIR04
Sponsor:

Acceptance Rates

Overall Acceptance Rate 792 of 3,983 submissions, 20%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)3
  • Downloads (Last 6 weeks)1
Reflects downloads up to 18 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2016)A study of the use of simulated work task situations in interactive information retrieval evaluationsJournal of Documentation10.1108/JD-06-2015-006872:3(394-413)Online publication date: 9-May-2016
  • (2012)Human question answering performance using an interactive document retrieval systemProceedings of the 4th Information Interaction in Context Symposium10.1145/2362724.2362735(35-44)Online publication date: 21-Aug-2012
  • (2010)Presenting query aspects to support exploratory searchProceedings of the Eleventh Australasian Conference on User Interface - Volume 10610.5555/1862280.1862286(23-32)Online publication date: 1-Jan-2010
  • (2010)Reconsideration of the simulated work task situationProceedings of the third symposium on Information interaction in context10.1145/1840784.1840808(155-164)Online publication date: 18-Aug-2010
  • (2009)Interactive information retrievalAnnual Review of Information Science and Technology10.1002/aris.2008.144042010942:1(43-91)Online publication date: 5-Nov-2009

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media