skip to main content
10.1145/1076034.1076109acmconferencesArticle/Chapter ViewAbstractPublication PagesirConference Proceedingsconference-collections
Article

When will information retrieval be "good enough"?

Published: 15 August 2005 Publication History

Abstract

We describe a user study that examined the relationship between the quality of an Information Retrieval system and the effectiveness of its users in performing a task. The task involves finding answer facets of questions pertaining to a collection of newswire documents over a six month period. We artificially created sets of ranked lists at increasing levels of quality by blending the output of a state-of-the-art retrieval system with truth data created by annotators. Subjects performed the task by using these ranked lists to guide their labeling of answer passages in the retrieved articles. We found that as system accuracy improves, subject time on task and error rate decrease, and the rate of finding new correct answers increases. There is a large intermediary region in which the utility difference is not significant; our results suggest that there is some threshold of accuracy for this task beyond which user utility improves rapidly, but more experiments are needed to examine the area around that threshold closely.

References

[1]
N. Abdul-Jaleel, J. Allan, W. Croft, F. Diaz, L. Larkey, X. Li, D. Metzler, M. Smucker, T. Strohman, H. Turtle, and C. Wade. UMass at TREC 2004: Notebook. In E. Voorhees, editor, The Thirteenth Text Retrieval Conference (TREC 2004) Notebook, pages 657--670, 2004. Available online at http://trec.nist.gov.
[2]
J. Allan. HARD track overview in TREC 2004 (notebook), high accuracy retrieval from documents. In E. Voorhees, editor, The Thirteenth Text Retrieval Conference (TREC 2004) Notebook, pages 226--235, 2004. Available online at http://trec.nist.gov.
[3]
Apache Software Foundation. Jakarta Lucene. Can be found at http://jakarta.apache.org/lucene/.
[4]
D. C. Blair and M. E. Maron. An evaluation of retrieval effectiveness for a full-text document-retrieval system. Communications of the ACM, 28(3):289--299, 1985.
[5]
S. Boyce and A. Gorin. User interface issues for natural spoken dialog systems. In Proceedings of the International Symposium on Spoken Dialogue 96, pages 65--68, 1996.
[6]
C. Buckley. Why current IR engines fail. Proceedings of the 27th annual ACM SIGIR conference, pages 584--585, 2004.
[7]
C. Buckley and E. Voorhees. Retrieval evaluation with incomplete information. Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval, pages 25--32, 2004.
[8]
J. P. Callan. Passage-level evidence in document retrieval. Proceedings of the 17th annual international ACM SIGIR conference on Research and development in information retrieval, pages 302--310, 1994.
[9]
Z. Chen and Y. Xu. User-oriented relevance judgment: A conceptual model. In Proceedings of the 38th Hawaii International Conference on System Sciences, page 101, 2005. Abstract is in proceedings; full paper in accompanying CD-ROM.
[10]
N. Dahlback, A. Jonsson, and L. Ahrenberg. Wizard of Oz studies: why and how. In Proceedings of the 1st international conference on Intelligent user interfaces, pages 193--200, 1993.
[11]
O. Drori and N. Alon. Using documents classification for displaying search results list. Technical Report 2002-46, Leibniz Center for Research in Computer Science, Hebrew University, Jerusalem, Israel, 2002.
[12]
M. Kaszkiel and J. Zobel. Passage retrieval revisited. Proceedings of the 20th annual international ACM SIGIR conference on Research and development in information retrieval, pages 178--185, 1997.
[13]
A. Kehler, J. Martin, A. Cheyer, L. Julia, J. Hobbs, and J. Bear. On representing salience and reference in multimodal human-computer interaction. In AAAI '98 (Representations for Multi-Model Human Computer Interaction), pages 33--39, 1998.
[14]
J. Koenemann and N. J. Belkin. A case for interaction: A study of interactive information retrieval behavior and effectiveness. In CHI, pages 205--212, 1996.
[15]
P. Over. TREC-6 interactive track report. In E. Voorhees and D. Harman, editors, The Sixth Text Retrieval Conference (TREC 6), pages 73--82, 1998. NIST Special Publication 500--240.
[16]
G. Salton. Automatic text processing: the transformation and analysis and and retrieval of information by computer. Addison-Wesley, 1989.
[17]
I. Soboroff. Overview of the TREC 2004 novelty track. In E. Voorhees, editor, The Thirteenth Text Retrieval Conference (TREC 2004) Notebook, pages 57--70, 2004.
[18]
R. Swan and J. Allan. Improving interactive information retrieval effectiveness with 3-d graphics. Technical Report IR-100, University of Massachusetts, Amherst, Center for Intelligent Information Retrieval, 1996.
[19]
E. Tanin, A. Lotem, I. Haddadin, B. Shneiderman, C. Plaisant, and L. Slaughter. Facilitating network data exploration with query previews: A study of user performance and preference. Technical Report CS-TR-3879, University of Maryland, College Park, 1998.
[20]
E. Voorhees. Overview of the TREC 2003 question answering track. In E. Voorhees, editor, The Twelfth Text Retrieval Conference (TREC 2003), pages 73--82, 2003. NIST Special Publication 500--255.
[21]
W. J. Weiland and B. Shneiderman. A graphical query interface based on aggregation/generalization hierarchies. Information Systems, 18(4):215--232, 1993.
[22]
M. Wu, M. Fuller, and R. Wilkinson. Searcher performance in question answering. Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval, pages 375--381, 2001.

Cited By

View all
  • (2023)Clustering of Relevant Documents Based on Findability Effort in Information RetrievalInternational Journal of Information Retrieval Research10.4018/IJIRR.31576412:1(1-18)Online publication date: 6-Jan-2023
  • (2021)Do better search engines really equate to better clinical decisions? If not, why not?Journal of the Association for Information Science and Technology10.1002/asi.2439872:2(141-155)Online publication date: 18-Jan-2021
  • (2020)ResearchInformation Retrieval: A Biomedical and Health Perspective10.1007/978-3-030-47686-1_8(337-405)Online publication date: 23-Jul-2020
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
SIGIR '05: Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval
August 2005
708 pages
ISBN:1595930345
DOI:10.1145/1076034
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 15 August 2005

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. information retrieval
  2. passage retrieval
  3. performance evaluation
  4. user study

Qualifiers

  • Article

Conference

SIGIR05
Sponsor:

Acceptance Rates

Overall Acceptance Rate 792 of 3,983 submissions, 20%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)20
  • Downloads (Last 6 weeks)6
Reflects downloads up to 18 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2023)Clustering of Relevant Documents Based on Findability Effort in Information RetrievalInternational Journal of Information Retrieval Research10.4018/IJIRR.31576412:1(1-18)Online publication date: 6-Jan-2023
  • (2021)Do better search engines really equate to better clinical decisions? If not, why not?Journal of the Association for Information Science and Technology10.1002/asi.2439872:2(141-155)Online publication date: 18-Jan-2021
  • (2020)ResearchInformation Retrieval: A Biomedical and Health Perspective10.1007/978-3-030-47686-1_8(337-405)Online publication date: 23-Jul-2020
  • (2020)On the Pluses and Minuses of RiskInformation Retrieval Technology10.1007/978-3-030-42835-8_8(81-93)Online publication date: 27-Feb-2020
  • (2019)Impact of a Search Engine on Clinical Decisions Under Time and System Effectiveness Constraints: Research ProtocolJMIR Research Protocols10.2196/128038:5(e12803)Online publication date: 28-May-2019
  • (2017)Online In-Situ Interleaved Evaluation of Real-Time Push Notification SystemsProceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3077136.3080808(415-424)Online publication date: 7-Aug-2017
  • (2017)BullseyeProceedings of the Australasian Computer Science Week Multiconference10.1145/3014812.3014846(1-4)Online publication date: 30-Jan-2017
  • (2016)On Obtaining Effort Based Judgements for Information RetrievalProceedings of the Ninth ACM International Conference on Web Search and Data Mining10.1145/2835776.2835840(277-286)Online publication date: 8-Feb-2016
  • (2015)Assessor Differences and User Preferences in Tweet Timeline GenerationProceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/2766462.2767699(615-624)Online publication date: 9-Aug-2015
  • (2015)How Do Gain and Discount Functions Affect the Correlation between DCG and User Satisfaction?Advances in Information Retrieval10.1007/978-3-319-16354-3_20(197-202)Online publication date: 2015
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media