Testing algorithms is like testing students

ABSTRACT

In this paper, we apply methods from educational testing to measure the reliability of an IR collection.

REFERENCES

• 1: Linda Crocker and James Algina. Introduction to Classical & Modern Test Theory. Holt, Rinehart and Winston, 1986.
• 2: R. Robert L. Brennan. Generalizability Theory. Springer-Verlag, 2001.
• 3: Richard J. Shavelson, Noreen M. Webb. Generalizability Theory: A Primer. Sage Publications, 1991.
• 4: Ellen M. Voorhees, Variations in relevance judgments and the measurement of retrieval effectiveness, Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, p.315-323, August 24-28, 1998, Melbourne, Australia [doi>10.1145/290941.291017]
• 5: Ellen M. Voorhees , Chris Buckley, The effect of topic set size on retrieval experiment error, Proceedings of the 25th annual international ACM SIGIR conference on Research and development in information retrieval, August 11-15, 2002, Tampere, Finland [doi>10.1145/564376.564432]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.4 Systems and Software
          Subjects: Performance evaluation (efficiency and effectiveness)

General Terms:
Experimentation, Measurement, Performance, Standardization

Keywords:
evaluation, generalizability theory, test collection, test theory