Mining for proposal reviewers: lessons learned at the national science foundation

ABSTRACT

In this paper, we discuss a prototype application deployed at the U.S. National Science Foundation for assisting program directors in identifying reviewers for proposals. The application helps program directors sort proposals into panels and find reviewers for proposals. To accomplish these tasks, it extracts information from the full text of proposals both to learn about the topics of proposals and the expertise of reviewers. We discuss a variety of alternatives that were explored, the solution that was implemented, and the experience in using the solution within the workflow of NSF.

REFERENCES

• 1: Peter Willett, Recent trends in hierarchic document clustering: a critical review, Information Processing and Management: an International Journal, v.24 n.5, p.577-597, 1988 [doi>10.1016/0306-4573(88)90027-1 ]
• 2: Bjornar Larsen , Chinatsu Aone, Fast and effective text mining using linear-time document clustering, Proceedings of the fifth ACM SIGKDD international conference on Knowledge discovery and data mining, p.16-22, August 15-18, 1999, San Diego, California, United States [doi>10.1145/312129.312186]
• 3: Hopcroft, J., Khan, O., Kulis, B. & Selman, B. Tracking evolving communities in large linked networks. Proc. Natl Acad. Sci. USA 101(Suppl.1), 5249--5253
• 4: Bradley, P., Bennett, P and. Demiriz., A. (2000) Constrained k-means clustering. Technical report, MSR-TR-2000-6 5 Microsoft Research.
• 5: Banerjee, A. & Ghosh, J. (2002). Frequency Sensitive Competitive Learning for Clustering on High-dimensional Hypersphere, International Joint Conference on Neural Networks (IJCNN), pp. 1590--95.
• 6: G. W. Furnas , T. K. Landauer , L. M. Gomez , S. T. Dumais, The vocabulary problem in human-system communication, Communications of the ACM, v.30 n.11, p.964-971, Nov. 1987 [doi>10.1145/32206.32212]
• 7: Wei Ding , Gary Marchionini, A study on video browsing strategies, University of Maryland at College Park, College Park, MD, 1997
• 8: G. W. Furnas , T. K. Landauer , L. M. Gomez , S. T. Dumais, The vocabulary problem in human-system communication, Communications of the ACM, v.30 n.11, p.964-971, Nov. 1987 [doi>10.1145/32206.32212]
• 9: van de Stadt, R. (2000). CyberChair, an Online Submission and Reviewing System or: A Program Chair's Best Friend, WWW9.
• 10: Gerard Salton , Michael J. McGill, Introduction to Modern Information Retrieval, McGraw-Hill, Inc., New York, NY, 1986
• 11: Porter, M. F., (1980), An algorithm for suffix stripping, Program, 14(3) :130--137
• 12: C. Lee Giles , Kurt D. Bollacker , Steve Lawrence, CiteSeer: an automatic citation indexing system, Proceedings of the third ACM conference on Digital libraries, p.89-98, June 23-26, 1998, Pittsburgh, Pennsylvania, United States [doi>10.1145/276675.276685]
• 13: Rocchio, J. (1971) Relevance feedback in information retrieval, in. The SMART Retrieval System: Experiments in Automatic Document Processing. Prentice-Hall Inc., pg 313--323.
• 14: Richard B. Segal , Jeffrey O. Kephart, MailCat: an intelligent assistant for organizing e-mail, Proceedings of the third annual conference on Autonomous Agents, p.276-282, April 1999, Seattle, Washington, United States [doi>10.1145/301136.301209]
• 15: Basu, C., Hirsh, H., Cohen, W., and Nevill-Manning, C., (1999). Recommending Papers by Mining the Web, Proc. IJCAI Workshops on Learning About Users and Machine Learning for Information Filtering, IJCAI 99, Stockholm, Sweden.
• 16: Geller, J. and Scherl, R., 1997 Challenge: Technology for Automated Reviewer Selection, IJCAI 1997 55--61
• 17: Susan T. Dumais , Jakob Nielsen, Automating the assignment of submitted manuscripts to reviewers, Proceedings of the 15th annual international ACM SIGIR conference on Research and development in information retrieval, p.233-244, June 21-24, 1992, Copenhagen, Denmark [doi>10.1145/133160.133205]
• 18: Mark Steyvers , Padhraic Smyth , Michal Rosen-Zvi , Thomas Griffiths, Probabilistic author-topic models for information discovery, Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, August 22-25, 2004, Seattle, WA, USA [doi>10.1145/1014052.1014087]
• 19: Gideon S. Mann , David Mimno , Andrew McCallum, Bibliometric impact measures leveraging topic analysis, Proceedings of the 6th ACM/IEEE-CS joint conference on Digital libraries, June 11-15, 2006, Chapel Hill, NC, USA [doi>10.1145/1141753.1141765]
• 20: Jaime Carbonell , Jade Goldstein, The use of MMR, diversity-based reranking for reordering documents and producing summaries, Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, p.335-336, August 24-28, 1998, Melbourne, Australia [doi>10.1145/290941.291025]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.2 DATABASE MANAGEMENT
      H.2.8 Database applications
          Subjects: Data mining

General Terms:
Algorithms, Human Factors

Keywords:
clustering, information retrieval, keyword extraction, similarity functions