Measuring conference quality by mining program committee characteristics

ABSTRACT

Bibliometrics are important measures for venue quality in digital libraries. Impacts of venues are usually the major consideration for subscription decision-making, and for ranking and recommending high-quality venues and documents. For digital libraries in the Computer Science literature domain, conferences play a major role as an important publication and dissemination outlet. However, with a recent profusion of conferences and rapidly expanding fields, it is increasingly challenging for researchers and librarians to assess the quality of conferences. We propose a set of novel heuristics to automatically discover prestigious (and low-quality) conferences by mining the characteristics of Program Committee members. We examine the proposed cues both in isolation and combination under a classification scheme. Evaluation on a collection of 2,979 conferences and 16,147 PC members shows that our heuristics, when combined, correctly classify about 92% of the conferences, with a low false positive rate of 0.035 and a recall of more than 73% for identifying reputable conferences. Furthermore, we demonstrate empirically that our heuristics can also effectively detect a set of low-quality conferences, with a false positive rate of merely 0.002. We also report our experience of detecting two previously unknown low-quality conferences. Finally, we apply the proposed techniques to the entire quality spectrum by ranking conferences in the collection.

REFERENCES

• 1: A. Barabasi, H. Jeong, Z. Neda, E. Ravasz, A. Schubert, and T. Vicsek, Evolution of the social network of scientific collaborations, Physica A, 311(3-4):590--614, 2002.
• 2: Philip A. Bernstein , David DeWitt , Andreas Heuer , Zachary Ives , Christian S. Jensen , Holger Meyer , M. Tamer Özsu , Richard T. Snodgrass , Kyu-Young Whang , Jennifer Widom, Database publication practices, Proceedings of the 31st international conference on Very large data bases, August 30-September 02, 2005, Trondheim, Norway
• 3: J. Bollen, M. A. Rodriguez, and H. Van de Sompel, Journal Status, retrieved at http://www.arxiv.org/abs/cs.GL/0601030, 2006.
• 4: Johan Bollen , Herbert Van de Sompel , Joan A. Smith , Rick Luce, Toward alternative metrics of journal impact: a comparison of download and citation data, Information Processing and Management: an International Journal, v.41 n.6, p.1419-1440, December 2005 [doi>10.1016/j.ipm.2005.03.024]
• 5: M. M. Dalkilic, W. T. Clark, J. C. Costello, and P. Radivojac, Using Compression to Identify Classes of Inauthentic Texts, Proceedings of the 2006 SIAM Conference on Data Mining, April 2006.
• 6: Ergin Elmacioglu , Dongwon Lee, On six degrees of separation in DBLP-DB and more, ACM SIGMOD Record, v.34 n.2, June 2005 [doi>10.1145/1083784.1083791]
• 7: E. Elmacioglu and D. Lee, Oracle, Where Shall I Submit My Papers?, to appear in the Communications of the ACM (CACM).
• 8: L. C. Freeman, A Set of Measures of Centrality Based on Betweenness, Sociometry, Vol. 40, pp. 35--41, 1977.
• 9: Yoav Freund , Robert E. Schapire, A decision-theoretic generalization of on-line learning and an application to boosting, Journal of Computer and System Sciences, v.55 n.1, p.119-139, Aug. 1997 [doi>10.1006/jcss.1997.1504 ]
• 10: E. Garfield, Citation Indexes for Science: A New Dimension in Documentation through Association of Ideas, Science, Vol:122, No:3159, p. 108--111, 1955.
• 11: J. E. Hirsch, An index to quantify an individual's scientific research output, Proceedings of the National Academy of Sciences, Vol. 102(46):16569--16572, 2005.
• 12: Y. Hong, B. -W. On, and D. Lee, System Support for Name Authority Control Problem in Digital Libraries: OpenDBLP Approach, In 8th European Conf. on Digital Libraries (ECDL), page 134--144, Bath, UK, September 2004.
• 13: Pairin Katerattanakul , Bernard Han , Soongoo Hong, Objective quality ranking of computing journals, Communications of the ACM, v.46 n.10, p.111-114, October 2003 [doi>10.1145/944217.944221]
• 14: R. Kohavi, A study of cross-validation and bootstrap for accuracy estimation and model selection, Proceedings of the International Joint Conference on Artificial Intelligence, pages 1137--1145, 1995.
• 15: Birger Larsen , Peter Ingwersen, Using citations for ranking in digital libraries, Proceedings of the 6th ACM/IEEE-CS joint conference on Digital libraries, June 11-15, 2006, Chapel Hill, NC, USA [doi>10.1145/1141753.1141865]
• 16: Gideon S. Mann , David Mimno , Andrew McCallum, Bibliometric impact measures leveraging topic analysis, Proceedings of the 6th ACM/IEEE-CS joint conference on Digital libraries, June 11-15, 2006, Chapel Hill, NC, USA [doi>10.1145/1141753.1141765]
• 17: Sridhar Nerur , Riyaz Sikora , George Mangalaraj , VenuGopal Balijepally, Assessing the relative influence of journals in a citation network, Communications of the ACM, v.48 n.11, p.71-74, November 2005 [doi>10.1145/1096000.1096007]
• 18: M. Newman, Who is the best connected scientist? A study of scientific coauthorship networks, Physical Review, Vol. 64, 2001.
• 19: Alexandros Ntoulas , Marc Najork , Mark Manasse , Dennis Fetterly, Detecting spam web pages through content analysis, Proceedings of the 15th international conference on World Wide Web, May 23-26, 2006, Edinburgh, Scotland [doi>10.1145/1135777.1135794]
• 20: J. Ross Quinlan, C4.5: programs for machine learning, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1993
• 21: J. R. Quinlan, Bagging, Boosting, and C4.5, Proceedings of the 13th National Conference on Artificial Intelligence Conference (AAAI), Vol.1:725--730, 1996.
• 22: Erhard Rahm , Andreas Thor, Citation analysis of database publications, ACM SIGMOD Record, v.34 n.4, p.48-53, December 2005 [doi>10.1145/1107499.1107505]
• 23: S. Saha, S. Saint, and D. A. Christakis, Impact factor: a valid measure of journal quality?, Journal of the Medical Library Association, Vol. 91(1): 42--46, 2003.
• 24: Bernhard Schölkopf , Christopher J. C. Burges , Alexander J. Smola, Advances in kernel methods: support vector learning, MIT Press, Cambridge, MA, 1999
• 25: S. Wasserman and K. Faust, Social Networks Analysis: Methods and Application, Cambridge University Press, United Kingdom, 1994.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.7 Digital Libraries

Additional Classification:
  H. Information Systems
  H.4 INFORMATION SYSTEMS APPLICATIONS
      H.4.m Miscellaneous

  I. Computing Methodologies
  I.7 DOCUMENT AND TEXT PROCESSING
      I.7.4 Electronic Publishing

General Terms:
Algorithms, Design, Experimentation, Measurement

Keywords:
bibliometrics, call for papers, data mining, impact factor, program committee, ranking, social network