Towards usage-based impact metrics: first results from the mesur project.

ABSTRACT

Scholarly usage data holds the potential to be used as a tool to study the dynamics of scholarship in real time, and to form the basis for the definition of novel metrics of scholarly impact. However, the formal groundwork to reliably and validly exploit usage data is lacking, and the exact nature, meaning and applicability of usage-based metrics is poorly understood. The MESUR project funded by the Andrew W. Mellon Foundation constitutes a systematic effort to define, validate and cross-validate a range of usage-based metrics of scholarly impact. MESUR has collected nearly 1 billion usage events as well as all associated bibliographic and citation data from significant publishers, aggregators and institutional consortia to construct a large-scale usage data reference set. This paper describes some major challenges related to aggregating and processing usage data, and discusses preliminary results obtained from analyzing the MESUR reference data set. The results confirm the intrinsic value of scholarly usage data, and support the feasibility of reliable and valid usage-based metrics of scholarly impact.

References

1. C. C. Aggarwal and P. S. Yu. Mining large itemsets for association rules. IEEE Data Engineering Bulletin, 21(1), March 1998.
2. J. Bollen, M. A. Rodriguez, and H. Van de Sompel. Journal status. Scientometrics, 69:669--687, 2006.
3. J. Bollen and H. Van de Sompel. An architecture for the aggregation and analysis of scholarly usage data. In Joint Conference on Digital Libraries (JCDL2006), pages 298--307, Chapel Hill, NC, June 2006.
4. J. Bollen and H. Van de Sompel. Usage impact factor: the effects of sample characteristics on usage-based impact metrics. Journal of the American Society for Information Science and technology, 59(1):001--014, 2008.
5. J. Bollen, H. Van de Sompel, J. Smith, and R. Luce. Toward alternative metrics of journal impact: a comparison of download and citation data. Information Processing and Management, 41(6):1419---1440, 2005.
6. M. Bordons, M. T. Fernandez, and I. Gomez. Advantages and limitations in the use of impact factor measures for the assessment of research performance. Scientometrics, 53(2):195--206, 2002.
7. K. W. Boyack, B. N. Wylie, and G. S. Davidson. Domain visualization using VxInsight for science and technology management. J. Am. Soc. Inf. Sci. Technol., 53(9):764--774, 2002.
8. T. Brody and S. Harnad. Earlier web usage statistics as predictors of later citation impact. Eprint 10647, University of Southampton, ECS, Intelligence, Agents, and Multimedia Group, 2005.
9. T. M. J. Fruchterman and E. M. Reingold. Graph drawing by force-directed placement. Software - Practice and Experience, 21(11):1129--1164, 1991.
10. I. Jolliffe. Principal Component Analysis. Springer, New York, 2002.
11. D.-H. Kim, I. Im, N. Adam, V. Atluri, M. Bieber, and Y. Yesha. A clickstream-based collaborative filtering personalization model: Towards a better performance. In Proceedings of the 6th ACM International Workshop on Web Information and Data Management (WIDM 2004), Washington, DC, November 2004.
12. R. Kothari, P. Mittal, V. Jain, and M. Mohania. On using page co-occurrences for computing clickstream similarity. In Third SIAM International Conference on Data Mining, pages 154--165, May 2003.
13. M. J. Kurtz, G. Eichhorn, A. Accomazzi, C. S. Grant, M. Demleitner, and S. S. Murray. The bibliometric properties of article readership information. Journal of the American Society for information science, 56(2):111--128, 2004.
14. M. J. Kurtz, G. Eichhorn, A. Accomazzi, C. S. Grant, M. Demleitner, and S. S. Murray. Worldwide use and impact of the NASA Astrophysics Data System digital library. Journal of the American Society for information science, 56(1):36--45, 2004.
15. E. Miller. An introduction to the resource description framework. D-Lib Magazine, May 1998.
16. Nature. Editorial: Not-so-deep impact. Nature, 435:1003--1004, June 2005.
17. M. E. J. Newman. A measure of betweenness centrality based on random walks. Social Networks, 27(1):39--54, January 2005.
18. T. Opthof. Sense and nonsense about the impact factor. Cardiovascular Research, 33:1--7, 1997.
19. M. A. Rodriguez, J. Bollen, and H. V. de Sompel. A practical ontology for the large-scale modeling of scholarly artifacts and their usage. In Proceedings of the Joint Conference on Digital Libraries (JCDL 2007), pages 278--287, Vancouver, June 2007.
20. B. Sarwar, G. Karypis, J. Konstan, and J. Reidl. Item-based collaborative filtering recommendation algorithms. In Proceedings of the Tenth International Conference on World Wide Web, pages 285--295. ACM Press, 2001.
21. P. T. Shepherd. Pro ject COUNTER - setting international standards for online usage statistics. Journal of Information Processing and Management, 47(4):245--257, 2004.
22. P. T. Shepherd. The feasibility of developing and implementing journal usage factors: a research project sponsored by UKSG. Serials: The Journal for the Serials Community, 20(2):117--123, 2007.
23. S. Wasserman and K. Faust. Social Network Analysis. Cambridge University Press, Cambridge, 1994.

Index Terms

• Applied computing

  • Computers in other domains

    • Digital libraries and archives

• Computing methodologies

  • Artificial intelligence

    • Knowledge representation and reasoning

      • Semantic networks

• Information systems

  • Information retrieval

    • Document representation

      • Document collection models

  • Information systems applications

    • Data mining

    • Digital libraries and archives