research-article

Towards usage-based impact metrics: first results from the mesur project.

Authors:

Herbert Van de Sompel,

Marko A. RodriguezAuthors Info & Claims

JCDL '08: Proceedings of the 8th ACM/IEEE-CS joint conference on Digital libraries

Pages 231 - 240

https://doi.org/10.1145/1378889.1378928

Published: 16 June 2008 Publication History

Abstract

Scholarly usage data holds the potential to be used as a tool to study the dynamics of scholarship in real time, and to form the basis for the definition of novel metrics of scholarly impact. However, the formal groundwork to reliably and validly exploit usage data is lacking, and the exact nature, meaning and applicability of usage-based metrics is poorly understood. The MESUR project funded by the Andrew W. Mellon Foundation constitutes a systematic effort to define, validate and cross-validate a range of usage-based metrics of scholarly impact. MESUR has collected nearly 1 billion usage events as well as all associated bibliographic and citation data from significant publishers, aggregators and institutional consortia to construct a large-scale usage data reference set. This paper describes some major challenges related to aggregating and processing usage data, and discusses preliminary results obtained from analyzing the MESUR reference data set. The results confirm the intrinsic value of scholarly usage data, and support the feasibility of reliable and valid usage-based metrics of scholarly impact.

References

[1]

C. C. Aggarwal and P. S. Yu. Mining large itemsets for association rules. IEEE Data Engineering Bulletin, 21(1), March 1998.

[2]

J. Bollen, M. A. Rodriguez, and H. Van de Sompel. Journal status. Scientometrics, 69:669--687, 2006.

[3]

J. Bollen and H. Van de Sompel. An architecture for the aggregation and analysis of scholarly usage data. In Joint Conference on Digital Libraries (JCDL2006), pages 298--307, Chapel Hill, NC, June 2006.

Digital Library

[4]

J. Bollen and H. Van de Sompel. Usage impact factor: the effects of sample characteristics on usage-based impact metrics. Journal of the American Society for Information Science and technology, 59(1):001--014, 2008.

Digital Library

[5]

J. Bollen, H. Van de Sompel, J. Smith, and R. Luce. Toward alternative metrics of journal impact: a comparison of download and citation data. Information Processing and Management, 41(6):1419---1440, 2005.

Digital Library

[6]

M. Bordons, M. T. Fernandez, and I. Gomez. Advantages and limitations in the use of impact factor measures for the assessment of research performance. Scientometrics, 53(2):195--206, 2002.

[7]

K. W. Boyack, B. N. Wylie, and G. S. Davidson. Domain visualization using VxInsight for science and technology management. J. Am. Soc. Inf. Sci. Technol., 53(9):764--774, 2002.

Digital Library

[8]

T. Brody and S. Harnad. Earlier web usage statistics as predictors of later citation impact. Eprint 10647, University of Southampton, ECS, Intelligence, Agents, and Multimedia Group, 2005.

[9]

T. M. J. Fruchterman and E. M. Reingold. Graph drawing by force-directed placement. Software - Practice and Experience, 21(11):1129--1164, 1991.

Digital Library

[10]

I. Jolliffe. Principal Component Analysis. Springer, New York, 2002.

[11]

D.-H. Kim, I. Im, N. Adam, V. Atluri, M. Bieber, and Y. Yesha. A clickstream-based collaborative filtering personalization model: Towards a better performance. In Proceedings of the 6th ACM International Workshop on Web Information and Data Management (WIDM 2004), Washington, DC, November 2004.

Digital Library

[12]

R. Kothari, P. Mittal, V. Jain, and M. Mohania. On using page co-occurrences for computing clickstream similarity. In Third SIAM International Conference on Data Mining, pages 154--165, May 2003.

[13]

M. J. Kurtz, G. Eichhorn, A. Accomazzi, C. S. Grant, M. Demleitner, and S. S. Murray. The bibliometric properties of article readership information. Journal of the American Society for information science, 56(2):111--128, 2004.

Digital Library

[14]

M. J. Kurtz, G. Eichhorn, A. Accomazzi, C. S. Grant, M. Demleitner, and S. S. Murray. Worldwide use and impact of the NASA Astrophysics Data System digital library. Journal of the American Society for information science, 56(1):36--45, 2004.

Digital Library

[15]

E. Miller. An introduction to the resource description framework. D-Lib Magazine, May 1998.

[16]

Nature. Editorial: Not-so-deep impact. Nature, 435:1003--1004, June 2005.

[17]

M. E. J. Newman. A measure of betweenness centrality based on random walks. Social Networks, 27(1):39--54, January 2005.

[18]

T. Opthof. Sense and nonsense about the impact factor. Cardiovascular Research, 33:1--7, 1997.

[19]

M. A. Rodriguez, J. Bollen, and H. V. de Sompel. A practical ontology for the large-scale modeling of scholarly artifacts and their usage. In Proceedings of the Joint Conference on Digital Libraries (JCDL 2007), pages 278--287, Vancouver, June 2007.

Digital Library

[20]

B. Sarwar, G. Karypis, J. Konstan, and J. Reidl. Item-based collaborative filtering recommendation algorithms. In Proceedings of the Tenth International Conference on World Wide Web, pages 285--295. ACM Press, 2001.

Digital Library

[21]

P. T. Shepherd. Pro ject COUNTER - setting international standards for online usage statistics. Journal of Information Processing and Management, 47(4):245--257, 2004.

[22]

P. T. Shepherd. The feasibility of developing and implementing journal usage factors: a research project sponsored by UKSG. Serials: The Journal for the Serials Community, 20(2):117--123, 2007.

[23]

S. Wasserman and K. Faust. Social Network Analysis. Cambridge University Press, Cambridge, 1994.

Cited By

Lynch CCha EPatel MJacob KPrabhu MPawlowski HVanjani NSingh K(2022)How Does Open Access Publication Impact Readership and Citation Rates of Lumbar Spine Literature?Clinical Spine Surgery: A Spine Publication10.1097/BSD.000000000000130335:6(E558-E565)Online publication date: 3-Mar-2022
https://doi.org/10.1097/BSD.0000000000001303
Jang HChun KKim H(2018)Comparison between Korean and foreign authors concerning the citation impact of Korean journals indexed in ScopusScience Editing10.6087/kcse.1476:1(47-57)Online publication date: 8-Nov-2018
https://doi.org/10.6087/kcse.147
Vogl SScherndl TKühberger A(2018)#PsychologyScientometrics10.5555/3238974.3239052115:3(1253-1269)Online publication date: 1-Jun-2018
https://dl.acm.org/doi/10.5555/3238974.3239052
Show More Cited By

Index Terms

Towards usage-based impact metrics: first results from the mesur project.

Recommendations

Usage impact factor: The effects of sample characteristics on usage-based impact metrics

There exist ample demonstrations that indicators of scholarly impact analogous to the citation-based ISI Impact Factor can be derived from usage data; however, so far, usage can practically be recorded only at the level of distinct information services. ...
Usage metrics vs classical metrics: analysis of Russia's research output

This paper discusses the results of a pilot project investigating Russian scholarly publications using the altmetric indicators "Usage Count Last 180 days" (U1) and "Usage Count Since 2013" (U2) introduced by Web of Science. We explored the relationship ...
A ranking and exploration service based on large-scale usage data.
JCDL '08: Proceedings of the 8th ACM/IEEE-CS joint conference on Digital libraries

This poster presents the architecture and user interface of a prototype service that was designed to allow end-users to explore the structure of science and perform assessments of scholarly impact on the basis of large-scale usage data. The underlying ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

JCDL '08: Proceedings of the 8th ACM/IEEE-CS joint conference on Digital libraries

June 2008

490 pages

ISBN:9781595939982

DOI:10.1145/1378889

General Chair:
Ronald Larsen
University of Pittsburgh, USA
,
Program Chairs:
Andreas Paepcke
Stanford University, USA
,
José Borbinha
INESC-ID, Portugal
,
Mor Naaman
Yahoo! Research Berkeley, USA

Copyright © 2008 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 16 June 2008

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

JCDL08

Sponsor:

JCDL08: Joint Conference on Digital Libraries

June 16 - 20, 2008

PA, Pittsburgh PA, USA

Acceptance Rates

JCDL '08 Paper Acceptance Rate 33 of 117 submissions, 28%;

Overall Acceptance Rate 415 of 1,482 submissions, 28%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

21
Total Citations
View Citations
538
Total Downloads

Downloads (Last 12 months)4
Downloads (Last 6 weeks)2

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Lynch CCha EPatel MJacob KPrabhu MPawlowski HVanjani NSingh K(2022)How Does Open Access Publication Impact Readership and Citation Rates of Lumbar Spine Literature?Clinical Spine Surgery: A Spine Publication10.1097/BSD.000000000000130335:6(E558-E565)Online publication date: 3-Mar-2022
https://doi.org/10.1097/BSD.0000000000001303
Jang HChun KKim H(2018)Comparison between Korean and foreign authors concerning the citation impact of Korean journals indexed in ScopusScience Editing10.6087/kcse.1476:1(47-57)Online publication date: 8-Nov-2018
https://doi.org/10.6087/kcse.147
Vogl SScherndl TKühberger A(2018)#PsychologyScientometrics10.5555/3238974.3239052115:3(1253-1269)Online publication date: 1-Jun-2018
https://dl.acm.org/doi/10.5555/3238974.3239052
Vogl SScherndl TKühberger A(2018)#Psychology: a bibliometric analysis of psychological literature in the online mediaScientometrics10.1007/s11192-018-2727-5115:3(1253-1269)Online publication date: 1-Jun-2018
https://dl.acm.org/doi/10.1007/s11192-018-2727-5
Pedrini MLangella VBattaglia MZaratin P(2018)Assessing the health research's social impactScientometrics10.1007/s11192-017-2585-6114:3(1227-1250)Online publication date: 1-Mar-2018
https://dl.acm.org/doi/10.1007/s11192-017-2585-6
Aduku KThelwall MKousha K(2017)Do Mendeley reader counts reflect the scholarly impact of conference papers? An investigation of computer science and engineeringScientometrics10.1007/s11192-017-2367-1112:1(573-581)Online publication date: 1-Jul-2017
https://dl.acm.org/doi/10.1007/s11192-017-2367-1
Fan YGuo JLan YXu JCheng X(2017)Academic Access Data Analysis for Literature RecommendationInformation Retrieval10.1007/978-3-319-68699-8_4(42-54)Online publication date: 21-Oct-2017
https://doi.org/10.1007/978-3-319-68699-8_4
Giuliani MMarasca S(2015)La valutazione della ricerca tramite indici bibliometrici: riflessioni da una prospettiva economico-aziendaleMANAGEMENT CONTROL10.3280/MACO2015-001006(133-151)Online publication date: May-2015
https://doi.org/10.3280/MACO2015-001006
Mikki SZygmuntowska MGjesdal ØAl Ruwehy H(2015)Digital Presence of Norwegian Scholars on Academic Network Sites—Where and Who Are They?PLOS ONE10.1371/journal.pone.014270910:11(e0142709)Online publication date: 13-Nov-2015
https://doi.org/10.1371/journal.pone.0142709
Wang Fvon Laszewski GFox GFurlani TDeLeon RGallo SLathrop SAlameda J(2014)Towards a Scientific Impact Measuring Framework for Large Computing Facilities - a Case Study on XSEDEProceedings of the 2014 Annual Conference on Extreme Science and Engineering Discovery Environment10.1145/2616498.2616507(1-8)Online publication date: 13-Jul-2014
https://dl.acm.org/doi/10.1145/2616498.2616507
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten