Towards usage-based impact metrics: first results from the mesur project.

ABSTRACT

Scholarly usage data holds the potential to be used as a tool to study the dynamics of scholarship in real time, and to form the basis for the definition of novel metrics of scholarly impact. However, the formal groundwork to reliably and validly exploit usage data is lacking, and the exact nature, meaning and applicability of usage-based metrics is poorly understood. The MESUR project funded by the Andrew W. Mellon Foundation constitutes a systematic effort to define, validate and cross-validate a range of usage-based metrics of scholarly impact. MESUR has collected nearly 1 billion usage events as well as all associated bibliographic and citation data from significant publishers, aggregators and institutional consortia to construct a large-scale usage data reference set. This paper describes some major challenges related to aggregating and processing usage data, and discusses preliminary results obtained from analyzing the MESUR reference data set. The results confirm the intrinsic value of scholarly usage data, and support the feasibility of reliable and valid usage-based metrics of scholarly impact.

REFERENCES

• 1: C. C. Aggarwal and P. S. Yu. Mining large itemsets for association rules. IEEE Data Engineering Bulletin, 21(1), March 1998.
• 2: J. Bollen, M. A. Rodriguez, and H. Van de Sompel. Journal status. Scientometrics, 69:669--687, 2006.
• 3: Johan Bollen , Herbert Van de Sompel, An architecture for the aggregation and analysis of scholarly usage data, Proceedings of the 6th ACM/IEEE-CS joint conference on Digital libraries, June 11-15, 2006, Chapel Hill, NC, USA [doi>10.1145/1141753.1141821]
• 4: Johan Bollen , Herbert Van de Sompel, Usage impact factor: The effects of sample characteristics on usage-based impact metrics, Journal of the American Society for Information Science and Technology, v.59 n.1, p.136-149, January 2008 [doi>10.1002/asi.v59:1]
• 5: Johan Bollen , Herbert Van de Sompel , Joan A. Smith , Rick Luce, Toward alternative metrics of journal impact: a comparison of download and citation data, Information Processing and Management: an International Journal, v.41 n.6, p.1419-1440, December 2005 [doi>10.1016/j.ipm.2005.03.024]
• 6: M. Bordons, M. T. Fernandez, and I. Gomez. Advantages and limitations in the use of impact factor measures for the assessment of research performance. Scientometrics, 53(2):195--206, 2002.
• 7: Kevin W. Boyack , Brian N. Wylie , George S. Davidson, Domain visualization using VxInsight for science and technology management, Journal of the American Society for Information Science and Technology, v.53 n.9, p.764-774, August 2002 [doi>10.1002/asi.10066]
• 8: T. Brody and S. Harnad. Earlier web usage statistics as predictors of later citation impact. Eprint 10647, University of Southampton, ECS, Intelligence, Agents, and Multimedia Group, 2005.
• 9: Thomas M. J. Fruchterman , Edward M. Reingold, Graph drawing by force-directed placement, Software—Practice & Experience, v.21 n.11, p.1129-1164, Nov. 1991 [doi>10.1002/spe.4380211102]
• 10: I. Jolliffe. Principal Component Analysis. Springer, New York, 2002.
• 11: Dong-Ho Kim , Vijayalakshmi Atluri , Michael Bieber , Nabil Adam , Yelena Yesha, A clickstream-based collaborative filtering personalization model: towards a better performance, Proceedings of the 6th annual ACM international workshop on Web information and data management, November 12-13, 2004, Washington DC, USA [doi>10.1145/1031453.1031470]
• 12: R. Kothari, P. Mittal, V. Jain, and M. Mohania. On using page co-occurrences for computing clickstream similarity. In Third SIAM International Conference on Data Mining, pages 154--165, May 2003.
• 13: Michael J. Kurtz , Guenther Eichhorn , Alberto Accomazzi , Carolyn Grant , Markus Demleitner , Stephen S. Murray , Nathalie Martimbeau , Barbara Elwell, The bibliometric properties of article readership information: Research Articles, Journal of the American Society for Information Science and Technology, v.56 n.2, p.111-128, January 2005 [doi>10.1002/asi.v56:2]
• 14: Michael J. Kurtz , Guenther Eichhorn , Alberto Accomazzi , Carolyn Grant , Markus Demleitner , Stephen S. Murray, Worldwide use and impact of the NASA Astrophysics Data System digital library: Research Articles, Journal of the American Society for Information Science and Technology, v.56 n.1, p.36-45, January 2005 [doi>10.1002/asi.v56:1]
• 15: E. Miller. An introduction to the resource description framework. D-Lib Magazine, May 1998.
• 16: Nature. Editorial: Not-so-deep impact. Nature, 435:1003--1004, June 2005.
• 17: M. E. J. Newman. A measure of betweenness centrality based on random walks. Social Networks, 27(1):39--54, January 2005.
• 18: T. Opthof. Sense and nonsense about the impact factor. Cardiovascular Research, 33:1--7, 1997.
• 19: Marko A. Rodriguez , Johan Bollen , Herbert Van de Sompel, A practical ontology for the large-scale modeling of scholarly artifacts and their usage, Proceedings of the 7th ACM/IEEE-CS joint conference on Digital libraries, June 18-23, 2007, Vancouver, BC, Canada [doi>10.1145/1255175.1255229]
• 20: Badrul Sarwar , George Karypis , Joseph Konstan , John Reidl, Item-based collaborative filtering recommendation algorithms, Proceedings of the 10th international conference on World Wide Web, p.285-295, May 01-05, 2001, Hong Kong, Hong Kong [doi>10.1145/371920.372071]
• 21: P. T. Shepherd. Pro ject COUNTER - setting international standards for online usage statistics. Journal of Information Processing and Management, 47(4):245--257, 2004.
• 22: P. T. Shepherd. The feasibility of developing and implementing journal usage factors: a research project sponsored by UKSG. Serials: The Journal for the Serials Community, 20(2):117--123, 2007.
• 23: S. Wasserman and K. Faust. Social Network Analysis. Cambridge University Press, Cambridge, 1994.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.7 Digital Libraries
          Subjects: Collection

Additional Classification:
  H. Information Systems
  H.2 DATABASE MANAGEMENT
      H.2.8 Database applications
          Subjects: Data mining

  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.4 Knowledge Representation Formalisms and Methods
          Subjects: Semantic networks

General Terms:
Experimentation, Human Factors, Measurement, Performance, Standardization

Keywords:
data mining, impact factor, mapping, social networks, usage data