Distinguishing citation quality for journal impact assessment

ABSTRACT

Introduction

The research community has long and often been fervently keen on debating the topic of journal impact. Well, just what is the impact of a journal? Today, the Science Citation Index (SCI) recognizes over 7,000 journals. The sheer number of available journals renders it pivotal for researchers to accurately gauge a journal's impact when submitting their papers, as it has become commonplace that researchers regard publishing their work in established journals to have significant influence on peer recognition. For journals in Management Information System (MIS), such research studies have continuously been published since the 1990s. Nine of them have been summarized by Carol Saunders, whereby seven were based on respondent perceptions by surveying experts, and two were based on the citation quantity to indicate the journal impact.

It is generally accepted that citation analysis is purported to be a more objective method than the expert survey. The main reason is citation analysis uses objective measurements, which are based on the viewpoint that the influence of a journal and its articles is determined by their usefulness to other journals and articles, and where their usage can be reflected by citations that they have received. However, using citation quantity only is also considered to have bias to a certain degree, due to a widely held notion that citation quantity does not represent citation quality. Regarding the impact in the MIS discipline for example, a citation by a paper published in a prestigious MIS journal should far outweigh a citation by a paper published in an unremarkable MIS journal or in an external journal outside the MIS field. Such intuition suggests that the citation quality can be divided into the following two aspects:

• Citation Relevance (CR): indicating how relevant the journal giving the citation is to the discipline we are interested in;

• Citation Importance (CI): indicating how important the journal giving the citation is in the discipline we are interested in.

However, these concerns about citation quality have not been properly addressed in citation analysis literature.

To address our concerns about citation quality for assessing journal impact, we propose a method, which first clusters "pure" MIS journals to identify relevant citations, and then score the impact for each journal, according to its citations that are received from pure MIS journals and weighted by citation importance. Although our method is only applied to MIS journals, it is general enough to evaluate the impact of journals in other disciplines.

REFERENCES

• 1: Roger A. Horn , Charles R. Johnson, Matrix analysis, Cambridge University Press, New York, NY, 1985
• 2: Pairin Katerattanakul , Bernard Han , Soongoo Hong, Objective quality ranking of computing journals, Communications of the ACM, v.46 n.10, p.111-114, October 2003 [doi>10.1145/944217.944221]
• 3: Pairin Katerattanakul , Bernard Han , Alan Rea, Is information systems a reference discipline?, Communications of the ACM, v.49 n.5, p.114-118, May 2006 [doi>10.1145/1125944.1125950]
• 4: Katerattanakul, P., Razi, M.A., Han, B.T. and Kam, H.J. Consistency AND Concern ON IS Journal Rankings. The Journal of Information Technology Theory and Application (JITTA) 7, 2. 1--20.
• 5: Nikolaos A. Mylonopoulos , Vasilis Theoharakis, On site: global perceptions of IS journals, Communications of the ACM, v.44 n.9, p.29-33, Sept. 2001 [doi>10.1145/383694.383701]
• 6: Sridhar Nerur , Riyaz Sikora , George Mangalaraj , VenuGopal Balijepally, Assessing the relative influence of journals in a citation network, Communications of the ACM, v.48 n.11, p.71-74, November 2005 [doi>10.1145/1096000.1096007]
• 7: Page, L., Brin, S., Motwani, R. and Winograd, T. The pagerank citation ranking: Bringing order to the Web. Technical report, Stanford Digital Library Technologies Project, 1998.
• 8: Pieters, R. and Baumgartner, H. Who talks to whom? Intra-and interdisciplinary communication of economics journals. Journal of Economic Literature 40, 2, 483--509.
• 9: Pinski, G.a.N., F. Citation influence for journal aggregates of scientific publications: Theory, with application to the literature of physics. Information Processing and Management 12, 297--312.
• 10: Punj, G. and Stewart, D.W. Cluster Analysis in Marketing Research: Review and Suggestions for Application. Journal of Marketing Research 20, 2, 134--148.
• 11: Saunders, C. MIS journal ranking. ISWorldnet, 2006; http://www.isworld.org/csaunders/rankings.htm, Associations for Information System, 2006.
• 12: Vermunt, J.K. LEM: Log-Linear and Event History Analysis with Missing Data. Tilburg, Netherlands: Tilburg University, version, 1.

INDEX TERMS

Primary Classification:
  A. General Literature
  A.2 REFERENCE

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.1 Content Analysis and Indexing