Life Sciences Forum
Comparison of SCImago journal rank indicator with journal impact factor
First published: 11 April 2008
The application of currently available sophisticated algorithms of citation analysis allows for the incorporation of the “quality” of citations in the evaluation of scientific journals. We sought to compare the newly introduced SCImago journal rank (SJR) indicator with the journal impact factor (IF). We retrieved relevant information from the official Web sites hosting the above indices and their source databases. The SJR indicator is an open-access resource, while the journal IF requires paid subscription. The SJR indicator (based on Scopus data) lists considerably more journal titles published in a wider variety of countries and languages, than the journal IF (based on Web of Science data). Both indices divide citations to a journal by articles of the journal, during a specific time period. However, contrary to the journal IF, the SJR indicator attributes different weight to citations depending on the “prestige” of the citing journal without the influence of journal self-citations; prestige is estimated with the application of the PageRank algorithm in the network of journals. In addition, the SJR indicator includes the total number of documents of a journal in the denominator of the relevant calculation, whereas the journal IF includes only “citable” articles (mainly original articles and reviews). A 3-yr period is analyzed in both indices but with the use of different approaches. Regarding the top 100 journals in the 2006 journal IF ranking order, the median absolute change in their ranking position with the use of the SJR indicator is 32 (1st quartile: 12; 3rd quartile: 75). Although further validation is warranted, the novel SJR indicator poses as a serious alternative to the well-established journal IF, mainly due to its openaccess nature, larger source database, and assessment of the quality of citations.—Falagas, M. E., Kouranos, V. D., Arencibia-Jorge, R., Karageorgopoulos, D. E. Comparison of SCImago journal rank indicator with journal impact factor. FASEB J. 22, 2623–2628 (2008)
THE EVALUATION OF THE QUALITY of research is important for various professional societies, individual scientists, scholarly institutions, and funding organizations (1). The quality of a scientific contribution is primarily estimated from the long-term impact that it has in science. The latter can be inferred from the citations in scientific articles that a contribution receives. These principles have been applied in the evaluation of scientific journals (2). The journal impact factor (IF), first conceived in 1955 by Eugene Garfield, the founder of the Institute for Scientific Information (ISI), has been extensively used in the past decades as an index of quality of scientific journals (3) and is based on citation analysis (4).
Although the journal impact factor has been widely regarded as the best instrument for the evaluation of the quality of scientific journals, it has not been spared from criticism(5
). Main points of consideration regarding methodological aspects in the calculation of this index include the lack of assessment of the quality of citations (9
), the inclusion of self-citations (10
), the poor comparability between different scientific fields (13
), and the analysis of mainly English-language publications (14
In fact, many researchers have proposed different approaches in the evaluation of the quality of scientific journals. The common point in most of these approaches is the assessment of the quality of citations received by a journal (17
). The quality of citations can be estimated analyzing the networks of scientific papers with sophisticated mathematical algorithms (21
). The PageRank algorithm, used in the evaluation of webpages by the popular Google search engines, has been proposed as an appropriate model for the evaluation of the quality of citations in scientific journals (22
). In fact, a group of researchers at the University of Washington developed a similar algorithm for the evaluation of the influence of scientific journals included in the Thompson Scientific Journal Citation Reports (JCR) dataset (24
). Furthermore, another research group from Spanish Universities developed an indicator, named SCImago Journal Rank (SJR) indicator, for the assessment of the quality of scientific journals, applying the PageRank algorithm on the Scopus database (25
). In this evolving context, we sought to identify and evaluate the main characteristics and differences between the widely used journal IF by ISI and the newly introduced SJR indicator.
We searched (January 2008) in the official relevant Web sites for information regarding the main characteristics of the journal IF, provided by JCR through ISI Web of Science, Thomson Scientific, and the SJR indicator, provided by the SCImago journal and country rank Web site, and developed by the SCImago research group. We also searched for information pertinent to the characteristics of these two indices of quality of scientific journals in the official Web sites hosting the databases used by each one of the indices (ISI Web of Science and Scopus for journal IF and SJR indicator, respectively).
In addition, we listed the journals with the top 100 journal IFs and retrieved information regarding their ranking in the SJR indicator list by matching their international standard serial number (ISSN). We also listed the journals with the top 100 SJR indicators and found their ranking in the list of journal IFs. Finally, we calculated the median value as well as the first and third quartile values of the absolute change in ranking order of the journals in the two top 100 lists (with the use of each one of the compared indices vs. the other).
The main characteristics of the evaluation of scientific journals in JCR and in SCImago journal rank are summarized in Table 1
. The journal IF of a specific journal for a specific calendar year is defined by the total number of citations (references) by articles published during the specific year in “source” journals (potentially including the respective journal) to any article of the specific journal that was published during the preceding 2 yr, divided by the total number of “citable” articles published in the respective journal during the preceding 2 yr. Articles regarded as citable are mainly original research and review articles.
Table 1. Main characteristics of the evaluation of scientific journals by journal citation reports and SCImago journal and country rank.
a 6166 science journals, 1768 social science journals.
The SJR indicator of a specific journal for a three-calendar-year period is calculated through an iteration process that computes the “prestige” gained by the journal through the transfer of prestige from all the other journals included in the network of journals, by their citations during the past 3 yr to all articles of the specific journal published in the past 3 yr, divided by the total number of articles of the specific journal during the 3 yr period in regard. The amount of prestige of each journal transferred to another journal in the network is computed by considering the percentage of citations of the former journal that are directed to articles of the latter journal (26
presents the first 20 ranked journals of all categories with each one of the compared indices and their corresponding rank using the other index. Of the 20 journals with the highest journal IFs, 13 retain a position in the top 20 journals with the use of the SJR indicator, and vice versa.
Regarding the top 100 journals in the 2006 journal IF ranking order, the median absolute change in their ranking position with the use of the SJR indicator is 32 (1st quartile: 12; 3rd quartile: 75). Conversely, regarding the top 100 journals with the current SJR indicator, the median absolute change in their ranking position with the use of the journal IF is 29 (1st quartile: 10.5; 3rd quartile: 65.5).
Table 2. Comparative rankings of the top 20 journals by journal impact factor and SCImago journal rank indicator
The journals with the greatest relative increase in their ranking order in the SJR indicator compared to the journal IF were the journals Immunity, Molecular Cell, Annual Review of Cell and Developmental Biology, Cell, and Current Opinion in Cell Biology. The journals with the greatest relative stability in their ranking order from the journal IF to the SJR classification were Annual Review of Physiology, Annual Review of Biophysics and Biomolecular Structure, PLoS Biology, An-
nual Review of Plant Biology,andBlood. The journals that exhibited the greatest relative decrease in their ranking order in the SJR indicator compared to the journal IF were Annual Review of Astronomy and Astrophysics, JAMA-Journal of the American Medical Association, Nature Physics, Advances in Catalysis, and Behavioral and Brain Sciences. It should be mentioned that all five of the latter journals exhibited a fall of more than 4000 positions in their ranking order with the use of the SJR indicator compared to the journal IF. Exploring the considerable discrepancy in the rankings of the latter five journals, we noted as potential causes a computational error, the presence of another journal with the same title but different ISSN, lack of source data, and a divergence between the number of citable articles and of total articles or documents. Furthermore, one of the top 100 journals according to the journal IF classification (Trends in Ecology & Evolution) was not included in the SJR database.
The main differences between the journal IF and the SJR indicator derive mainly from differences in the scientific databases used as the sources of citations, as well as from differences in the methodology of estimation of these indices. The latter primarily regard the weight attributed to citations, the way of handling self-citations, the temporal window analyzed, and the type and number of the articles of a journal considered in the denominator of calculation ofthe aforementioned indices. In terms of utility, the main novelty introduced by the SJR indicator is open access.
Regarding the differences in the scientific databases, on which the compared indices of evaluation of scientific journals are applied, Scopus includes a substantially larger collection of journals, originating from remarkably more countries and published in a greater variety of languages (27
). Thus, in this regard it can be assumed that SJR may provide a more comprehensive estimation of the scientific value of journals, particularly so for those published in non-English languages (16
). This is why the latter category of journals receives a great percentage of the overall incoming citations from non-English journals (14
). The latter are rather underrepresented in the Web of Science database. Furthermore, the Web of Science takes into consideration citations originating from a subset of source journals (28
), potentially excluding some journals published in non-English languages, a factor that may influence the evaluation of this category of journals (14
). Yet, it should be mentioned that although Scopus includes a larger collection of non-English journals, the latter are still underrepresented, constituting ~15% of the total number of included journals (29
). Moreover, the apparent advantage of Scopus in citation analysis originating from the breadth of its database is limited to the time period after 1996 for which citation analysis is available (30
Regarding the methodology of the calculation of the two compared indices, the most significant difference lies in the fact that the SJR indicator takes into account not only the absolute number but also the “quality” of citations received by a journal, whereas the journal IF considers incoming citations only in a quantitative manner. It is plausible that the articles of a journal have a greater impact in science if they are cited by journals of higher scientific quality. Such an analysis may not have been feasible in the past, though in today's electronic era, powerful computational systems provide the opportunity to apply sophisticated algorithms for the evaluation of interactions between journals in a huge network or universe of scientific publications. However, the simple and comprehensible methodology used in the calculation of the journal IF, despite the controversy raised over the years for the output data (31
), has been regarded as one ofthe most favorable attributes of this long-used standard of reference in the field of analysis of scientific citations.
It should be noted though that the simple methodology used in the calculation of the journal IF has allowed editors to use various practices aiming to increase the impact factor of their journals (5
). The principal ones are probably the promotion of self-citations (10
), the predilection for review articles (36
), and the decrease in the total number of included articles (37
). Notably, the use of the SJR indicator allows for the estimation of a journal's impact without the influence of self-citations, since prestige can be transferred to a journal by all other journals but not by itself. Instead, in JCR providing the journal IFs, self-citation analysis for each journal can be separately performed. However, this factor is not incorporated in the calculation of the journal IFs. Regarding the weight of the different types of articles in the process of calculating the two compared indices of scientific journals, no provision has been made for differentiating between the value assigned to original research articles compared to review articles in any of the indices.
One of the major shortcomings of the SJR indicator may be the fact that it divides the prestige gained by a journal, through the citations of its articles, to the total number of articles included, rather than to the number of citable articles, as is used in the calculation of the journal IF. Although the strategy used in the calculation of the SJR indicator may be mathematically valid, since in theory any published article can be cited and all citations are taken into account in the numerator of the fraction, in practice, article types such as correspondence articles, letters to the editor, commentaries, perspectives, news, obituaries, editorials, interviews, and tributes are unlikely to receive a significant number of citations (3
). In this regard, the scientific quality of journals that contain a large number of the latter types of articles, which may otherwise be of interest to the reader, are expected to be appreciably underestimated with the SJR indicator. Yet, it should be mentioned that neither is the journal IF an optimal index of quality of scientific journals regarding this issue, since it does not adjust for the fact that a great number of total citations addressed to a journal are received by a relatively short portion of the included articles (3
The effect of the differences in the time periods used for the assessment of the two herein compared indices of quality of scientific journals on the accuracy of the estimates cannot be directly inferred in the context of the present study. It should be noted though that both of the indices refer to the past three calendar-year periods, although through a different approach. This temporal window may be relatively short, particularly for journals with an appreciable interval between receipt of an article and publication (3
An indisputable advantage of journal IF over any new index of evaluation of quality of scientific journals is tradition. On the one hand, authors compete for publishing the products of their research in highly ranked journals (41
). On the other hand, journal editors elect to publish scientific articles that are expected to be highly cited (37
). Thus, it can be assumed that journal IF rankings would have influenced accordingly the quality of the journals over the years in which they have been considered as the sole standard of reference (3
Although our study did not aim to systematically assess the comparability in journal rankings between the novel SJR indicator and the well-established journal IF index, we observed that the introduction of the SJR indicator does not bring about radical changes in this regard. In fact, half of the journals in the top 100 journal IF list are placed within a reasonable range of 32 ranking places in the SJR indicator journal list. Some striking inconsistencies observed in the rankings of the new SJR indicator, compared to the journal impact factor, should be attributed to the “precocity” of the new indicator rather than to systematic differences in the methodologies used. In fact, the two methodologies have been systematically compared in previous studies. A weighted PageRank algorithm applied in the same dataset used for the derivation of the journal impact factor provided results that correlated significantly with the later index (23
). However, this correlation was moderately strong and differed between various scientific fields, for instance, for medical journals (9
The SCImago journal rank indicator is a novel instrument for the evaluation of scientific journals that may challenge the established premiership of the journal IF in ranking scientific journals. It provides unrestricted (open) access, is based on a larger source journal database, and focuses on the quality of citations that a journal receives by other journals, rather than the absolute number. However, the sophisticated methodology used in the calculation of the SJR indicator needs to be adequately validated, and certain characteristics may need to be reconsidered before definitive conclusions for its applicability could be drawn. It appears, though, that the election of one index or the other would be mostly a matter of whether the popularity or the quality of a journal is considered as the primary criterion for the evaluation of scientific journals.
The authors have no conflicts of interest to declare. No funding was received for this study.
11 April 2008
Version of Record online:
11 April 2008
© 2021 Federation of American Societies for Experimental Biology (FASEB)
About Wiley Online Library
Help & Support
Connect with Wiley