Skip to main content
Log in

Research on impact evaluation of open access journals

  • Published:
Scientometrics Aims and scope Submit manuscript

Abstract

Impact evaluation of open access journals represent how citation indicator and open metric indicators may be affected by open access. Citation indicators and open metric indicators are used in integrated assessments of impact evaluation. Here we test, systematically across many indicators, how well such impact indicators capture the impacts of OA journals. Using the Scopus as data sources for impact evaluation of OA journals, we find that the majority of indicators underestimate the value of freedom in impact evaluation such as citation indicators, while impacts on citation impact are overestimated in impact evaluation. This has important implications for impact evaluation of academic exchange that rely on citation frequency. It also means that citation indicators reflect part of impact of OA journal. This research evaluates the impact of OA journals from citation indicators and open metric indicators, which plays an important role in the development of OA journals.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  • Abramo, G., & D’Angelo, C. A. (2011). Evaluating research: From informed peer review to bibliometrics. Scientometrics,87(3), 499–514.

    Google Scholar 

  • Adler, N. J., & Harzing, A. W. (2009). When knowledge wins: Transcending the sense and nonsense of academic rankings. Academy of Management Learning & Education,8(1), 72–95.

    Google Scholar 

  • Berhidi, A., Szluka, P., et al. (2009). New bibliometric indicators. Is this the end of the impact factor era? Magyar Onkologia,53(2), 115.

    Google Scholar 

  • Bornmann, L., & Daniel, H. D. (2005). Does the h-index for ranking of scientists really work? Scientometrics,65(3), 391–392.

    Google Scholar 

  • Bornmann, L., & Daniel, H. D. (2007). What do we know about the h index? Journal of the Association for Information Science and Technology,58(9), 1381–1385.

    Google Scholar 

  • Bornmann, L., & Daniel, H. D. (2008). Are there better indices for evaluation purposes than the h index? A comparison of nine different variants of the h index using data from biomedicine. New York: Wiley.

    Google Scholar 

  • Braun, T. and W. Nzel, et al. (2005). A Hirsch-type index for journals the scientist magazine of the life sciences. Scientist 19.

  • Braun, T., Glänzel, W., et al. (2006). A Hirsch-type index for journals. Scientometrics,69(1), 169–173.

    Google Scholar 

  • Butler, D. (2008). Free journal-ranking tool enters citation market. Nature,451(7174), 6.

    Google Scholar 

  • Colledge, L., Moya Anegón, F. D., et al. (2010). SJR and SNIP: Two new journal metrics in Elsevier’s Scopus. Serials the Journal for the Serials Community,23(3), 215–221.

    Google Scholar 

  • Costas, R., & Bordons, M. (2007). The h-index: Advantages, limitations and its relation with other bibliometric indicators at the micro level. Journal of Informetrics,1(3), 193–203.

    Google Scholar 

  • Cronin, B., & Meho, L. (2014). Using the h-index to rank influential information scientists. Journal of the Association for Information Science & Technology,57(9), 1275–1278.

    Google Scholar 

  • Da Silva, J. A. T., & Memon, A. R. (2017). CiteScore: A cite for sore eyes, or a valuable, transparent metric? Scientometrics,111, 1–4.

    Google Scholar 

  • Efremenkova, V. M., & Gonnova, S. M. (2016). A comparison of Scopus and WoS database subject classifiers in mathematical disciplines. Scientific & Technical Information Processing,43(2), 115–122.

    Google Scholar 

  • Eger, T., Scheufen, M., et al. (2016). The determinants of open access publishing: Survey evidence from countries in the Mediterranean Open Access Network (MedOANet). Economia E Politica Industriale forthcoming,4, 1–27.

    Google Scholar 

  • Egghe, L. (2000). Theory and practice of the g-Index. Scientometrics,69(1), 131–152.

    Google Scholar 

  • Falagas, M. E., Kouranos, V. D., et al. (2008). Comparison of SCImago journal rank indicator with journal impact factor. Faseb Journal Official Publication of the Federation of American Societies for Experimental Biology,22(8), 2623.

    Google Scholar 

  • Franceschet, M. (2010). A comparison of bibliometric indicators for computer science scholars and journals on Web of Science and Google Scholar. Scientometrics,83(1), 243–258.

    Google Scholar 

  • Garfield, E. (1955). Citation indexes for science; A new dimension in documentation through association of ideas. International Journal of Epidemiology,122(3159), 108.

    Google Scholar 

  • Garfield, E. (1964). “Science Citation Index”—a new dimension in indexing. Science,144(3619), 649.

    Google Scholar 

  • Garfield, E. (1972). Citation analysis as a tool in journal evaluation. Science,178(4060), 471–479.

    Google Scholar 

  • Garfield, E., & Sher, I. H. (1963). New factors in the evaluation of scientific literature through citation indexing. Journal of the Association for Information Science and Technology,14(3), 195–201.

    Google Scholar 

  • Glänzel, W. (2006). On the h-index—a mathematical approach to a new measure of publication activity and citation impact. Scientometrics,67(2), 315–321.

    Google Scholar 

  • Glänzel, W., & Moed, H. F. (2002). Journal impact measures in bibliometric research. Scientometrics,53(2), 171–193.

    Google Scholar 

  • González-Pereira, B., Guerrero-Bote, V. P., et al. (2010). A new approach to the metric of journals’ scientific prestige: The SJR indicator. Journal of Informetrics,4(3), 379–391.

    Google Scholar 

  • Guerrero-Bote, V. P., & Moya-Anegón, F. (2012). A further step forward in measuring journals’scientific prestige: The SJR2 indicator. Journal of Informetrics,6(4), 674–688.

    Google Scholar 

  • Harzing, A. W., & Ron, V. D. W. (2009). A Google scholar h-index for journals: An alternative metric to measure journal impact in economics and business? Journal of the American Society for Information Science and Technology,60(1), 41–46.

    Google Scholar 

  • Harzing, A. W., & Wal, R. V. D. (2009). A Google scholar h-index for journals: An alternative metric to measure journal impact in economics and business. Journal of the Association for Information Science and Technology,60(1), 41–46.

    Google Scholar 

  • Heneberg, P. (2019). The troubles of high-profile open access megajournals. Scientometrics,120(2), 733–746.

    Google Scholar 

  • Hirsch, J. E. (2005). An index to quantify an individual’s scientific research output. Proceedings of the National Academy of Sciences of the United States of America,102(46), 16569–16572.

    MATH  Google Scholar 

  • Hodge, D. R., & Lacasse, J. R. (2011). Evaluating Journal Quality: Is the H-Index a Better Measure than Impact Factors? Research on Social Work Practice,20(2), 222–230.

    Google Scholar 

  • Jin, B. H. (2007). The R- and AR-indices: Complementing the -index. Chinese Science Bulletin,52(6), 855–863.

    Google Scholar 

  • Laakso, M., & Björk, B. (2016). Hybrid open access—A longitudinal study. Journal of Informetrics,10, 919–932.

    Google Scholar 

  • Lancho-Barrantes, B. S., Guerrero-Bote, V. P., et al. (2010). What lies behind the averages and significance of citation indicators in different disciplines? Journal of Information Science,36(3), 371–382.

    Google Scholar 

  • Lehmann, S., Jackson, A. D., et al. (2006). Measures for measures. Nature,444(7122), 1003–1004.

    Google Scholar 

  • Leydesdorff, L., & Milojević, S. (2015). The citation impact of german sociology journals: Some problems with the use of scientometric indicators in journal and research evaluations. Zeitschrift Für Sozialwissenschaftliche Forschung Und Praxis,66(2), 193–204.

    Google Scholar 

  • Leydesdorff, L., & Opthof, T. (2010). Scopus’s Source Normalized Impact per Paper (SNIP) versus a journal impact factor based on fractional counting of citations. Journal of the Association for Information Science and Technology,61(11), 2365–2369.

    Google Scholar 

  • Mingers, J. (2009). Measuring the research contribution of management academics using the Hirsch-index. Journal of the Operational Research Society,60(9), 1143–1153.

    Google Scholar 

  • Mingers, J., & Leydesdorff, L. (2015). Identifying research fields within business and management: A journal cross-citation analysis. Journal of the Operational Research Society,66(8), 1370–1384.

    Google Scholar 

  • Mingers, J., Macri, F., et al. (2012). Using the h-index to measure the quality of journals in the field of business and management. Information Processing and Management,48(2), 234–241.

    Google Scholar 

  • Moed, H. F. (2010). Measuring contextual citation impact of scientific journals. Journal of Informetrics,4(3), 265–277.

    Google Scholar 

  • Moed, H. F. (2011). The source normalized impact per paper is a valid and sophisticated indicator of journal citation impact. Journal of the Association for Information Science and Technology,62(1), 211–213.

    Google Scholar 

  • Moed, H. F. (2012). The future of research evaluation rests with an intelligent combination of advanced metrics and transparent peer review. Science & Public Policy,34(8), 575–583.

    Google Scholar 

  • Moussa, S., & Touzani, M. (2010). Ranking marketing journals using the Google scholar-based hg -index. Journal of Informetrics,4(1), 107–117.

    Google Scholar 

  • Norris, M., & Oppenheim, C. (2010). The h-index: a broad review of a new bibliometric indicator. Journal of Documentation,66(5), 681–705.

    Google Scholar 

  • Oppenheim, C. (2007). Using the h-index to rank influential British researchers in information science and librarianship: Brief communication. New York: Wiley.

    Google Scholar 

  • Oppenheim, C. (2014). Using the h-index to rank influential British researchers in information science and librarianship. Journal of the American Society for Information Science and Technology,58(2), 297–301.

    Google Scholar 

  • Priem, J. & H. Piwowar (2012). The launch of impact story: Using altmetrics to tell data-driven stories.

  • Raan, A. F. J. V. (2006). Comparisonof the Hirsch-index with standard bibliometric indicators and with peerjudgment. Scientometrics,67(3), 491–502.

    Google Scholar 

  • Ramin, S., & Sarraf, S. A. (2012). Comparison between Impact factor, SCImago journal rank indicator and Eigenfactor score of nuclear medicine journals. Nuclear Medicine Review Central & Eastern Europe,15(2), 132–136.

    Google Scholar 

  • Saad, G. (2006). Exploring the h-index at the author and journal levels using bibliometric data of productive consumer scholars and business-related journals respectively. Scientometrics,69(1), 117–120.

    Google Scholar 

  • Schubert, W. (2007). The use of bibliometrics to measure research quality in UK higher education institutions Glänzel. Evidence Ltd for Universities Uk,57(1), 19–32.

    Google Scholar 

  • Schubert, A., & Glänzel, W. (2007). A systematic analysis of Hirsch-type indices for journals. Journal of Informetrics,1(3), 179–184.

    Google Scholar 

  • Truex, D. P. and M. J. Cuellar, et al. (2009). Assessing scholarly influence: Using the hirsch indices to reframe the discourse.

  • Van Noorden, N. (2016). Controversial impact factor gets a heavyweight rival. Nature News,540(7633), 325–326.

    Google Scholar 

  • Waltman, L. (2016). A review of the literature on citation impact indicators. Journal of Informetrics,10(2), 365–391.

    Google Scholar 

  • Wang, X., Chen, L., et al. (2015). The open access advantage considering citation, article usage and social media attention. Scientometrics,103(2), 555–564.

    Google Scholar 

  • Wilsdon, J., Allen, L., et al. (2015). The metric tide: Report of the independent review of the role of metrics in research assessment and management. London: HEFCE. https://doi.org/10.13140/RG.2.1.4929.1363.

    Book  Google Scholar 

Download references

Acknowledgements

The author would like to thank the anonymous reviewers for the valuable comments and suggestions. This paper is supported by Social Science Foundation of Hebei Province in China (Grant No. HB19TQ014).

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Mingkun Wei.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wei, M. Research on impact evaluation of open access journals. Scientometrics 122, 1027–1049 (2020). https://doi.org/10.1007/s11192-019-03306-6

Download citation

  • Received:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11192-019-03306-6

Keywords

Navigation