Semantometrics: Towards Fulltext-based Research Evaluation

Editor's Notes

1. In this presentation, I would like to talk about the problems of existing impact metrics. These problems led us to the development of a new approach, based on the processing of publication full-texts, for automatically assesing the research contribution.
2. As you might know, there is a wide range of metrics that are currently used for evaluation of research. You can see them in the foloowing tag cloud. These metrics have one thing in common. They are all based on citations. This has both advantages and disdvanatages.They are … Let me say a few more things about the cons Evidence for citations comes externally. Does it mean that people who cite a paper need to read it? Not necessarily.
3. As you might know, there is a wide range of metrics that are currently used for evaluation of research. You can see them in the foloowing tag cloud. These metrics have one thing in common. They are all based on citations. This has both advantages and disdvanatages.They are … Let me say a few more things about the cons Evidence for citations comes externally. Does it mean that people who cite a paper need to read it? Not necessarily.
4. There is no correlation. Does it mean that citation counting does not work, does it mean that peer-review does not work? Probably a bit of both.
5. There are a number of properties a good research evaluation metric should have. (Jennings, 2006) mentions the following: Reliability – Level of accuracy comparable or better than that of the current peer-review system. Digestibility – It must allow to make quick decisions about what to read. Economical and fast to produce – The production of the metric should be possible in reasonable time and shouldn’t be expensive or human labour intensive. Resistance to gaming – It shouldn’t be possible to improve the metric in any other way than by improving the research.
6. There is now also a lot of interest in new metrics, which are referred to as Alt and Webometrics. However, these metrics are (in the same way as citation based metrics) still dependent on the number of interactions in the scholarly communication network. We are proposing a new class of evluation metrics => Semantometrics. These metrics are based on the assujmption that it is not possible to evaluate research impact without accessing the research outputs  . The problem with Altmetrics is that they measure popularity, the size of the community, etc, but not quality.
7. It is an insult to researchers and to science that the current metrics do not consider the text of the manuscript in evaluation, but take external evidence into account only.
8. This hypothesis is based on the process of how research builds on the existing knowledge in order to create new knowledge on which others can build. A publication, which in this way creates a \bridge" between what we already know and something new which will people develop based on this knowledge, brings a contribution to science. A publication has a high contribution if it creates a \long bridge" between more distant areas of science.
9. Dasha and Petr have participated in the challenge which is part of the upcoming Web Search and Data Mining (WSDM) conference. The challenge, coorganised by Microsoft and Elsevier, was to assess the importance of scholarly articles, using data from Microsoft Academic Graph -- a large heterogeneous graph comprised of more than 120 million publications and the related authors, venues, organizations, and fields of study. Dasha and Petr (team called BletchleyPark) were the best out of 32 teams in the training round of the competition and after the validation round were invited to take part in the second phase of the challenge as one of the eight best of the 32 teams. They will also present their method at the workshop in San Francisco, California, in February. - See more at: http://kmi.open.ac.uk/news/article/18807#sthash.kn2oaULQ.dpuf
10. The leads us to the conlusion that it is not possible to evaluate research impact without accessing the research outputs 