About F3-index and Reviewer Contribution Index (RCI)

The F3-index was presented by Federico Bianchi (University of Milan, Italy), Francisco Grimaldo (University of Valencia, Spain) and Flaminio Squazzoni (University of Milan, Italy) in their Open Access paper published by the Journal of Informetrics.

It is the result of comparing the performance of peer reviewers assigned to the same manuscript, given that any submission is different from another in length, depth and required effort. F3-index thus can change over time and regardless of the general activity of the individual peer reviewer. It’s a relative and context-specific index that considers three quantifiable dimensions.

 

Federico Bianchi, Francisco Grimaldo, Flaminio Squazzoni, The F3-index. Valuing reviewers for scholarly journals, Journal of Informetrics, Volume 13, Issue 1, 2019, Pages 78-86, ISSN 1751-1577, https://doi.org/10.1016/j.joi.2018.11.007

 

The choice of dimensions and parameters reflected previous attempts at quantifying reviewer performance (Casnici, Grimaldo, Gilbert, Dondio, et al., 2017; Hartonen & Alava, 2013; Laband, 1990). (…) It does not mean that other factors or dimensions are irrelevant. We have proposed a tool that could be adapted to the context-specific interests of editors and journals, as well as to the quality of available data. Furthermore, we did not aim to identify a “one-size-fits-all” methodology that could reflect context-specific journal characteristics and include varying dimensions [1].

 

The F3-index considers peer reviewers assigned to the same manuscript as match players who compete to deliver a pertinent, informative and timely review.

 

We adapted Keener’s method to develop a rating and ranking algorithm for reviewers of scholarly journals. We considered reviewers as participants of a tournament in which each reviewer was matched over time with others as they were assigned to evaluate the same manuscripts. Similarly to participants in a tournament, reviewers’ strength can be measured by calculating a rating based on their reviewing behaviour, by measuring it together with the behaviour of other reviewers assigned to the same manuscripts and their strength. (…) We calculate attribute measurements by standardizing raw statistics. Then, following a recommendation by Keener (1993), we transform standardized statistics by applying Laplace’s Rule of Succession (1995 [1825]) in order to avoid ‘winner-takes-all’ effects [1].

From F3-index to RCI

ReviewerCredits has implented the F3-index in collaboration with Francisco Grimaldo and Daniel Garcia Costa (Members of Project PREWAIT: Advanced information tools about peer review of scientific manuscripts – University of Valencia), calling it Reviewer Contribution Index (RCI) to enlight:

  • the relevance of the voluntary activity performed by peer reviewers
  • the need to measure it, in order to properly recognize and reward it


At the moment, the calculation and the assignment of the RCI covers peer reviewers of the registered Journals using our OJS Plugin.