The university research assessment dilemma: a decision support system for the next evaluation campaigns

Paolo Fantozzi; Valerio Ficcadenti; Maurizio Naldi

doi:10.1007/s11192-025-05263-9

The university research assessment dilemma: a decision support system for the next evaluation campaigns

Paolo Fantozzi, Valerio Ficcadenti, Maurizio Naldi

Research output: Contribution to journal › Article › peer-review

Abstract

Our study examines the UK’s Research Excellence Framework 2021 employing an algorithmic method to mimic its outcomes (expressed by their panel experts) and introduce a decision support system for evaluating research outputs. Using CrossRef, Scopus databases, and the Chartered Association of Business Schools’ journal classification, we assessed bibliometric features, finding the citation-based algorithm most effective in producing results close to the ones resulting from the REF panellists. Simulating panellists manually adjusting algorithmic paper classifications, our results closely align with actual evaluations, demonstrating the potential of algorithms to augment human assessments. We also show that the Grade Point Average metric may lead to evaluations that are far from those of panellists and should be avoided.

Original language	English
Article number	103957
Journal	Scientometrics
DOIs	https://doi.org/10.1007/s11192-025-05263-9
Publication status	Published - 27 Feb 2025

Keywords

Outputs evaluation
REF
Research Excellence Framework
Research evaluation
Research quality classification algorithms

Access to Document

10.1007/s11192-025-05263-9

Manuscript_SCIEAccepted author manuscript, 1.03 MBLicence: CC BY 4.0

Cite this

@article{04752c6a880e4f9db1caa9026ebf4b6e,

title = "The university research assessment dilemma: a decision support system for the next evaluation campaigns",

abstract = "Our study examines the UK{\textquoteright}s Research Excellence Framework 2021 employing an algorithmic method to mimic its outcomes (expressed by their panel experts) and introduce a decision support system for evaluating research outputs. Using CrossRef, Scopus databases, and the Chartered Association of Business Schools{\textquoteright} journal classification, we assessed bibliometric features, finding the citation-based algorithm most effective in producing results close to the ones resulting from the REF panellists. Simulating panellists manually adjusting algorithmic paper classifications, our results closely align with actual evaluations, demonstrating the potential of algorithms to augment human assessments. We also show that the Grade Point Average metric may lead to evaluations that are far from those of panellists and should be avoided.",

keywords = "Outputs evaluation, REF, Research Excellence Framework, Research evaluation, Research quality classification algorithms",

author = "Paolo Fantozzi and Valerio Ficcadenti and Maurizio Naldi",

year = "2025",

month = feb,

day = "27",

doi = "10.1007/s11192-025-05263-9",

language = "English",

journal = "Scientometrics",

issn = "0138-9130",

publisher = "Akademiai Kiado",

}

TY - JOUR

T1 - The university research assessment dilemma: a decision support system for the next evaluation campaigns

AU - Fantozzi, Paolo

AU - Ficcadenti, Valerio

AU - Naldi, Maurizio

PY - 2025/2/27

Y1 - 2025/2/27

N2 - Our study examines the UK’s Research Excellence Framework 2021 employing an algorithmic method to mimic its outcomes (expressed by their panel experts) and introduce a decision support system for evaluating research outputs. Using CrossRef, Scopus databases, and the Chartered Association of Business Schools’ journal classification, we assessed bibliometric features, finding the citation-based algorithm most effective in producing results close to the ones resulting from the REF panellists. Simulating panellists manually adjusting algorithmic paper classifications, our results closely align with actual evaluations, demonstrating the potential of algorithms to augment human assessments. We also show that the Grade Point Average metric may lead to evaluations that are far from those of panellists and should be avoided.

AB - Our study examines the UK’s Research Excellence Framework 2021 employing an algorithmic method to mimic its outcomes (expressed by their panel experts) and introduce a decision support system for evaluating research outputs. Using CrossRef, Scopus databases, and the Chartered Association of Business Schools’ journal classification, we assessed bibliometric features, finding the citation-based algorithm most effective in producing results close to the ones resulting from the REF panellists. Simulating panellists manually adjusting algorithmic paper classifications, our results closely align with actual evaluations, demonstrating the potential of algorithms to augment human assessments. We also show that the Grade Point Average metric may lead to evaluations that are far from those of panellists and should be avoided.

KW - Outputs evaluation

KW - REF

KW - Research Excellence Framework

KW - Research evaluation

KW - Research quality classification algorithms

U2 - 10.1007/s11192-025-05263-9

DO - 10.1007/s11192-025-05263-9

M3 - Article

SN - 0138-9130

JO - Scientometrics

JF - Scientometrics

M1 - 103957

ER -