Rethinking research impact assessment: a multidimensional approach

Sergio Manrique, Marta Natalia Wroblewska, Bradley Good

Research output: Working paperProfessional

45 Downloads (Pure)

Abstract

An interest in the evaluation of research impact – or the influence of scientific research beyond academia – has been observable worldwide. Several countries have introduced national research assessment systems which take into account this new element of evaluation. So far, research on this practice has focused mainly on the practicalities of the different existing policies: the definition of the term ‘research impact’, different approaches to measuring it, their relative challenges and the possible use of such evaluations. But the introduction of a new element of evaluation gives rise not only to challenges of a practical nature, but also to important ethical consequences in terms of academic identity, reflexivity, power structures, distribution of labour in terms of workloads etc. In order to address these questions and the relevant needs of researchers, in this paper we propose a multidimensional model that considers different attributes of research impact: Responsiveness, Accessibility, Reflexivity, Ecology and Adaptability. This holistic, multidimensional model of evaluation, designed particularly for self-assessment or internal assessment, recognises the qualities a project has on these different scales in a broader perspective, rather than offering a simple and single numerical evaluation. This model addresses many of the ethical dilemmas that accompany conducting impact-producing research. To exemplify the usefulness of the proposed model, the authors provide real-life research project assessment examples conducted with the use of the Multidimensional Approach for Research Impact Assessment (MARIA Model).
Original languageEnglish
Place of PublicationEnschede
PublisherCenter for Higher Education Policy Studies (CHEPS)
Number of pages37
DOIs
Publication statusPublished - 2018

Publication series

NameRUNIN Working Paper Series
No.15
Volume2018
ISSN (Electronic)2535-5686

Fingerprint

impact research
evaluation
reflexivity
distribution structure
self-assessment
workload
ecology
research project
labor

Cite this

Manrique, S., Wroblewska, M. N., & Good, B. (2018). Rethinking research impact assessment: a multidimensional approach. (RUNIN Working Paper Series; Vol. 2018, No. 15). Enschede: Center for Higher Education Policy Studies (CHEPS). https://doi.org/10.3990/4.2535-5686.2018.15
Manrique, Sergio ; Wroblewska, Marta Natalia ; Good, Bradley. / Rethinking research impact assessment : a multidimensional approach. Enschede : Center for Higher Education Policy Studies (CHEPS), 2018. (RUNIN Working Paper Series; 15).
@techreport{8ed9d6c49ce54957a8fd686555daf1b5,
title = "Rethinking research impact assessment: a multidimensional approach",
abstract = "An interest in the evaluation of research impact – or the influence of scientific research beyond academia – has been observable worldwide. Several countries have introduced national research assessment systems which take into account this new element of evaluation. So far, research on this practice has focused mainly on the practicalities of the different existing policies: the definition of the term ‘research impact’, different approaches to measuring it, their relative challenges and the possible use of such evaluations. But the introduction of a new element of evaluation gives rise not only to challenges of a practical nature, but also to important ethical consequences in terms of academic identity, reflexivity, power structures, distribution of labour in terms of workloads etc. In order to address these questions and the relevant needs of researchers, in this paper we propose a multidimensional model that considers different attributes of research impact: Responsiveness, Accessibility, Reflexivity, Ecology and Adaptability. This holistic, multidimensional model of evaluation, designed particularly for self-assessment or internal assessment, recognises the qualities a project has on these different scales in a broader perspective, rather than offering a simple and single numerical evaluation. This model addresses many of the ethical dilemmas that accompany conducting impact-producing research. To exemplify the usefulness of the proposed model, the authors provide real-life research project assessment examples conducted with the use of the Multidimensional Approach for Research Impact Assessment (MARIA Model).",
author = "Sergio Manrique and Wroblewska, {Marta Natalia} and Bradley Good",
note = "Subseries of the CHEPS working paper series as part of the UT & ECIU RUNIN Ph.D. training network. Paul Benneworth is the series editor This working paper and its related poster were presented at the Austrian Presidency of the EU Council Conference on the Impact of Social Sciences and Humanities for a European Research Agenda – Valuation of SSH in Mission-oriented Research, held in Vienna, Austria on November 28th-29th 2018, where the authors were awarded the 1st Prize for the Best Poster of this conference by the ZSI – Centre for Social Innovation. Both this preprint and the poster are being published as part of the conference proceedings in the fteval Journal for Research and Technology Policy Evaluation. This project has received funding from the European Union’s Horizon 2020 research and innovation programme under Marie Skłodowska-Curie grant agreement No. 722295.",
year = "2018",
doi = "10.3990/4.2535-5686.2018.15",
language = "English",
series = "RUNIN Working Paper Series",
publisher = "Center for Higher Education Policy Studies (CHEPS)",
number = "15",
address = "Netherlands",
type = "WorkingPaper",
institution = "Center for Higher Education Policy Studies (CHEPS)",

}

Manrique, S, Wroblewska, MN & Good, B 2018 'Rethinking research impact assessment: a multidimensional approach' RUNIN Working Paper Series, no. 15, vol. 2018, Center for Higher Education Policy Studies (CHEPS), Enschede. https://doi.org/10.3990/4.2535-5686.2018.15

Rethinking research impact assessment : a multidimensional approach. / Manrique, Sergio; Wroblewska, Marta Natalia; Good, Bradley.

Enschede : Center for Higher Education Policy Studies (CHEPS), 2018. (RUNIN Working Paper Series; Vol. 2018, No. 15).

Research output: Working paperProfessional

TY - UNPB

T1 - Rethinking research impact assessment

T2 - a multidimensional approach

AU - Manrique, Sergio

AU - Wroblewska, Marta Natalia

AU - Good, Bradley

N1 - Subseries of the CHEPS working paper series as part of the UT & ECIU RUNIN Ph.D. training network. Paul Benneworth is the series editor This working paper and its related poster were presented at the Austrian Presidency of the EU Council Conference on the Impact of Social Sciences and Humanities for a European Research Agenda – Valuation of SSH in Mission-oriented Research, held in Vienna, Austria on November 28th-29th 2018, where the authors were awarded the 1st Prize for the Best Poster of this conference by the ZSI – Centre for Social Innovation. Both this preprint and the poster are being published as part of the conference proceedings in the fteval Journal for Research and Technology Policy Evaluation. This project has received funding from the European Union’s Horizon 2020 research and innovation programme under Marie Skłodowska-Curie grant agreement No. 722295.

PY - 2018

Y1 - 2018

N2 - An interest in the evaluation of research impact – or the influence of scientific research beyond academia – has been observable worldwide. Several countries have introduced national research assessment systems which take into account this new element of evaluation. So far, research on this practice has focused mainly on the practicalities of the different existing policies: the definition of the term ‘research impact’, different approaches to measuring it, their relative challenges and the possible use of such evaluations. But the introduction of a new element of evaluation gives rise not only to challenges of a practical nature, but also to important ethical consequences in terms of academic identity, reflexivity, power structures, distribution of labour in terms of workloads etc. In order to address these questions and the relevant needs of researchers, in this paper we propose a multidimensional model that considers different attributes of research impact: Responsiveness, Accessibility, Reflexivity, Ecology and Adaptability. This holistic, multidimensional model of evaluation, designed particularly for self-assessment or internal assessment, recognises the qualities a project has on these different scales in a broader perspective, rather than offering a simple and single numerical evaluation. This model addresses many of the ethical dilemmas that accompany conducting impact-producing research. To exemplify the usefulness of the proposed model, the authors provide real-life research project assessment examples conducted with the use of the Multidimensional Approach for Research Impact Assessment (MARIA Model).

AB - An interest in the evaluation of research impact – or the influence of scientific research beyond academia – has been observable worldwide. Several countries have introduced national research assessment systems which take into account this new element of evaluation. So far, research on this practice has focused mainly on the practicalities of the different existing policies: the definition of the term ‘research impact’, different approaches to measuring it, their relative challenges and the possible use of such evaluations. But the introduction of a new element of evaluation gives rise not only to challenges of a practical nature, but also to important ethical consequences in terms of academic identity, reflexivity, power structures, distribution of labour in terms of workloads etc. In order to address these questions and the relevant needs of researchers, in this paper we propose a multidimensional model that considers different attributes of research impact: Responsiveness, Accessibility, Reflexivity, Ecology and Adaptability. This holistic, multidimensional model of evaluation, designed particularly for self-assessment or internal assessment, recognises the qualities a project has on these different scales in a broader perspective, rather than offering a simple and single numerical evaluation. This model addresses many of the ethical dilemmas that accompany conducting impact-producing research. To exemplify the usefulness of the proposed model, the authors provide real-life research project assessment examples conducted with the use of the Multidimensional Approach for Research Impact Assessment (MARIA Model).

U2 - 10.3990/4.2535-5686.2018.15

DO - 10.3990/4.2535-5686.2018.15

M3 - Working paper

T3 - RUNIN Working Paper Series

BT - Rethinking research impact assessment

PB - Center for Higher Education Policy Studies (CHEPS)

CY - Enschede

ER -

Manrique S, Wroblewska MN, Good B. Rethinking research impact assessment: a multidimensional approach. Enschede: Center for Higher Education Policy Studies (CHEPS). 2018. (RUNIN Working Paper Series; 15). https://doi.org/10.3990/4.2535-5686.2018.15