Artificial intelligence in systematic reviews: promising when appropriately used

Sanne H.B. van Dijk; Marjolein G.J. Brusse-Keizer; Charlotte C. Bucsán; Job van der Palen; Carine J.M. Doggen; Anke Lenferink

doi:10.1136/bmjopen-2023-072254

Artificial intelligence in systematic reviews: promising when appropriately used

Sanne H.B. van Dijk, Marjolein G.J. Brusse-Keizer, Charlotte C. Bucsán, Job van der Palen, Carine J.M. Doggen, Anke Lenferink^*

^*Corresponding author for this work

Research output: Contribution to journal › Article › Academic › peer-review

86 Citations (Scopus)

300 Downloads (Pure)

Abstract

BACKGROUND: Systematic reviews provide a structured overview of the available evidence in medical-scientific research. However, due to the increasing medical-scientific research output, it is a time-consuming task to conduct systematic reviews. To accelerate this process, artificial intelligence (AI) can be used in the review process. In this communication paper, we suggest how to conduct a transparent and reliable systematic review using the AI tool 'ASReview' in the title and abstract screening.

METHODS: Use of the AI tool consisted of several steps. First, the tool required training of its algorithm with several prelabelled articles prior to screening. Next, using a researcher-in-the-loop algorithm, the AI tool proposed the article with the highest probability of being relevant. The reviewer then decided on relevancy of each article proposed. This process was continued until the stopping criterion was reached. All articles labelled relevant by the reviewer were screened on full text.

RESULTS: Considerations to ensure methodological quality when using AI in systematic reviews included: the choice of whether to use AI, the need of both deduplication and checking for inter-reviewer agreement, how to choose a stopping criterion and the quality of reporting. Using the tool in our review resulted in much time saved: only 23% of the articles were assessed by the reviewer.

CONCLUSION: The AI tool is a promising innovation for the current systematic reviewing practice, as long as it is appropriately used and methodological quality can be assured.

PROSPERO REGISTRATION NUMBER: CRD42022283952.

Original language	English
Article number	e072254
Number of pages	6
Journal	BMJ open
Volume	13
Issue number	7
DOIs	https://doi.org/10.1136/bmjopen-2023-072254
Publication status	Published - 7 Jul 2023

Access to Document

10.1136/bmjopen-2023-072254Licence: CC BY

ArticleFinal published version, 612 KBLicence: CC BY

Cite this

@article{281d1e3d2a3747b8837b5ced40e94c1e,

title = "Artificial intelligence in systematic reviews: promising when appropriately used",

abstract = "BACKGROUND: Systematic reviews provide a structured overview of the available evidence in medical-scientific research. However, due to the increasing medical-scientific research output, it is a time-consuming task to conduct systematic reviews. To accelerate this process, artificial intelligence (AI) can be used in the review process. In this communication paper, we suggest how to conduct a transparent and reliable systematic review using the AI tool 'ASReview' in the title and abstract screening. METHODS: Use of the AI tool consisted of several steps. First, the tool required training of its algorithm with several prelabelled articles prior to screening. Next, using a researcher-in-the-loop algorithm, the AI tool proposed the article with the highest probability of being relevant. The reviewer then decided on relevancy of each article proposed. This process was continued until the stopping criterion was reached. All articles labelled relevant by the reviewer were screened on full text.RESULTS: Considerations to ensure methodological quality when using AI in systematic reviews included: the choice of whether to use AI, the need of both deduplication and checking for inter-reviewer agreement, how to choose a stopping criterion and the quality of reporting. Using the tool in our review resulted in much time saved: only 23% of the articles were assessed by the reviewer.CONCLUSION: The AI tool is a promising innovation for the current systematic reviewing practice, as long as it is appropriately used and methodological quality can be assured.PROSPERO REGISTRATION NUMBER: CRD42022283952.",

author = "{van Dijk}, {Sanne H.B.} and Brusse-Keizer, {Marjolein G.J.} and Bucs{\'a}n, {Charlotte C.} and Palen, {Job van der} and Doggen, {Carine J.M.} and Anke Lenferink",

note = "Funding Information: The systematic review is conducted as part of the RE-SAMPLE project. RE-SAMPLE has received funding from the European Union{\textquoteright}s Horizon 2020 research and innovation programme (grant agreement no. 965315). Publisher Copyright: {\textcopyright} Author(s) (or their employer(s)) 2023. Re-use permitted under CC BY. Published by BMJ. Financial transaction number: 2500079316",

year = "2023",

month = jul,

day = "7",

doi = "10.1136/bmjopen-2023-072254",

language = "English",

volume = "13",

journal = "BMJ open",

issn = "2044-6055",

publisher = "BMJ",

number = "7",

}

TY - JOUR

T1 - Artificial intelligence in systematic reviews: promising when appropriately used

AU - van Dijk, Sanne H.B.

AU - Brusse-Keizer, Marjolein G.J.

AU - Bucsán, Charlotte C.

AU - Palen, Job van der

AU - Doggen, Carine J.M.

AU - Lenferink, Anke

N1 - Funding Information: The systematic review is conducted as part of the RE-SAMPLE project. RE-SAMPLE has received funding from the European Union’s Horizon 2020 research and innovation programme (grant agreement no. 965315). Publisher Copyright: © Author(s) (or their employer(s)) 2023. Re-use permitted under CC BY. Published by BMJ. Financial transaction number: 2500079316

PY - 2023/7/7

Y1 - 2023/7/7

N2 - BACKGROUND: Systematic reviews provide a structured overview of the available evidence in medical-scientific research. However, due to the increasing medical-scientific research output, it is a time-consuming task to conduct systematic reviews. To accelerate this process, artificial intelligence (AI) can be used in the review process. In this communication paper, we suggest how to conduct a transparent and reliable systematic review using the AI tool 'ASReview' in the title and abstract screening. METHODS: Use of the AI tool consisted of several steps. First, the tool required training of its algorithm with several prelabelled articles prior to screening. Next, using a researcher-in-the-loop algorithm, the AI tool proposed the article with the highest probability of being relevant. The reviewer then decided on relevancy of each article proposed. This process was continued until the stopping criterion was reached. All articles labelled relevant by the reviewer were screened on full text.RESULTS: Considerations to ensure methodological quality when using AI in systematic reviews included: the choice of whether to use AI, the need of both deduplication and checking for inter-reviewer agreement, how to choose a stopping criterion and the quality of reporting. Using the tool in our review resulted in much time saved: only 23% of the articles were assessed by the reviewer.CONCLUSION: The AI tool is a promising innovation for the current systematic reviewing practice, as long as it is appropriately used and methodological quality can be assured.PROSPERO REGISTRATION NUMBER: CRD42022283952.

AB - BACKGROUND: Systematic reviews provide a structured overview of the available evidence in medical-scientific research. However, due to the increasing medical-scientific research output, it is a time-consuming task to conduct systematic reviews. To accelerate this process, artificial intelligence (AI) can be used in the review process. In this communication paper, we suggest how to conduct a transparent and reliable systematic review using the AI tool 'ASReview' in the title and abstract screening. METHODS: Use of the AI tool consisted of several steps. First, the tool required training of its algorithm with several prelabelled articles prior to screening. Next, using a researcher-in-the-loop algorithm, the AI tool proposed the article with the highest probability of being relevant. The reviewer then decided on relevancy of each article proposed. This process was continued until the stopping criterion was reached. All articles labelled relevant by the reviewer were screened on full text.RESULTS: Considerations to ensure methodological quality when using AI in systematic reviews included: the choice of whether to use AI, the need of both deduplication and checking for inter-reviewer agreement, how to choose a stopping criterion and the quality of reporting. Using the tool in our review resulted in much time saved: only 23% of the articles were assessed by the reviewer.CONCLUSION: The AI tool is a promising innovation for the current systematic reviewing practice, as long as it is appropriately used and methodological quality can be assured.PROSPERO REGISTRATION NUMBER: CRD42022283952.

U2 - 10.1136/bmjopen-2023-072254

DO - 10.1136/bmjopen-2023-072254

M3 - Article

SN - 2044-6055

VL - 13

JO - BMJ open

JF - BMJ open

IS - 7

M1 - e072254

ER -

Artificial intelligence in systematic reviews: promising when appropriately used

Abstract

Access to Document

Fingerprint

Cite this