Can artificial intelligence separate the wheat from the chaff in systematic reviews of health economic articles?

M. J. Oude Wolcherink, X. G.L.V. Pouwels, S. H.B. van Dijk, C. J.M. Doggen, H. Koffijberg*

*Corresponding author for this work

Research output: Contribution to journalArticleAcademicpeer-review

11 Downloads (Pure)


Objectives: Artificial intelligence-powered tools, such as ASReview, could reduce the burden of title and abstract screening. This study aimed to assess the accuracy and efficiency of using ASReview in a health economic context. Methods: A sample from a previous systematic literature review containing 4,994 articles was used. Previous manual screening resulted in 134 articles included for full-text screening (FT) and 50 for data extraction (DE). Here, accuracy and efficiency was evaluated by comparing the number of identified relevant articles with ASReview versus manual screening. Pre-defined stopping rules using sampling criteria and heuristic criteria were tested. Robustness of the AI-tool’s performance was determined using 1,000 simulations. Results: Considering included stopping rules, median accuracy for FT articles remained below 85%, but reached 100% for DE articles. To identify all relevant articles, a median of 89.9% of FT articles needed to be screened, compared to 7.7% for DE articles. Potential time savings between 49 and 59 hours could be achieved, depending on the stopping rule. Conclusions: In our case study, all DE articles were identified after screening 7.7% of the sample, allowing for substantial time savings. ASReview likely has the potential to substantially reduce screening time in systematic reviews of health economic articles.

Original languageEnglish
Pages (from-to)1049-1056
Number of pages8
JournalExpert Review of Pharmacoeconomics and Outcomes Research
Issue number9
Early online date13 Aug 2023
Publication statusPublished - 21 Oct 2023


  • accuracy
  • Artificial intelligence
  • asreview
  • efficiency
  • simulation
  • stopping rule
  • systematic review
  • UT-Hybrid-D


Dive into the research topics of 'Can artificial intelligence separate the wheat from the chaff in systematic reviews of health economic articles?'. Together they form a unique fingerprint.

Cite this