TY - JOUR
T1 - Choosing a Metamodel of a Simulation Model for Uncertainty Quantification
AU - de Carvalho, Tiago M.
AU - van Rosmalen, Joost
AU - Wolff, Harold B.
AU - Koffijberg, Hendrik
AU - Coupé, Veerle M.H.
N1 - Publisher Copyright:
© The Author(s) 2021.
Funding Information:
We thank 4 anonymous reviewers, whose comments greatly improved this article. The authors received no financial support for the research, authorship, and/or publication of this article.
Publisher Copyright:
© The Author(s) 2021.
PY - 2022/1
Y1 - 2022/1
N2 - Background: Metamodeling may substantially reduce the computational expense of individual-level state transition simulation models (IL-STM) for calibration, uncertainty quantification, and health policy evaluation. However, because of the lack of guidance and readily available computer code, metamodels are still not widely used in health economics and public health. In this study, we provide guidance on how to choose a metamodel for uncertainty quantification. Methods: We built a simulation study to evaluate the prediction accuracy and computational expense of metamodels for uncertainty quantification using life-years gained (LYG) by treatment as the IL-STM outcome. We analyzed how metamodel accuracy changes with the characteristics of the simulation model using a linear model (LM), Gaussian process regression (GP), generalized additive models (GAMs), and artificial neural networks (ANNs). Finally, we tested these metamodels in a case study consisting of a probabilistic analysis of a lung cancer IL-STM. Results: In a scenario with low uncertainty in model parameters (i.e., small confidence interval), sufficient numbers of simulated life histories, and simulation model runs, commonly used metamodels (LM, ANNs, GAMs, and GP) have similar, good accuracy, with errors smaller than 1% for predicting LYG. With a higher level of uncertainty in model parameters, the prediction accuracy of GP and ANN is superior to LM. In the case study, we found that in the worst case, the best metamodel had an error of about 2.1%. Conclusion: To obtain good prediction accuracy, in an efficient way, we recommend starting with LM, and if the resulting accuracy is insufficient, we recommend trying ANNs and eventually also GP regression.
AB - Background: Metamodeling may substantially reduce the computational expense of individual-level state transition simulation models (IL-STM) for calibration, uncertainty quantification, and health policy evaluation. However, because of the lack of guidance and readily available computer code, metamodels are still not widely used in health economics and public health. In this study, we provide guidance on how to choose a metamodel for uncertainty quantification. Methods: We built a simulation study to evaluate the prediction accuracy and computational expense of metamodels for uncertainty quantification using life-years gained (LYG) by treatment as the IL-STM outcome. We analyzed how metamodel accuracy changes with the characteristics of the simulation model using a linear model (LM), Gaussian process regression (GP), generalized additive models (GAMs), and artificial neural networks (ANNs). Finally, we tested these metamodels in a case study consisting of a probabilistic analysis of a lung cancer IL-STM. Results: In a scenario with low uncertainty in model parameters (i.e., small confidence interval), sufficient numbers of simulated life histories, and simulation model runs, commonly used metamodels (LM, ANNs, GAMs, and GP) have similar, good accuracy, with errors smaller than 1% for predicting LYG. With a higher level of uncertainty in model parameters, the prediction accuracy of GP and ANN is superior to LM. In the case study, we found that in the worst case, the best metamodel had an error of about 2.1%. Conclusion: To obtain good prediction accuracy, in an efficient way, we recommend starting with LM, and if the resulting accuracy is insufficient, we recommend trying ANNs and eventually also GP regression.
KW - cost-effectiveness analysis
KW - metamodels/emulators
KW - probabilistic sensitivity analyses
KW - simulation models
KW - uncertainty quantification
UR - http://www.scopus.com/inward/record.url?scp=85107582446&partnerID=8YFLogxK
U2 - 10.1177/0272989X211016307
DO - 10.1177/0272989X211016307
M3 - Article
AN - SCOPUS:85107582446
SN - 0272-989X
VL - 42
SP - 28
EP - 42
JO - Medical decision making
JF - Medical decision making
IS - 1
ER -