TY - JOUR
T1 - Handling missing values in the analysis of between-hospital differences in ordinal and dichotomous outcomes
T2 - A simulation study
AU - Van Linschoten, Reinier C.A.
AU - Amini, Marzyeh
AU - Van Leeuwen, Nikki
AU - Eijkenaar, Frank
AU - Den Hartog, Sanne J.
AU - Nederkoorn, Paul J.
AU - Hofmeijer, Jeannette
AU - Emmer, Bart J.
AU - Postma, Alida A.
AU - Van Zwam, Wim
AU - Roozenbeek, Bob
AU - Dippel, Diederik
AU - Lingsma, Hester F.
N1 - Funding Information:
This study was funded by Amsterdam University Medical Centers (no award/grant number), Erasmus Medisch Centrum (no award/grant number), TWIN Foundation (no award/grant number), Maastricht Universitair Medisch Centrum (no award/grant number).
Publisher Copyright:
© Author(s) (or their employer(s)) 2023. No commercial re-use. See rights and permissions. Published by BMJ.
PY - 2023/12
Y1 - 2023/12
N2 - Missing data are frequently encountered in registries that are used to compare performance across hospitals. The most appropriate method for handling missing data when analysing differences in outcomes between hospitals with a generalised linear mixed model is unclear. We aimed to compare methods for handling missing data when comparing hospitals on ordinal and dichotomous outcomes. We performed a simulation study using data from the Multicentre Randomised Controlled Trial of Endovascular Treatment for Acute Ischaemic Stroke in the Netherlands (MR CLEAN) Registry, a prospective cohort study in 17 hospitals performing endovascular therapy for ischaemic stroke in the Netherlands. The investigated methods for handling missing data, both case-mix adjustment variables and outcomes, were complete case analysis, single imputation, multiple imputation, single imputation with deletion of imputed outcomes and multiple imputation with deletion of imputed outcomes. Data were generated as missing completely at random (MCAR), missing at random and missing not at random (MNAR) in three scenarios: (1) 10% missing data in case-mix and outcome; (2) 40% missing data in case-mix and outcome; and (3) 40% missing data in case-mix and outcome with varying degree of missing data among hospitals. Bias and reliability of the methods were compared on the mean squared error (MSE, a summary measure combining bias and reliability) relative to the hospital effect estimates from the complete reference data set. For both the ordinal outcome (ie, the modified Rankin Scale) and a common dichotomised version thereof, all methods of handling missing data were biased, likely due to shrinkage of the random effects. The MSE of all methods was on average lowest under MCAR and with fewer missing data, and highest with more missing data and under MNAR. The 'multiple imputation, then deletion' method had the lowest MSE for both outcomes under all simulated patterns of missing data. Thus, when estimating hospital effects on ordinal and dichotomous outcomes in the presence of missing data, the least biased and most reliable method to handle these missing data is 'multiple imputation, then deletion'.
AB - Missing data are frequently encountered in registries that are used to compare performance across hospitals. The most appropriate method for handling missing data when analysing differences in outcomes between hospitals with a generalised linear mixed model is unclear. We aimed to compare methods for handling missing data when comparing hospitals on ordinal and dichotomous outcomes. We performed a simulation study using data from the Multicentre Randomised Controlled Trial of Endovascular Treatment for Acute Ischaemic Stroke in the Netherlands (MR CLEAN) Registry, a prospective cohort study in 17 hospitals performing endovascular therapy for ischaemic stroke in the Netherlands. The investigated methods for handling missing data, both case-mix adjustment variables and outcomes, were complete case analysis, single imputation, multiple imputation, single imputation with deletion of imputed outcomes and multiple imputation with deletion of imputed outcomes. Data were generated as missing completely at random (MCAR), missing at random and missing not at random (MNAR) in three scenarios: (1) 10% missing data in case-mix and outcome; (2) 40% missing data in case-mix and outcome; and (3) 40% missing data in case-mix and outcome with varying degree of missing data among hospitals. Bias and reliability of the methods were compared on the mean squared error (MSE, a summary measure combining bias and reliability) relative to the hospital effect estimates from the complete reference data set. For both the ordinal outcome (ie, the modified Rankin Scale) and a common dichotomised version thereof, all methods of handling missing data were biased, likely due to shrinkage of the random effects. The MSE of all methods was on average lowest under MCAR and with fewer missing data, and highest with more missing data and under MNAR. The 'multiple imputation, then deletion' method had the lowest MSE for both outcomes under all simulated patterns of missing data. Thus, when estimating hospital effects on ordinal and dichotomous outcomes in the presence of missing data, the least biased and most reliable method to handle these missing data is 'multiple imputation, then deletion'.
KW - Healthcare quality improvement
KW - Performance measures
KW - Quality improvement
KW - Quality improvement methodologies
KW - Simulation
KW - 2023 OA procedure
UR - http://www.scopus.com/inward/record.url?scp=85172986651&partnerID=8YFLogxK
U2 - 10.1136/bmjqs-2023-016387
DO - 10.1136/bmjqs-2023-016387
M3 - Article
C2 - 37734955
AN - SCOPUS:85172986651
SN - 2044-5415
VL - 32
SP - 742
EP - 749
JO - BMJ quality and safety
JF - BMJ quality and safety
IS - 12
M1 - 016387
ER -
