Development and validation of clinical prediction models: Marginal differences between logistic regression, penalized maximum likelihood estimation, and genetic programming

Kristel J.M. Janssen, Ivar Siccama, Yvonne Vergouwe, Hendrik Koffijberg, Thomas P.A. Debray, Maarten Keijzer, Diederick E. Grobbee, Karel G.M. Moons

Research output: Contribution to journalArticleAcademicpeer-review

8 Citations (Scopus)

Abstract

Objective: Many prediction models are developed by multivariable logistic regression. However, there are several alternative methods to develop prediction models. We compared the accuracy of a model that predicts the presence of deep venous thrombosis (DVT) when developed by four different methods.

Study Design and Setting: We used the data of 2,086 primary care patients suspected of DVT, which included 21 candidate predictors. The cohort was split into a derivation set (1,668 patients, 329 with DVT) and a validation set (418 patients, 86 with DVT). Also, 100 cross-validations were conducted in the full cohort. The models were developed by logistic regression, logistic regression with shrinkage by bootstrapping techniques, logistic regression with shrinkage by penalized maximum likelihood estimation, and genetic programming. The accuracy of the models was tested by assessing discrimination and calibration.

Results: There were only marginal differences in the discrimination and calibration of the models in the validation set and cross-validations. Conclusion: The accuracy measures of the models developed by the four different methods were only slightly different, and the 95% confidence intervals were mostly overlapped. We have shown that models with good predictive accuracy are most likely developed by sensible modeling strategies rather than by complex development methods.
Original languageEnglish
Pages (from-to)404-412
Number of pages9
JournalJournal of clinical epidemiology
Volume65
Issue number4
DOIs
Publication statusPublished - 1 Apr 2012

Fingerprint

Venous Thrombosis
Logistic Models
Calibration
Primary Health Care
Confidence Intervals

Keywords

  • Genetic programming
  • Logistic regression
  • Penalized maximum likelihood estimation
  • Prediction model

Cite this

Janssen, Kristel J.M. ; Siccama, Ivar ; Vergouwe, Yvonne ; Koffijberg, Hendrik ; Debray, Thomas P.A. ; Keijzer, Maarten ; Grobbee, Diederick E. ; Moons, Karel G.M. / Development and validation of clinical prediction models: Marginal differences between logistic regression, penalized maximum likelihood estimation, and genetic programming. In: Journal of clinical epidemiology. 2012 ; Vol. 65, No. 4. pp. 404-412.
@article{e7585c12b80046e88ef2d124fee07142,
title = "Development and validation of clinical prediction models: Marginal differences between logistic regression, penalized maximum likelihood estimation, and genetic programming",
abstract = "Objective: Many prediction models are developed by multivariable logistic regression. However, there are several alternative methods to develop prediction models. We compared the accuracy of a model that predicts the presence of deep venous thrombosis (DVT) when developed by four different methods.Study Design and Setting: We used the data of 2,086 primary care patients suspected of DVT, which included 21 candidate predictors. The cohort was split into a derivation set (1,668 patients, 329 with DVT) and a validation set (418 patients, 86 with DVT). Also, 100 cross-validations were conducted in the full cohort. The models were developed by logistic regression, logistic regression with shrinkage by bootstrapping techniques, logistic regression with shrinkage by penalized maximum likelihood estimation, and genetic programming. The accuracy of the models was tested by assessing discrimination and calibration. Results: There were only marginal differences in the discrimination and calibration of the models in the validation set and cross-validations. Conclusion: The accuracy measures of the models developed by the four different methods were only slightly different, and the 95{\%} confidence intervals were mostly overlapped. We have shown that models with good predictive accuracy are most likely developed by sensible modeling strategies rather than by complex development methods.",
keywords = "Genetic programming, Logistic regression, Penalized maximum likelihood estimation, Prediction model",
author = "Janssen, {Kristel J.M.} and Ivar Siccama and Yvonne Vergouwe and Hendrik Koffijberg and Debray, {Thomas P.A.} and Maarten Keijzer and Grobbee, {Diederick E.} and Moons, {Karel G.M.}",
year = "2012",
month = "4",
day = "1",
doi = "10.1016/j.jclinepi.2011.08.011",
language = "English",
volume = "65",
pages = "404--412",
journal = "Journal of clinical epidemiology",
issn = "0895-4356",
publisher = "Elsevier",
number = "4",

}

Development and validation of clinical prediction models: Marginal differences between logistic regression, penalized maximum likelihood estimation, and genetic programming. / Janssen, Kristel J.M.; Siccama, Ivar; Vergouwe, Yvonne; Koffijberg, Hendrik; Debray, Thomas P.A.; Keijzer, Maarten; Grobbee, Diederick E.; Moons, Karel G.M.

In: Journal of clinical epidemiology, Vol. 65, No. 4, 01.04.2012, p. 404-412.

Research output: Contribution to journalArticleAcademicpeer-review

TY - JOUR

T1 - Development and validation of clinical prediction models: Marginal differences between logistic regression, penalized maximum likelihood estimation, and genetic programming

AU - Janssen, Kristel J.M.

AU - Siccama, Ivar

AU - Vergouwe, Yvonne

AU - Koffijberg, Hendrik

AU - Debray, Thomas P.A.

AU - Keijzer, Maarten

AU - Grobbee, Diederick E.

AU - Moons, Karel G.M.

PY - 2012/4/1

Y1 - 2012/4/1

N2 - Objective: Many prediction models are developed by multivariable logistic regression. However, there are several alternative methods to develop prediction models. We compared the accuracy of a model that predicts the presence of deep venous thrombosis (DVT) when developed by four different methods.Study Design and Setting: We used the data of 2,086 primary care patients suspected of DVT, which included 21 candidate predictors. The cohort was split into a derivation set (1,668 patients, 329 with DVT) and a validation set (418 patients, 86 with DVT). Also, 100 cross-validations were conducted in the full cohort. The models were developed by logistic regression, logistic regression with shrinkage by bootstrapping techniques, logistic regression with shrinkage by penalized maximum likelihood estimation, and genetic programming. The accuracy of the models was tested by assessing discrimination and calibration. Results: There were only marginal differences in the discrimination and calibration of the models in the validation set and cross-validations. Conclusion: The accuracy measures of the models developed by the four different methods were only slightly different, and the 95% confidence intervals were mostly overlapped. We have shown that models with good predictive accuracy are most likely developed by sensible modeling strategies rather than by complex development methods.

AB - Objective: Many prediction models are developed by multivariable logistic regression. However, there are several alternative methods to develop prediction models. We compared the accuracy of a model that predicts the presence of deep venous thrombosis (DVT) when developed by four different methods.Study Design and Setting: We used the data of 2,086 primary care patients suspected of DVT, which included 21 candidate predictors. The cohort was split into a derivation set (1,668 patients, 329 with DVT) and a validation set (418 patients, 86 with DVT). Also, 100 cross-validations were conducted in the full cohort. The models were developed by logistic regression, logistic regression with shrinkage by bootstrapping techniques, logistic regression with shrinkage by penalized maximum likelihood estimation, and genetic programming. The accuracy of the models was tested by assessing discrimination and calibration. Results: There were only marginal differences in the discrimination and calibration of the models in the validation set and cross-validations. Conclusion: The accuracy measures of the models developed by the four different methods were only slightly different, and the 95% confidence intervals were mostly overlapped. We have shown that models with good predictive accuracy are most likely developed by sensible modeling strategies rather than by complex development methods.

KW - Genetic programming

KW - Logistic regression

KW - Penalized maximum likelihood estimation

KW - Prediction model

UR - http://www.scopus.com/inward/record.url?scp=84857651910&partnerID=8YFLogxK

U2 - 10.1016/j.jclinepi.2011.08.011

DO - 10.1016/j.jclinepi.2011.08.011

M3 - Article

VL - 65

SP - 404

EP - 412

JO - Journal of clinical epidemiology

JF - Journal of clinical epidemiology

SN - 0895-4356

IS - 4

ER -