Dropout Regularization Versus l2-Penalization in the Linear Model

Gabriel Clara*, Sophie Langer, Johannes Schmidt-Hieber

*Corresponding author for this work

Research output: Contribution to journalArticleAcademicpeer-review

7 Downloads (Pure)

Abstract

We investigate the statistical behavior of gradient descent iterates with dropout in the linear regression model. In particular, non-asymptotic bounds for the convergence of expectations and covariance matrices of the iterates are derived. The results shed more light on the widely cited connection between dropout and ℓ2-regularization in the linear model. We indicate a more subtle relationship, owing to interactions between the gradient descent dynamics and the additional randomness induced by dropout. Further, we study a simplified variant of dropout which does not have a regularizing effect and converges to the least squares estimator.
Original languageEnglish
Article number204
Pages (from-to)1-48
JournalJournal of machine learning research
Volume25
Publication statusPublished - 1 Jul 2024

Fingerprint

Dive into the research topics of 'Dropout Regularization Versus l2-Penalization in the Linear Model'. Together they form a unique fingerprint.

Cite this