Abstract
We investigate the statistical behavior of gradient descent iterates with dropout in the linear regression model. In particular, non-asymptotic bounds for the convergence of expectations and covariance matrices of the iterates are derived. The results shed more light on the widely cited connection between dropout and ℓ2-regularization in the linear model. We indicate a more subtle relationship, owing to interactions between the gradient descent dynamics and the additional randomness induced by dropout. Further, we study a simplified variant of dropout which does not have a regularizing effect and converges to the least squares estimator.
Original language | English |
---|---|
Article number | 204 |
Pages (from-to) | 1-48 |
Journal | Journal of machine learning research |
Volume | 25 |
Publication status | Published - 1 Jul 2024 |