On lower bounds for the bias-variance trade-off

Alexis Derumigny, Johannes Schmidt-Hieber

Research output: Working paper

2 Downloads (Pure)

Abstract

It is a common phenomenon that for high-dimensional and nonparametric statistical models, rate-optimal estimators balance squared bias and variance. Although this balancing is widely observed, little is known whether methods exist that could avoid the trade-off between bias and variance. We propose a general strategy to obtain lower bounds on the variance of any estimator with bias smaller than a prespecified bound. This shows to which extent the bias-variance trade-off is unavoidable and allows to quantify the loss of performance for methods that do not obey it. The approach is based on a number of abstract lower bounds for the variance involving the change of expectation with respect to different probability measures as well as information measures such as the Kullback-Leibler or chi-square divergence. Some of these inequalities rely on a new concept of information matrices. In a second part of the article, the abstract lower bounds are applied to several statistical models including the Gaussian white noise model, a boundary estimation problem, the Gaussian sequence model and the high-dimensional linear regression model. For these specific statistical applications, different types of bias-variance trade-offs occur that vary considerably in their strength. For the trade-off between integrated squared bias and integrated variance in the Gaussian white noise model, we propose to combine the general strategy for lower bounds with a reduction technique. This allows us to reduce the original problem to a lower bound on the bias-variance trade-off for estimators with additional symmetry properties in a simpler statistical model. To highlight possible extensions of the proposed framework, we moreover briefly discuss the trade-off between bias and mean absolute deviation.
Original languageEnglish
Place of PublicationIthaca, NY
PublisherarXiv.org
Publication statusPublished - 30 May 2020

Keywords

  • Bias-variance decomposition
  • Cramér-Rao inequality
  • high-dimensional statistics
  • Minimax estimation
  • Mean absolute deviation
  • Nonparametric estimation

Fingerprint Dive into the research topics of 'On lower bounds for the bias-variance trade-off'. Together they form a unique fingerprint.

  • Cite this

    Derumigny, A., & Schmidt-Hieber, J. (2020). On lower bounds for the bias-variance trade-off. Ithaca, NY: arXiv.org.