## Abstract

It is a common phenomenon that for high-dimensional and nonparametric

statistical models, rate-optimal estimators balance squared bias and variance.

Although this balancing is widely observed, little is known whether

methods exist that could avoid the trade-off between bias and variance. We

propose a general strategy to obtain lower bounds on the variance of any estimator

with bias smaller than a prespecified bound. This shows to which

extent the bias-variance trade-off is unavoidable and allows to quantify the

loss of performance for methods that do not obey it. The approach is based

on a number of abstract lower bounds for the variance involving the change

of expectation with respect to different probability measures as well as information

measures such as the Kullback–Leibler or χ2-divergence. Some of

these inequalities rely on a new concept of information matrices. In a second

part of the article, the abstract lower bounds are applied to several statistical

models including the Gaussian white noise model, a boundary estimation

problem, the Gaussian sequence model and the high-dimensional linear regression

model. For these specific statistical applications, different types of

bias-variance trade-offs occur that vary considerably in their strength. For

the trade-off between integrated squared bias and integrated variance in the

Gaussian white noise model, we propose to combine the general strategy for

lower bounds with a reduction technique. This allows us to reduce the original

problem to a lower bound on the bias-variance trade-off for estimators with

additional symmetry properties in a simpler statistical model. In the Gaussian

sequence model, different phase transitions of the bias-variance trade-off occur.

Although there is a non-trivial interplay between bias and variance, the

rate of the squared bias and the variance do not have to be balanced in order

to achieve the minimax estimation rate.

statistical models, rate-optimal estimators balance squared bias and variance.

Although this balancing is widely observed, little is known whether

methods exist that could avoid the trade-off between bias and variance. We

propose a general strategy to obtain lower bounds on the variance of any estimator

with bias smaller than a prespecified bound. This shows to which

extent the bias-variance trade-off is unavoidable and allows to quantify the

loss of performance for methods that do not obey it. The approach is based

on a number of abstract lower bounds for the variance involving the change

of expectation with respect to different probability measures as well as information

measures such as the Kullback–Leibler or χ2-divergence. Some of

these inequalities rely on a new concept of information matrices. In a second

part of the article, the abstract lower bounds are applied to several statistical

models including the Gaussian white noise model, a boundary estimation

problem, the Gaussian sequence model and the high-dimensional linear regression

model. For these specific statistical applications, different types of

bias-variance trade-offs occur that vary considerably in their strength. For

the trade-off between integrated squared bias and integrated variance in the

Gaussian white noise model, we propose to combine the general strategy for

lower bounds with a reduction technique. This allows us to reduce the original

problem to a lower bound on the bias-variance trade-off for estimators with

additional symmetry properties in a simpler statistical model. In the Gaussian

sequence model, different phase transitions of the bias-variance trade-off occur.

Although there is a non-trivial interplay between bias and variance, the

rate of the squared bias and the variance do not have to be balanced in order

to achieve the minimax estimation rate.

Original language | English |
---|---|

Pages (from-to) | 1510 - 1533 |

Number of pages | 24 |

Journal | Annals of the Institute of Statistical Mathematics |

Volume | 51 |

Issue number | 4 |

DOIs | |

Publication status | Published - Aug 2023 |

## Keywords

- Bias-variance decomposition
- Cramér–Rao inequality
- high-dimensional statistics
- minimax estimation
- nonparametric estimation