Nonparametric regression using deep neural networks with ReLU activation function

Anselm Johannes Schmidt-Hieber

Research output: Contribution to journalArticleAcademicpeer-review

7 Citations (Scopus)

Abstract

Consider the multivariate nonparametric regression model. It is shown that estimators based on sparsely connected deep neural networks with ReLU activation function and properly chosen network architecture achieve the minimax rates of convergence (up to logn log⁡n -factors) under a general composition assumption on the regression function. The framework includes many well-studied structural constraints such as (generalized) additive models. While there is a lot of flexibility in the network architecture, the tuning parameter is the sparsity of the network. Specifically, we consider large networks with number of potential network parameters exceeding the sample size. The analysis gives some insights into why multilayer feedforward neural networks perform well in practice. Interestingly, for ReLU activation function the depth (number of layers) of the neural network architectures plays an important role, and our theory suggests that for nonparametric regression, scaling the network depth with the sample size is natural. It is also shown that under the composition assumption wavelet estimators can only achieve suboptimal rates.
Original languageEnglish
Pages (from-to)1875-1897
JournalAnnals of statistics
Volume48
Issue number4
DOIs
Publication statusPublished - 1 Aug 2020

Keywords

  • Nonparametric regression
  • multilayer neural networks
  • ReLU activation function
  • minimax estimation risk
  • additive models
  • wavelets

Fingerprint Dive into the research topics of 'Nonparametric regression using deep neural networks with ReLU activation function'. Together they form a unique fingerprint.

Cite this