Approximating smooth functions by deep neural networks with sigmoid activation function

Sophie Langer*

*Corresponding author for this work

Research output: Contribution to journalArticleAcademicpeer-review

66 Citations (Scopus)

Abstract

We study the power of deep neural networks (DNNs) with sigmoid activation function. Recently, it was shown that DNNs approximate any d-dimensional, smooth function on a compact set with a rate of order W −p∕d, where W is the number of nonzero weights in the network and p is the smoothness of the function. Unfortunately, these rates only hold for a special class of sparsely connected DNNs. We ask ourselves if we can show the same approximation rate for a simpler and more general class, i.e., DNNs which are only defined by its width and depth. In this article we show that DNNs with fixed depth and a width of order M d achieve an approximation rate of M −2p. As a conclusion we quantitatively characterize the approximation power of DNNs in terms of the overall weights W 0 in the network and show an approximation rate of W 0 −p∕d. This more general result finally helps us to understand which network topology guarantees a special target accuracy.

Original languageEnglish
Article number104696
Number of pages21
JournalJournal of multivariate analysis
Volume182
Issue numberC
Early online date10 Nov 2020
DOIs
Publication statusPublished - Mar 2021
Externally publishedYes

Fingerprint

Dive into the research topics of 'Approximating smooth functions by deep neural networks with sigmoid activation function'. Together they form a unique fingerprint.

Cite this