On the expressivity of deep Heaviside networks

Insung Kong; Juntong Chen; Sophie Langer; Johannes Schmidt-Hieber

doi:10.48550/arXiv.2505.00110

On the expressivity of deep Heaviside networks

Insung Kong
, Juntong Chen
, Sophie Langer
, Johannes Schmidt-Hieber

Mathematics of Operations Research

Research output: Working paper › Preprint › Academic

61 Downloads (Pure)

Abstract

We show that deep Heaviside networks (DHNs) have limited expressiveness but that this can be overcome by including either skip connections or neurons with linear activation. We provide lower and upper bounds for the Vapnik-Chervonenkis (VC) dimensions and approximation rates of these network classes. As an application, we derive statistical convergence rates for DHN fits in the nonparametric regression model.

Original language	English
Publisher	ArXiv.org
DOIs	https://doi.org/10.48550/arXiv.2505.00110
Publication status	Published - 30 Apr 2025

Keywords

stat.ML
cs.LG
cs.NA
math.NA

Access to Document

10.48550/arXiv.2505.00110

2505.00110v1Final published version, 631 KB

Cite this

@techreport{85bebe3157e94d3191dfe14a959f37e7,

title = "On the expressivity of deep Heaviside networks",

abstract = " We show that deep Heaviside networks (DHNs) have limited expressiveness but that this can be overcome by including either skip connections or neurons with linear activation. We provide lower and upper bounds for the Vapnik-Chervonenkis (VC) dimensions and approximation rates of these network classes. As an application, we derive statistical convergence rates for DHN fits in the nonparametric regression model. ",

keywords = "stat.ML, cs.LG, cs.NA, math.NA",

author = "Insung Kong and Juntong Chen and Sophie Langer and Johannes Schmidt-Hieber",

note = "61 pages, 16 figures",

year = "2025",

month = apr,

day = "30",

doi = "10.48550/arXiv.2505.00110",

language = "English",

publisher = "ArXiv.org",

type = "WorkingPaper",

institution = "ArXiv.org",

}

TY - UNPB

T1 - On the expressivity of deep Heaviside networks

AU - Kong, Insung

AU - Chen, Juntong

AU - Langer, Sophie

AU - Schmidt-Hieber, Johannes

N1 - 61 pages, 16 figures

PY - 2025/4/30

Y1 - 2025/4/30

N2 - We show that deep Heaviside networks (DHNs) have limited expressiveness but that this can be overcome by including either skip connections or neurons with linear activation. We provide lower and upper bounds for the Vapnik-Chervonenkis (VC) dimensions and approximation rates of these network classes. As an application, we derive statistical convergence rates for DHN fits in the nonparametric regression model.

AB - We show that deep Heaviside networks (DHNs) have limited expressiveness but that this can be overcome by including either skip connections or neurons with linear activation. We provide lower and upper bounds for the Vapnik-Chervonenkis (VC) dimensions and approximation rates of these network classes. As an application, we derive statistical convergence rates for DHN fits in the nonparametric regression model.

KW - stat.ML

KW - cs.LG

KW - cs.NA

KW - math.NA

U2 - 10.48550/arXiv.2505.00110

DO - 10.48550/arXiv.2505.00110

M3 - Preprint

BT - On the expressivity of deep Heaviside networks

PB - ArXiv.org

ER -

On the expressivity of deep Heaviside networks

Abstract

Keywords

Access to Document

Fingerprint

Cite this