The Unreasonable Effectiveness of Random Pruning: Return of the Most Naive Baseline for Sparse Training

Shiwei Liu*, Tianlong Chen, Xiaohan Chen, Li Shen, Decebal Constantin Mocanu, Zhangyang Wang, Mykola Pechenizkiy

*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

666 Downloads (Pure)

Abstract

Random pruning is arguably the most naive way to attain sparsity in neural networks, but has been deemed uncompetitive by either post-training pruning or sparse training. In this paper, we focus on sparse training and highlight a perhaps counter intuitive finding, that random pruning at initialization can be quite powerful for the sparse training of modern neural networks. Without any delicate pruning criteria or carefully pursued sparsity structures, we empirically demonstrate that sparsely training a randomly pruned network from scratch can match the performance of its dense equivalent. There are two key factors that contribute to this revival: (i) the network sizes matter: as the original dense networks grow wider and deeper, the performance of training a randomly pruned sparse network will quickly grow to matching that of its dense equivalent, even at high sparsity ratios; (ii) appropriate layer-wise sparsity ratios can be pre-chosen for sparse training, which shows to be another important performance booster. Simple as it looks, a randomly pruned subnetwork of Wide ResNet-50 can be sparsely trained to outperforming a dense Wide ResNet-50, on ImageNet. We also observed such randomly pruned networks outperform dense counterparts in other favorable aspects, such as out-of-distribution detection, uncertainty estimation, and adversarial robustness. Overall, our results strongly suggest there is larger-than-expected room for sparse training at scale, and the benefits of sparsity might be more universal beyond carefully designed pruning. Our source code can be found at this https URL.
Original languageEnglish
Title of host publicationThe Tenth International Conference on Learning Representations, ICLR 2022
PublisherOpenReview
Number of pages21
Publication statusPublished - 20 Jan 2022
Event10th International Conference on Learning Representations, ICLR 2022 - Virtual
Duration: 25 Apr 202229 Apr 2022
Conference number: 10
https://iclr.cc/

Conference

Conference10th International Conference on Learning Representations, ICLR 2022
Abbreviated titleICLR
Period25/04/2229/04/22
Internet address

Fingerprint

Dive into the research topics of 'The Unreasonable Effectiveness of Random Pruning: Return of the Most Naive Baseline for Sparse Training'. Together they form a unique fingerprint.

Cite this