Can Less Yield More? Insights into Truly Sparse Training

Qiao Xiao; Boqian Wu; Lu Yin; Maurice van Keulen; Mykola Pechenizkiy

Can Less Yield More? Insights into Truly Sparse Training

Qiao Xiao, Boqian Wu, Lu Yin, Maurice van Keulen, Mykola Pechenizkiy

Research output: Contribution to conference › Poster › Academic

15 Downloads (Pure)

Abstract

Truly dynamic sparse training (T-DST), unlocking the great potential of dynamic sparse training for achieving comparable or even higher accuracy with less resource cost (less yields more), has been a renewed research topic towards green AI. However, certain aspects of T-DST, such as its sensitivity to the dataset, network architecture, and sparsity strategy, are still not well understood. In this paper, we first implement the truly sparse training for Rigged Lottery (RigL) algorithm, then evaluate its “less yields more” hypothesis by demonstrating that “95% fewer parameters and FLOPs yield up to 33% test accuracy improvement” on CIFAR100. Further on, we take broader insights into how the dataset size, the activation function, and weights distribution affect the performance of the neural network with T-DST. Based on the empirical study, we summarize a guideline in order to
exploit “less yields more” in T-DST, hoping to catalyze research progress on the topic of T-DST. Our code will be available online.

Original language	English
Number of pages	7
Publication status	Published - 1 May 2023
Event	ICLR 2023 Workshop on Sparsity in Neural Networks - Kigali, Rwanda Duration: 5 May 2023 → 5 May 2023 https://www.sparseneural.net/

Workshop

Workshop	ICLR 2023 Workshop on Sparsity in Neural Networks
Country/Territory	Rwanda
City	Kigali
Period	5/05/23 → 5/05/23
Internet address	https://www.sparseneural.net/

Access to Document

LYM_poster_SNN - boqian wuFinal published version, 1.18 MB

https://drive.google.com/file/d/1kbWZ9ejU9XvtOMRtAcVYmcoRCDIWj3zy/view

Cite this

@conference{17e9d54b0d8c4b3e87ccb907a4a3a354,

title = "Can Less Yield More? Insights into Truly Sparse Training",

abstract = "Truly dynamic sparse training (T-DST), unlocking the great potential of dynamic sparse training for achieving comparable or even higher accuracy with less resource cost (less yields more), has been a renewed research topic towards green AI. However, certain aspects of T-DST, such as its sensitivity to the dataset, network architecture, and sparsity strategy, are still not well understood. In this paper, we first implement the truly sparse training for Rigged Lottery (RigL) algorithm, then evaluate its “less yields more” hypothesis by demonstrating that “95% fewer parameters and FLOPs yield up to 33% test accuracy improvement” on CIFAR100. Further on, we take broader insights into how the dataset size, the activation function, and weights distribution affect the performance of the neural network with T-DST. Based on the empirical study, we summarize a guideline in order toexploit “less yields more” in T-DST, hoping to catalyze research progress on the topic of T-DST. Our code will be available online.",

author = "Qiao Xiao and Boqian Wu and Lu Yin and {van Keulen}, Maurice and Mykola Pechenizkiy",

year = "2023",

month = may,

day = "1",

language = "English",

note = "ICLR 2023 Workshop on Sparsity in Neural Networks ; Conference date: 05-05-2023 Through 05-05-2023",

url = "https://www.sparseneural.net/",

}

TY - CONF

T1 - Can Less Yield More? Insights into Truly Sparse Training

AU - Xiao, Qiao

AU - Wu, Boqian

AU - Yin, Lu

AU - van Keulen, Maurice

AU - Pechenizkiy, Mykola

PY - 2023/5/1

Y1 - 2023/5/1

N2 - Truly dynamic sparse training (T-DST), unlocking the great potential of dynamic sparse training for achieving comparable or even higher accuracy with less resource cost (less yields more), has been a renewed research topic towards green AI. However, certain aspects of T-DST, such as its sensitivity to the dataset, network architecture, and sparsity strategy, are still not well understood. In this paper, we first implement the truly sparse training for Rigged Lottery (RigL) algorithm, then evaluate its “less yields more” hypothesis by demonstrating that “95% fewer parameters and FLOPs yield up to 33% test accuracy improvement” on CIFAR100. Further on, we take broader insights into how the dataset size, the activation function, and weights distribution affect the performance of the neural network with T-DST. Based on the empirical study, we summarize a guideline in order toexploit “less yields more” in T-DST, hoping to catalyze research progress on the topic of T-DST. Our code will be available online.

AB - Truly dynamic sparse training (T-DST), unlocking the great potential of dynamic sparse training for achieving comparable or even higher accuracy with less resource cost (less yields more), has been a renewed research topic towards green AI. However, certain aspects of T-DST, such as its sensitivity to the dataset, network architecture, and sparsity strategy, are still not well understood. In this paper, we first implement the truly sparse training for Rigged Lottery (RigL) algorithm, then evaluate its “less yields more” hypothesis by demonstrating that “95% fewer parameters and FLOPs yield up to 33% test accuracy improvement” on CIFAR100. Further on, we take broader insights into how the dataset size, the activation function, and weights distribution affect the performance of the neural network with T-DST. Based on the empirical study, we summarize a guideline in order toexploit “less yields more” in T-DST, hoping to catalyze research progress on the topic of T-DST. Our code will be available online.

M3 - Poster

T2 - ICLR 2023 Workshop on Sparsity in Neural Networks

Y2 - 5 May 2023 through 5 May 2023

ER -

Can Less Yield More? Insights into Truly Sparse Training

Abstract

Workshop

Access to Document

Fingerprint

Cite this