Dynamic Sparse Training for Deep Reinforcement Learning

Ghada A.Z.N. Sokar; Elena Mocanu; Decebal Constantin Mocanu; Mykola Pechenizkiy; Peter Stone

doi:10.24963/ijcai.2022/477

Dynamic Sparse Training for Deep Reinforcement Learning

Ghada A.Z.N. Sokar, Elena Mocanu, Decebal Constantin Mocanu, Mykola Pechenizkiy, Peter Stone

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

12 Citations (Scopus)

106 Downloads (Pure)

Abstract

Deep reinforcement learning (DRL) agents are trained through trial-and-error interactions with the environment. This leads to a long training time
for dense neural networks to achieve good performance. Hence, prohibitive computation and memory resources are consumed. Recently, learning efficient DRL agents has received increasing attention. Yet, current methods focus on accelerating inference time. In this paper, we introduce for the first time a dynamic sparse training approach for deep reinforcement learning to accelerate the training process. The proposed approach trains a sparse neural network from scratch and dynamically adapts its topology to the changing data distribution during training. Experiments on continuous control tasks show that our dynamic sparse agents achieve higher performance than the equivalent dense methods, reduce the parameter count and floating-point operations (FLOPs) by 50%, and have a faster learning speed that enables reaching the performance of dense agents with 40 − 50% reduction in the training steps.

Original language	English
Title of host publication	Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI 2022
Editors	Luc De Raedt
Pages	3437-3443
Number of pages	7
ISBN (Electronic)	9781956792003
DOIs	https://doi.org/10.24963/ijcai.2022/477
Publication status	Published - Jul 2022
Event	31st International Joint Conference on Artificial Intelligence, IJCAI 2022 - Messe Wien, Vienna, Austria Duration: 23 Jul 2018 → 29 Sept 2022 Conference number: 31 https://ijcai-22.org/

Conference

Conference	31st International Joint Conference on Artificial Intelligence, IJCAI 2022
Abbreviated title	IJCAI 2022
Country/Territory	Austria
City	Vienna
Period	23/07/18 → 29/09/22
Internet address	https://ijcai-22.org/

Keywords

Machine Learning
Deep reinforcement learning
Learning Sparse Models
Representation learning

Access to Document

10.24963/ijcai.2022/477

Sokar-2022-Dynamic-sparse-training-for-deep-reFinal published version, 3.48 MB

Cite this

@inproceedings{b22bac78c5e742d490bb3617291ecc15,

title = "Dynamic Sparse Training for Deep Reinforcement Learning",

abstract = "Deep reinforcement learning (DRL) agents are trained through trial-and-error interactions with the environment. This leads to a long training timefor dense neural networks to achieve good performance. Hence, prohibitive computation and memory resources are consumed. Recently, learning efficient DRL agents has received increasing attention. Yet, current methods focus on accelerating inference time. In this paper, we introduce for the first time a dynamic sparse training approach for deep reinforcement learning to accelerate the training process. The proposed approach trains a sparse neural network from scratch and dynamically adapts its topology to the changing data distribution during training. Experiments on continuous control tasks show that our dynamic sparse agents achieve higher performance than the equivalent dense methods, reduce the parameter count and floating-point operations (FLOPs) by 50%, and have a faster learning speed that enables reaching the performance of dense agents with 40 − 50% reduction in the training steps.",

keywords = "Machine Learning, Deep reinforcement learning, Learning Sparse Models, Representation learning",

author = "Sokar, {Ghada A.Z.N.} and Elena Mocanu and Mocanu, {Decebal Constantin} and Mykola Pechenizkiy and Peter Stone",

year = "2022",

month = jul,

doi = "10.24963/ijcai.2022/477",

language = "English",

pages = "3437--3443",

editor = "{De Raedt}, Luc",

booktitle = "Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI 2022",

note = "31st International Joint Conference on Artificial Intelligence, IJCAI 2022, IJCAI 2022 ; Conference date: 23-07-2018 Through 29-09-2022",

url = "https://ijcai-22.org/",

}

Sokar, GAZN, Mocanu, E, Mocanu, DC, Pechenizkiy, M & Stone, P 2022, Dynamic Sparse Training for Deep Reinforcement Learning. in L De Raedt (ed.), Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI 2022. pp. 3437-3443, 31st International Joint Conference on Artificial Intelligence, IJCAI 2022, Vienna, Austria, 23/07/18. https://doi.org/10.24963/ijcai.2022/477

Dynamic Sparse Training for Deep Reinforcement Learning. / Sokar, Ghada A.Z.N.; Mocanu, Elena; Mocanu, Decebal Constantin et al.
Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI 2022. ed. / Luc De Raedt. 2022. p. 3437-3443.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

TY - GEN

T1 - Dynamic Sparse Training for Deep Reinforcement Learning

AU - Sokar, Ghada A.Z.N.

AU - Mocanu, Elena

AU - Mocanu, Decebal Constantin

AU - Pechenizkiy, Mykola

AU - Stone, Peter

N1 - Conference code: 31

PY - 2022/7

Y1 - 2022/7

N2 - Deep reinforcement learning (DRL) agents are trained through trial-and-error interactions with the environment. This leads to a long training timefor dense neural networks to achieve good performance. Hence, prohibitive computation and memory resources are consumed. Recently, learning efficient DRL agents has received increasing attention. Yet, current methods focus on accelerating inference time. In this paper, we introduce for the first time a dynamic sparse training approach for deep reinforcement learning to accelerate the training process. The proposed approach trains a sparse neural network from scratch and dynamically adapts its topology to the changing data distribution during training. Experiments on continuous control tasks show that our dynamic sparse agents achieve higher performance than the equivalent dense methods, reduce the parameter count and floating-point operations (FLOPs) by 50%, and have a faster learning speed that enables reaching the performance of dense agents with 40 − 50% reduction in the training steps.

AB - Deep reinforcement learning (DRL) agents are trained through trial-and-error interactions with the environment. This leads to a long training timefor dense neural networks to achieve good performance. Hence, prohibitive computation and memory resources are consumed. Recently, learning efficient DRL agents has received increasing attention. Yet, current methods focus on accelerating inference time. In this paper, we introduce for the first time a dynamic sparse training approach for deep reinforcement learning to accelerate the training process. The proposed approach trains a sparse neural network from scratch and dynamically adapts its topology to the changing data distribution during training. Experiments on continuous control tasks show that our dynamic sparse agents achieve higher performance than the equivalent dense methods, reduce the parameter count and floating-point operations (FLOPs) by 50%, and have a faster learning speed that enables reaching the performance of dense agents with 40 − 50% reduction in the training steps.

KW - Machine Learning

KW - Deep reinforcement learning

KW - Learning Sparse Models

KW - Representation learning

UR - https://arxiv.org/abs/2106.04217

UR - https://github.com/GhadaSokar/Dynamic-Sparse-Training-for-Deep-Reinforcement-Learning

U2 - 10.24963/ijcai.2022/477

DO - 10.24963/ijcai.2022/477

M3 - Conference contribution

SP - 3437

EP - 3443

BT - Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI 2022

A2 - De Raedt, Luc

T2 - 31st International Joint Conference on Artificial Intelligence, IJCAI 2022

Y2 - 23 July 2018 through 29 September 2022

ER -

Dynamic Sparse Training for Deep Reinforcement Learning

Abstract

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Best Paper Award at AAMAS 2022 Workshop on Adaptive Learning Agents (ALA 2022)