32nd International Joint Conferences on Artificial Intelligence, IJCAI 2023

Description

Organising a IJCAI 2023 tutorial on "Sparse Training for Supervised, Unsupervised, Continual, and Deep Reinforcement Learning with Deep Neural Networks"

Abstract:
Motivated by the success of GPT-3, a trillion parameters model race appears to be taking shape, drawing in more technological giants with significant investment. In concert with the increasingly strong results, the resources required to train and deploy those massive models are prohibitive. While sparse neural networks have been widely used to substantially reduce the computational demands of inference, researchers have recently started to investigate techniques to train intrinsically sparse neural networks from scratch to accelerate training (sparse training). As a relatively new avenue, sparse training receives upsurging attention and quickly evolves as a universal approach that has demonstrated strong results in a wide variety of architectures. This tutorial aims to give a comprehensive discussion of sparsity in neural networks training. We first revisit the existing approaches to obtain sparse neural networks from the perspective of the accuracy-efficiency trade-off. Then we look deeper into the performance of sparse neural networks training for different machine learning paradigms, including supervised learning, unsupervised learning, and reinforcement learning. We look to both, single task and continual learning. Finally, we point out the current challenges of sparse neural networks training in scale and promising future directions.

Period	21 Aug 2023
Event type	Conference
Conference number	32
Location	Macao, ChinaShow on map
Degree of Recognition	International