SpaceNet: Make Free Space for Continual Learning

Ghada Sokar*, Decebal Constantin Mocanu, Mykola Pechenizkiy

*Corresponding author for this work

Research output: Contribution to journalArticleAcademicpeer-review

2 Downloads (Pure)

Abstract

The continual learning (CL) paradigm aims to enable neural networks to learn tasks continually in a sequential fashion. The fundamental challenge in this learning paradigm is catastrophic forgetting previously learned tasks when the model is optimized for a new task, especially when their data is not accessible. Current architectural-based methods aim at alleviating the catastrophic forgetting problem but at the expense of expanding the capacity of the model. Regularization-based methods maintain a fixed model capacity; however, previous studies showed the huge performance degradation of these methods when the task identity is not available during inference (e.g. class incremental learning scenario). In this work, we propose a novel architectural-based method referred as SpaceNet1 for class incremental learning scenario where we utilize the available fixed capacity of the model intelligently. SpaceNet trains sparse deep neural networks from scratch in an adaptive way that compresses the sparse connections of each task in a compact number of neurons. The adaptive training of the sparse connections results in sparse representations that reduce the interference between the tasks. Experimental results show the robustness of our proposed method against catastrophic forgetting old tasks and the efficiency of SpaceNet in utilizing the available capacity of the model, leaving space for more tasks to be learned. In particular, when SpaceNet is tested on the well-known benchmarks for CL: split MNIST, split Fashion-MNIST, CIFAR-10/100, and iCIFAR100, it outperforms regularization-based methods by a big performance gap. Moreover, it achieves better performance than architectural-based methods without model expansion and achieves comparable results with rehearsal-based methods, while offering a huge memory reduction.

Original languageEnglish
Pages (from-to)1-11
Number of pages11
JournalNeurocomputing
Volume439
Early online date26 Jan 2021
DOIs
Publication statusE-pub ahead of print/First online - 26 Jan 2021

Keywords

  • Class incremental learning
  • Continual learning
  • Deep neural networks
  • Lifelong learning
  • Sparse training

Fingerprint Dive into the research topics of 'SpaceNet: Make Free Space for Continual Learning'. Together they form a unique fingerprint.

Cite this