SpaceNet: Make Free Space For Continual Learning (Extended Abstract)

  • Ghada A.Z.N. Sokar*
  • , Decebal Constantin Mocanu
  • , Mykola Pechenizkiy
  • *Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

39 Downloads (Pure)

Abstract

Continual learning aims to build intelligent agents that can continuously learn new tasks over time while preserving the old learned knowledge. Ideally, the agent should continually learn without adding a huge computational and memory overhead to learn a new task or remember the old ones. The main challenge in this paradigm is catastrophically forgetting previous tasks when the model is optimized for a new one. Existing methods mitigate this problem at the expense of increasing the model capacity or replaying the old samples. This hinders its applicability to real-world applications where the old data might not be available and computation and memory efficiency is required. To address these limitations, we proposed SpaceNet a new architectural-based strategy that utilizes the available fixed-capacity of the model efficiently. We harness the significant redundancy of deep neural networks and learn each task in a compact space using dynamic sparse training. SpaceNet learns semi- distributed sparse representation for each task. This representation has two key advantages: (1) it reduces the interference between tasks. (2) It leaves free neurons for future tasks without adding extra computation and memory overhead.
Original languageEnglish
Title of host publicationBNAIC/BENELEARN 2021
Subtitle of host publicationThe 33rd Benelux Conference on Artificial Intelligence and the 30th Belgian Dutch Conference on Machine Learning
Publication statusPublished - 10 Nov 2021
EventBNAIC/BENELEARN 2021: The 33rd Benelux Conference on Artificial Intelligence and the 30th Belgian Dutch Conference on Machine Learning - University of Luxembourg, Belval Campus: Faculty of Science, Technology and Medicine, Esch-sur-Alzette, Luxembourg
Duration: 10 Nov 202112 Nov 2021
https://bnaic2021.uni.lu/

Conference

ConferenceBNAIC/BENELEARN 2021
Country/TerritoryLuxembourg
CityEsch-sur-Alzette
Period10/11/2112/11/21
Internet address

Fingerprint

Dive into the research topics of 'SpaceNet: Make Free Space For Continual Learning (Extended Abstract)'. Together they form a unique fingerprint.
  • SpaceNet: Make Free Space for Continual Learning

    Sokar, G., Mocanu, D. C. & Pechenizkiy, M., 7 Jun 2021, In: Neurocomputing. 439, p. 1-11 11 p.

    Research output: Contribution to journalArticleAcademicpeer-review

    Open Access
    File
    57 Citations (Scopus)
    141 Downloads (Pure)

Cite this