Self-Attention Meta-Learner for Continual Learning

Ghada A.Z.N. Sokar, Decebal Constantin Mocanu, Mykola Pechenizkiy

Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

24 Downloads (Pure)


Continual learning aims to provide intelligent agents capable of learning multiple tasks sequentially with neural networks. One of its main challenging, catastrophic forgetting, is caused by the neural networks non-optimal ability to learn in non-stationary distributions. In most settings of the current approaches, the agent starts from randomly initialized parameters and is optimized to master the current task regardless of the usefulness of the learned representation for future tasks. Moreover, each of the future tasks uses all the previously learned knowledge although parts of this knowledge might not be helpful for its learning. These cause interference among tasks, especially when the data of previous tasks is not accessible. In this paper, we propose a new method, named Self-Attention Meta-Learner (SAM)1, which learns a prior knowledge for continual learning that permits learning a sequence of tasks, while avoiding catastrophic forgetting. SAM incorporates an attention mechanism that learns to select the particular relevant representation for each future task. Each task builds a specific representation branch on the top of the selected knowledge, avoiding the interference between tasks. We evaluate the proposed method on the Split CIFAR-10/100 and Split MNIST benchmarks in the task agnostic inference. We empirically show that we can achieve a better performance than several state-of-the-art methods for continual learning by building on the top of selected representation learned by SAM.We also show the role of the meta-attention mechanism in boosting informative features corresponding to the input data and identifying the correct target in the task agnostic inference. Finally, we demonstrate that popular existing continual learning methods gain a performance boost when they adopt SAM as a starting point.
Original languageEnglish
Title of host publicationAAMAS '21: Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems
Publication statusPublished - 3 May 2021
Event20th International Conference on Autonomous Agents and Multiagent Systems, AAMAS 2021 - Virtual Event, United Kingdom
Duration: 3 May 20217 May 2021
Conference number: 20


Conference20th International Conference on Autonomous Agents and Multiagent Systems, AAMAS 2021
Abbreviated titleAAMAS
Country/TerritoryUnited Kingdom
CityVirtual Event
Internet address


Dive into the research topics of 'Self-Attention Meta-Learner for Continual Learning'. Together they form a unique fingerprint.

Cite this