Self-Attention Meta-Learner for Continual Learning

Ghada A.Z.N. Sokar, Decebal Constantin Mocanu, Mykola Pechenizkiy

Research output: Contribution to conferencePaperAcademicpeer-review

Abstract

Continual learning aims to provide intelligent agents capable of learning multiple tasks sequentially with neural networks. One of its main challenging, catastrophic forgetting, is caused by the neural networks non-optimal ability to learn in non-stationary distributions. In most settings of the current approaches, the agent starts from randomly initialized parameters and is optimized to master the current task regardless of the usefulness of the learned representation for future tasks. Moreover, each of the future tasks uses all the previously learned knowledge although parts of this knowledge might not be helpful for its learning. These cause interference among tasks, especially when the data of previous tasks is not accessible. In this paper, we propose a new method, named Self-Attention Meta-Learner (SAM)1, which learns a prior knowledge for continual learning that permits learning a sequence of tasks, while avoiding catastrophic forgetting. SAM incorporates an attention mechanism that learns to select the particular relevant representation for each future task. Each task builds a specific representation branch on the top of the selected knowledge, avoiding the interference between tasks. We evaluate the proposed method on the Split CIFAR-10/100 and Split MNIST benchmarks in the task agnostic inference. We empirically show that we can achieve a better performance than several state-of-the-art methods for continual learning by building on the top of selected representation learned by SAM.We also show the role of the meta-attention mechanism in boosting informative features corresponding to the input data and identifying the correct target in the task agnostic inference. Finally, we demonstrate that popular existing continual learning methods gain a performance boost when they adopt SAM as a starting point.
Original languageEnglish
Publication statusAccepted/In press - 17 Dec 2020
Event20th International Conference on Autonomous Agents and Multiagent Systems, AAMAS 2021 - London, United Kingdom
Duration: 3 May 20217 May 2021
Conference number: 20
https://aamas2021.soton.ac.uk/

Conference

Conference20th International Conference on Autonomous Agents and Multiagent Systems, AAMAS 2021
Abbreviated titleAAMAS
CountryUnited Kingdom
CityLondon
Period3/05/217/05/21
Internet address

Fingerprint Dive into the research topics of 'Self-Attention Meta-Learner for Continual Learning'. Together they form a unique fingerprint.

Cite this