SpaceNet: Make Free Space For Continual Learning (Extended Abstract)

Continual learning aims to build intelligent agents that can continuously learn new tasks over time while preserving the old learned knowledge. Ideally, the agent should continually learn without adding a huge computational and memory overhead to learn a new task or remember the old ones. The main challenge in this paradigm is catastrophically forgetting previous tasks when the model is optimized for a new one. Existing methods mitigate this problem at the expense of increasing the model capacity or replaying the old samples. This hinders its applicability to real-world applications where the old data might not be available and computation and memory efficiency is required. To address these limitations, we proposed SpaceNet a new architectural-based strategy that utilizes the available fixed-capacity of the model efficiently. We harness the significant redundancy of deep neural networks and learn each task in a compact space using dynamic sparse training. SpaceNet learns semi- distributed sparse representation for each task. This representation has two key advantages: (1) it reduces the interference between tasks. (2) It leaves free neurons for future tasks without adding extra computation and memory overhead.