Abstract
Generalizing over temporal variations is a prerequisite for effective action recognition in videos. Despite
significant advances in deep neural networks, it remains a challenge to focus on short-term discriminative
motions in relation to the overall performance of an action. We address this challenge by allowing some
flexibility in discovering relevant spatio-temporal features. We introduce Squeeze and Recursion Temporal
Gates (SRTG), an approach that favors inputs with similar activations with potential temporal variations.
We implement this idea with a novel CNN block that uses an LSTM to encapsulate feature dynamics,
in conjunction with a temporal gate that is responsible for evaluating the consistency of the discovered
dynamics and the modeled features. We show consistent improvement when using SRTG blocks, with
only a minimal increase in the number of GFLOPs. On Kinetics-700, we perform on par with current
state-of-the-art models, and outperform these on HACS, Moments in Time, UCF-101 and HMDB-51.1
significant advances in deep neural networks, it remains a challenge to focus on short-term discriminative
motions in relation to the overall performance of an action. We address this challenge by allowing some
flexibility in discovering relevant spatio-temporal features. We introduce Squeeze and Recursion Temporal
Gates (SRTG), an approach that favors inputs with similar activations with potential temporal variations.
We implement this idea with a novel CNN block that uses an LSTM to encapsulate feature dynamics,
in conjunction with a temporal gate that is responsible for evaluating the consistency of the discovered
dynamics and the modeled features. We show consistent improvement when using SRTG blocks, with
only a minimal increase in the number of GFLOPs. On Kinetics-700, we perform on par with current
state-of-the-art models, and outperform these on HACS, Moments in Time, UCF-101 and HMDB-51.1
Original language | English |
---|---|
Pages (from-to) | 1-7 |
Number of pages | 7 |
Journal | Pattern recognition letters |
Volume | 141 |
DOIs | |
Publication status | Published - Nov 2020 |
Externally published | Yes |