InstaIndoor and multi-modal deep learning for indoor scene recognition

Andreea Glavan*, Estefanía Talavera

*Corresponding author for this work

Research output: Contribution to journalArticleAcademicpeer-review

9 Citations (Scopus)
96 Downloads (Pure)

Abstract

Indoor scene recognition is a growing field with great potential for behaviour understanding, robot localization, and elderly monitoring, among others. In this study, we approach the task of scene recognition from a novel standpoint, using multi-modal learning and video data gathered from social media. The accessibility and variety of social media videos can provide realistic data for modern scene recognition techniques and applications. We propose a model based on fusion of transcribed speech to text and visual features, which is used for classification on a novel dataset of social media videos of indoor scenes named InstaIndoor. Our model achieves up to 70% accuracy and 0.7 F1-Score. Furthermore, we highlight the potential of our approach by benchmarking on a YouTube-8M subset of indoor scenes as well, where it achieves 74% accuracy and 0.74 F1-Score. We hope the contributions of this work pave the way to novel research in the challenging field of indoor scene recognition.
Original languageEnglish
Pages (from-to)6861-6877
Number of pages17
JournalNeural Computing and Applications
Volume34
Early online date22 Jan 2022
DOIs
Publication statusPublished - May 2022
Externally publishedYes

Fingerprint

Dive into the research topics of 'InstaIndoor and multi-modal deep learning for indoor scene recognition'. Together they form a unique fingerprint.

Cite this