A Survey on Datasets for Emotion Recognition from Vision: Limitations and In-the-Wild Applicability

Willams Costa*, Estefanía Talavera, Renato Oliveira, Lucas Figueiredo, João Marcelo Teixeira, João Paulo Lima, Veronica Teichrieb

*Corresponding author for this work

Research output: Contribution to journalArticleAcademicpeer-review

1 Citation (Scopus)
6 Downloads (Pure)


Emotion recognition is the task of identifying and understanding human emotions from data. In the field of computer vision, there is a growing interest due to the wide range of possible applications in smart cities, health, marketing, and surveillance, among others. To date, several datasets have been proposed to allow techniques to be trained, validated, and finally deployed to production. However, these techniques have several limitations related to the construction of these datasets. In this work, we survey the datasets currently employed in state-of-the-art emotion recognition, to list and discuss their applicability and limitations in real-world scenarios. We propose experiments on the data to extract essential insights related to the provided visual information in each dataset and discuss how they impact the training and validation of techniques. We also investigate the presence of nonverbal cues in the datasets and propose experiments regarding their representativeness, visibility, and data quality. Among other discussions, we show that EMOTIC has more diverse context representations than CAER, however, with conflicting annotations. Finally, we discuss application scenarios and how techniques to approach them could leverage these datasets, suggesting approaches based on findings from these datasets to help guide future research and deployment. With this work we expect to provide a roadmap for upcoming research and experimentation in emotion recognition under real-world conditions.
Original languageEnglish
Article number5697
JournalApplied Sciences
Issue number9
Early online date5 May 2023
Publication statusPublished - May 2023


Dive into the research topics of 'A Survey on Datasets for Emotion Recognition from Vision: Limitations and In-the-Wild Applicability'. Together they form a unique fingerprint.

Cite this