Benchmarking deep networks for facial emotion recognition in the wild

Antonio Greco*, Nicola Strisciuglio, Mario Vento, Vincenzo Vigilante

*Corresponding author for this work

Research output: Contribution to journalArticleAcademicpeer-review

10 Citations (Scopus)
100 Downloads (Pure)

Abstract

Emotion recognition from face images is a challenging task that gained interest in recent years for its applications to business intelligence and social robotics. Researchers in computer vision and affective computing focused on optimizing the classification error on benchmark data sets, which do not extensively cover possible variations that face images may undergo in real environments. Following on investigations carried out in the field of object recognition, we evaluated the robustness of existing methods for emotion recognition when their input is subjected to corruptions caused by factors present in real-world scenarios. We constructed two data sets on top of the RAF-DB test set, named RAF-DB-C and RAF-DB-P, that contain images modified with 18 types of corruption and 10 of perturbation. We benchmarked existing networks (VGG, DenseNet, SENet and Xception) trained on the original images of RAF-DB and compared them with ARM, the current state-of-the-art method on the RAF-DB test set. We carried out an extensive study on the effects that modifications to the training data or network architecture have on the classification of corrupted and perturbed data. We observed a drop of recognition performance of ARM, with the classification error raising up to 200% of that achieved on the original RAF-DB test set. We demonstrate that the use of the AutoAugment data augmentation and an anti-aliasing filter within down-sampling layers provide existing networks with increased robustness to out-of-distribution variations, substantially reducing the error on corrupted inputs and outperforming ARM. We provide insights about the resilience of existing emotion recognition methods and an estimation of their performance in real scenarios. The processing time required by the modifications we investigated (35 ms in the worst case) supports their suitability for application in real-world scenarios. The RAF-DB-C and RAF-DB-P test sets, trained models and evaluation framework are available at https://github.com/MiviaLab/emotion-robustness.

Original languageEnglish
Pages (from-to)11189-11220
Number of pages32
JournalMultimedia tools and applications
Volume82
Issue number8
Early online date26 Mar 2022
DOIs
Publication statusPublished - Mar 2023

Keywords

  • Affective computing
  • Benchmarking
  • Deep learning
  • Emotion recognition
  • Expression recognition
  • Face analysis
  • Robustness

Fingerprint

Dive into the research topics of 'Benchmarking deep networks for facial emotion recognition in the wild'. Together they form a unique fingerprint.

Cite this