Continuous emotion detection using EEG signals and facial expressions

Mohammad Soleymani, Sadjad Asghari-Esfeden, Maja Pantic, Yun Fu

    Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

    52 Citations (Scopus)
    456 Downloads (Pure)

    Abstract

    Emotions play an important role in how we select and consume multimedia. Recent advances on affect detection are focused on detecting emotions continuously. In this paper, for the first time, we continuously detect valence from electroencephalogram (EEG) signals and facial expressions in response to videos. Multiple annotators provided valence levels continuously by watching the frontal facial videos of participants who watched short emotional videos. Power spectral features from EEG signals as well as facial fiducial points are used as features to detect valence levels for each frame continuously. We study the correlation between features from EEG and facial expressions with continuous valence. We have also verified our model's performance for the emotional highlight detection using emotion recognition from EEG signals. Finally the results of multimodal fusion between facial expression and EEG signals are presented. Having such models we will be able to detect spontaneous and subtle affective responses over time and use them for video highlight detection.
    Original languageUndefined
    Title of host publicationProceedings of the IEEE International Conference on Multimedia and Expo (ICME 2014)
    Place of PublicationUSA
    PublisherIEEE Computer Society
    Pages1-6
    Number of pages6
    ISBN (Print)978-1-4799-4761-4
    DOIs
    Publication statusPublished - Jul 2014

    Publication series

    Name
    PublisherIEEE Computer Society

    Keywords

    • EWI-25827
    • HMI-HF: Human Factors
    • video highlight detection
    • Implicit Tagging
    • METIS-309952
    • EEG
    • IR-95233
    • Affect
    • Facial expressions

    Cite this

    Soleymani, M., Asghari-Esfeden, S., Pantic, M., & Fu, Y. (2014). Continuous emotion detection using EEG signals and facial expressions. In Proceedings of the IEEE International Conference on Multimedia and Expo (ICME 2014) (pp. 1-6). USA: IEEE Computer Society. https://doi.org/10.1109/ICME.2014.6890301