Continuous emotion detection using EEG signals and facial expressions

Mohammad Soleymani, Sadjad Asghari-Esfeden, Maja Pantic, Yun Fu

    Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

    78 Citations (Scopus)
    764 Downloads (Pure)


    Emotions play an important role in how we select and consume multimedia. Recent advances on affect detection are focused on detecting emotions continuously. In this paper, for the first time, we continuously detect valence from electroencephalogram (EEG) signals and facial expressions in response to videos. Multiple annotators provided valence levels continuously by watching the frontal facial videos of participants who watched short emotional videos. Power spectral features from EEG signals as well as facial fiducial points are used as features to detect valence levels for each frame continuously. We study the correlation between features from EEG and facial expressions with continuous valence. We have also verified our model's performance for the emotional highlight detection using emotion recognition from EEG signals. Finally the results of multimodal fusion between facial expression and EEG signals are presented. Having such models we will be able to detect spontaneous and subtle affective responses over time and use them for video highlight detection.
    Original languageUndefined
    Title of host publicationProceedings of the IEEE International Conference on Multimedia and Expo (ICME 2014)
    Place of PublicationUSA
    Number of pages6
    ISBN (Print)978-1-4799-4761-4
    Publication statusPublished - Jul 2014
    EventIEEE International Conference on Multimedia and Expo, ICME 2014 - Chengdu, China
    Duration: 14 Jul 201418 Jul 2014

    Publication series

    PublisherIEEE Computer Society


    ConferenceIEEE International Conference on Multimedia and Expo, ICME 2014
    Other14-18 July 2014


    • EWI-25827
    • HMI-HF: Human Factors
    • video highlight detection
    • Implicit Tagging
    • METIS-309952
    • EEG
    • IR-95233
    • Affect
    • Facial expressions

    Cite this