A Multi-layer Hybrid Framework for Dimensional Emotion Classification

Mihalis A. Nicolaou, Hatice Gunes, Maja Pantic

    Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

    35 Citations (Scopus)
    7 Downloads (Pure)


    This paper investigates dimensional emotion prediction and classification from naturalistic facial expressions. Similarly to many pattern recognition problems, dimensional emotion classification requires generating multi-dimensional outputs. To date, classification for valence and arousal dimensions has been done separately, assuming that they are independent. However, various psychological findings suggest that these dimensions are correlated. We therefore propose a novel, multi-layer hybrid framework for emotion classification that is able to model inter-dimensional correlations. Firstly, we derive a novel geometric feature set based on the (a)symmetric spatio-temporal characteristics of facial expressions. Subsequently, we use the proposed feature set to train a multi-layer hybrid framework composed of a tem- poral regression layer for predicting emotion dimensions, a graphical model layer for modeling valence-arousal correlations, and a final classification and fusion layer exploiting informative statistics extracted from the lower layers. This framework (i) introduces the Auto-Regressive Coupled HMM (ACHMM), a graphical model specifically tailored to accommodate not only inter-dimensional correlations but also to exploit the internal dynamics of the actual observations, and (ii) replaces the commonly used Maximum Likelihood principle with a more robust final classification and fusion layer. Subject-independent experimental validation, performed on a naturalistic set of facial expressions, demonstrates the effectiveness of the derived feature set, and the robustness and flexibility of the proposed framework.
    Original languageUndefined
    Title of host publicationMM '11 : Proceedings of the 19th ACM International Conference on Multimedia
    Place of PublicationNew York
    PublisherAssociation for Computing Machinery
    Number of pages4
    ISBN (Print)978-1-4503-0616-4
    Publication statusPublished - 28 Nov 2011
    Event19th ACM Multimedia Conference, MM 2011 - Scottsdale, United States
    Duration: 28 Nov 20111 Dec 2011
    Conference number: 19

    Publication series



    Conference19th ACM Multimedia Conference, MM 2011
    Abbreviated titleMM
    Country/TerritoryUnited States
    Internet address


    • METIS-285011
    • Experimentation
    • Classifier design and evaluation
    • IR-79394
    • EWI-21297
    • Feature evaluation and selection
    • EC Grant Agreement nr.: ERC/203143
    • Human Factors

    Cite this