Correlated-Spaces Regression for Learning Continuous Emotion Dimensions

M. Nicolaou, S. Zafeiriou, Maja Pantic

    Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

    18 Citations (Scopus)
    112 Downloads (Pure)


    Adopting continuous dimensional annotations for affective analysis has been gaining rising attention by researchers over the past years. Due to the idiosyncratic nature of this problem, many subproblems have been identified, spanning from the fusion of multiple continuous annotations to exploiting output-correlations amongst emotion dimensions. In this paper, we firstly empirically answer several important questions which have found partial or no answer at all so far in related literature. In more detail, we study the correlation of each emotion dimension (i) with respect to other emotion dimensions, (ii) to basic emotions (e.g., happiness, anger). As a measure for comparison, we use video and audio features. Interestingly enough, we find that (i) each emotion dimension is more correlated with other emotion dimensions rather than with face and audio features, and similarly (ii) that each basic emotion is more correlated with emotion dimensions than with audio and video features. A similar conclusion holds for discrete emotions which are found to be highly correlated to emotion dimensions as compared to audio and/or video features. Motivated by these findings, we present a novel regression algorithm (Correlated-Spaces Regression, CSR), inspired by Canonical Correlation Analysis (CCA) which learns output-correlations and performs supervised dimensionality reduction and multimodal fusion by (i) projecting features extracted from all modalities and labels onto a common space where their inter-correlation is maximised and (ii) learning mappings from the projected feature space onto the projected, uncorrelated label space.
    Original languageUndefined
    Title of host publicationProceedings of the 21st ACM international conference on Multimedia, MM 2013
    Place of PublicationNew York
    PublisherAssociation for Computing Machinery (ACM)
    Number of pages4
    ISBN (Print)978-1-4503-2404-5
    Publication statusPublished - Oct 2013
    Event21st ACM Multimedia Conference, MM 2013 - Barcelona, Spain
    Duration: 21 Oct 201325 Oct 2013
    Conference number: 21

    Publication series



    Conference21st ACM Multimedia Conference, MM 2013
    Abbreviated titleMM
    Internet address


    • EWI-24345
    • HMI-HF: Human Factors
    • output-correlations
    • valence
    • EC Grant Agreement nr.: FP7/2007-2013
    • EC Grant Agreement nr.: FP7/288235
    • IR-89375
    • Continuous and dimensional emotion descriptions
    • Arousal
    • Component Analysis
    • Multi-modal Fusion
    • Feature Selection
    • METIS-302664
    • EC Grant Agreement nr.: ERC-2007-STG-203143 (MAHNOB)

    Cite this