Audio-visual Classification and Fusion of Spontaneous Affect Data in Likelihood Space

Mihalis A. Nicolaou, Hatice Gunes, Maja Pantic

    Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

    53 Citations (Scopus)
    122 Downloads (Pure)


    This paper focuses on audio-visual (using facial expression, shoulder and audio cues) classification of spontaneous affect, utilising generative models for classification (i) in terms of Maximum Likelihood Classification with the assumption that the generative model structure in the classifier is correct, and (ii) Likelihood Space Classification with the assumption that the generative model structure in the classifier may be incorrect, and therefore, the classification performance can be improved by projecting the results of generative classifiers onto likelihood space, and then using discriminative classifiers. Experiments are conducted by utilising Hidden Markov Models for single cue classification, and 2 and 3-chain coupled Hidden Markov Models for fusing multiple cues and modalities. For discriminative classification, we utilise Support Vector Machines. Results show that Likelihood Space Classification improves the performance (91.76%) of Maximum Likelihood Classification (79.1%). Thereafter, we introduce the concept of fusion in the likelihood space, which is shown to outperform the typically used model-level fusion, attaining a classification accuracy of 94.01% and further improving all previous results.
    Original languageUndefined
    Title of host publication20th International Conference on Pattern Recognition, ICPR 2010
    Place of PublicationUSA
    PublisherIEEE Computer Society
    Number of pages5
    ISBN (Print)978-0-7695-4109-9
    Publication statusPublished - 26 Aug 2010
    Event20th International Conference on Pattern Recognition 2010 - Istanbul Convention & Exhibition Centre, Istanbul, Turkey
    Duration: 23 Aug 201026 Aug 2010
    Conference number: 20

    Publication series

    PublisherIEEE Computer Society


    Conference20th International Conference on Pattern Recognition 2010
    Abbreviated titleICPR 2010
    Internet address


    • METIS-276353
    • IR-75937
    • EWI-19536
    • EC Grant Agreement nr.: FP7/211486

    Cite this