Modeling hidden dynamics of multimodal cues for spontaneous agreement and disagreement recognition

Konstantinos Bousmalis, Louis–Philippe Morencey, Maja Pantic

    Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

    48 Citations (Scopus)
    17 Downloads (Pure)


    This paper attempts to recognize spontaneous agreement and disagreement based only on nonverbal multi-modal cues. Related work has mainly used verbal and prosodic cues. We demonstrate that it is possible to correctly recognize agreement and disagreement without the use of verbal context (i.e. words, syntax). We propose to explicitly model the complex hidden dynamics of the multimodal cues using a sequential discriminative model, the Hidden Conditional Random Field (HCRF). In this paper, we show that the HCRF model is able to capture what makes each of these social attitudes unique. We present an efficient technique to analyze the concepts learned by the HCRF model and show that these coincide with the findings from social psychology regarding which cues are most prevalent in agreement and disagreement. Our experiments are performed on a spontaneous dataset of real televised debates. The HCRF model outperforms conventional approaches such as Hidden Markov Models and Support Vector Machines.
    Original languageUndefined
    Title of host publicationIEEE International Conference on Automatic Face & Gesture Recognition and Workshops (FG 2011)
    Place of PublicationUSA
    Number of pages7
    ISBN (Print)978-1-4244-9140-7
    Publication statusPublished - Mar 2011
    Event9th IEEE International Conference on Automatic Face and Gesture Recognition, FG 2011 - Santa Barbara, United States
    Duration: 21 Mar 201125 Mar 2011
    Conference number: 9

    Publication series

    PublisherIEEE Computer Society


    Conference9th IEEE International Conference on Automatic Face and Gesture Recognition, FG 2011
    Abbreviated titleFG
    Country/TerritoryUnited States
    CitySanta Barbara


    • METIS-285042
    • IR-79505
    • Eyebrows
    • Analytical models
    • Psychology
    • Speech
    • Training
    • Visualization
    • EC Grant Agreement nr.: FP7/231287
    • EWI-21350
    • Hidden Markov models

    Cite this