Modeling hidden dynamics of multimodal cues for spontaneous agreement and disagreement recognition

Konstantinos Bousmalis, Louis–Philippe Morencey, Maja Pantic

    Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

    41 Citations (Scopus)

    Abstract

    This paper attempts to recognize spontaneous agreement and disagreement based only on nonverbal multi-modal cues. Related work has mainly used verbal and prosodic cues. We demonstrate that it is possible to correctly recognize agreement and disagreement without the use of verbal context (i.e. words, syntax). We propose to explicitly model the complex hidden dynamics of the multimodal cues using a sequential discriminative model, the Hidden Conditional Random Field (HCRF). In this paper, we show that the HCRF model is able to capture what makes each of these social attitudes unique. We present an efficient technique to analyze the concepts learned by the HCRF model and show that these coincide with the findings from social psychology regarding which cues are most prevalent in agreement and disagreement. Our experiments are performed on a spontaneous dataset of real televised debates. The HCRF model outperforms conventional approaches such as Hidden Markov Models and Support Vector Machines.
    Original languageUndefined
    Title of host publicationIEEE International Conference on Automatic Face & Gesture Recognition and Workshops (FG 2011)
    Place of PublicationUSA
    PublisherIEEE Computer Society
    Pages746-752
    Number of pages7
    ISBN (Print)978-1-4244-9140-7
    DOIs
    Publication statusPublished - Mar 2011
    Event9th IEEE International Conference on Automatic Face and Gesture Recognition, FG 2011 - Santa Barbara, United States
    Duration: 21 Mar 201125 Mar 2011
    Conference number: 9

    Publication series

    Name
    PublisherIEEE Computer Society

    Conference

    Conference9th IEEE International Conference on Automatic Face and Gesture Recognition, FG 2011
    Abbreviated titleFG
    CountryUnited States
    CitySanta Barbara
    Period21/03/1125/03/11

    Keywords

    • METIS-285042
    • IR-79505
    • Eyebrows
    • Analytical models
    • Psychology
    • HMI-MI: MULTIMODAL INTERACTIONS
    • Speech
    • Training
    • Visualization
    • EC Grant Agreement nr.: FP7/231287
    • EWI-21350
    • Hidden Markov models

    Cite this

    Bousmalis, K., Morencey, LP., & Pantic, M. (2011). Modeling hidden dynamics of multimodal cues for spontaneous agreement and disagreement recognition. In IEEE International Conference on Automatic Face & Gesture Recognition and Workshops (FG 2011) (pp. 746-752). USA: IEEE Computer Society. https://doi.org/10.1109/FG.2011.5771341