Modeling hidden dynamics of multimodal cues for spontaneous agreement and disagreement recognition

Konstantinos Bousmalis, Louis–Philippe Morencey, Maja Pantic

Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

41 Citations (Scopus)

Abstract

This paper attempts to recognize spontaneous agreement and disagreement based only on nonverbal multi-modal cues. Related work has mainly used verbal and prosodic cues. We demonstrate that it is possible to correctly recognize agreement and disagreement without the use of verbal context (i.e. words, syntax). We propose to explicitly model the complex hidden dynamics of the multimodal cues using a sequential discriminative model, the Hidden Conditional Random Field (HCRF). In this paper, we show that the HCRF model is able to capture what makes each of these social attitudes unique. We present an efficient technique to analyze the concepts learned by the HCRF model and show that these coincide with the findings from social psychology regarding which cues are most prevalent in agreement and disagreement. Our experiments are performed on a spontaneous dataset of real televised debates. The HCRF model outperforms conventional approaches such as Hidden Markov Models and Support Vector Machines.
Original languageUndefined
Title of host publicationIEEE International Conference on Automatic Face & Gesture Recognition and Workshops (FG 2011)
Place of PublicationUSA
PublisherIEEE Computer Society
Pages746-752
Number of pages7
ISBN (Print)978-1-4244-9140-7
DOIs
Publication statusPublished - Mar 2011
Event9th IEEE International Conference on Automatic Face and Gesture Recognition, FG 2011 - Santa Barbara, United States
Duration: 21 Mar 201125 Mar 2011
Conference number: 9

Publication series

Name
PublisherIEEE Computer Society

Conference

Conference9th IEEE International Conference on Automatic Face and Gesture Recognition, FG 2011
Abbreviated titleFG
CountryUnited States
CitySanta Barbara
Period21/03/1125/03/11

Keywords

  • METIS-285042
  • IR-79505
  • Eyebrows
  • Analytical models
  • Psychology
  • HMI-MI: MULTIMODAL INTERACTIONS
  • Speech
  • Training
  • Visualization
  • EC Grant Agreement nr.: FP7/231287
  • EWI-21350
  • Hidden Markov models

Cite this

Bousmalis, K., Morencey, LP., & Pantic, M. (2011). Modeling hidden dynamics of multimodal cues for spontaneous agreement and disagreement recognition. In IEEE International Conference on Automatic Face & Gesture Recognition and Workshops (FG 2011) (pp. 746-752). USA: IEEE Computer Society. https://doi.org/10.1109/FG.2011.5771341
Bousmalis, Konstantinos ; Morencey, Louis–Philippe ; Pantic, Maja. / Modeling hidden dynamics of multimodal cues for spontaneous agreement and disagreement recognition. IEEE International Conference on Automatic Face & Gesture Recognition and Workshops (FG 2011). USA : IEEE Computer Society, 2011. pp. 746-752
@inproceedings{13d746253e974b6ea8ea6b102a4e8d3e,
title = "Modeling hidden dynamics of multimodal cues for spontaneous agreement and disagreement recognition",
abstract = "This paper attempts to recognize spontaneous agreement and disagreement based only on nonverbal multi-modal cues. Related work has mainly used verbal and prosodic cues. We demonstrate that it is possible to correctly recognize agreement and disagreement without the use of verbal context (i.e. words, syntax). We propose to explicitly model the complex hidden dynamics of the multimodal cues using a sequential discriminative model, the Hidden Conditional Random Field (HCRF). In this paper, we show that the HCRF model is able to capture what makes each of these social attitudes unique. We present an efficient technique to analyze the concepts learned by the HCRF model and show that these coincide with the findings from social psychology regarding which cues are most prevalent in agreement and disagreement. Our experiments are performed on a spontaneous dataset of real televised debates. The HCRF model outperforms conventional approaches such as Hidden Markov Models and Support Vector Machines.",
keywords = "METIS-285042, IR-79505, Eyebrows, Analytical models, Psychology, HMI-MI: MULTIMODAL INTERACTIONS, Speech, Training, Visualization, EC Grant Agreement nr.: FP7/231287, EWI-21350, Hidden Markov models",
author = "Konstantinos Bousmalis and Louis–Philippe Morencey and Maja Pantic",
note = "10.1109/FG.2011.5771341",
year = "2011",
month = "3",
doi = "10.1109/FG.2011.5771341",
language = "Undefined",
isbn = "978-1-4244-9140-7",
publisher = "IEEE Computer Society",
pages = "746--752",
booktitle = "IEEE International Conference on Automatic Face & Gesture Recognition and Workshops (FG 2011)",
address = "United States",

}

Bousmalis, K, Morencey, LP & Pantic, M 2011, Modeling hidden dynamics of multimodal cues for spontaneous agreement and disagreement recognition. in IEEE International Conference on Automatic Face & Gesture Recognition and Workshops (FG 2011). IEEE Computer Society, USA, pp. 746-752, 9th IEEE International Conference on Automatic Face and Gesture Recognition, FG 2011, Santa Barbara, United States, 21/03/11. https://doi.org/10.1109/FG.2011.5771341

Modeling hidden dynamics of multimodal cues for spontaneous agreement and disagreement recognition. / Bousmalis, Konstantinos; Morencey, Louis–Philippe; Pantic, Maja.

IEEE International Conference on Automatic Face & Gesture Recognition and Workshops (FG 2011). USA : IEEE Computer Society, 2011. p. 746-752.

Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

TY - GEN

T1 - Modeling hidden dynamics of multimodal cues for spontaneous agreement and disagreement recognition

AU - Bousmalis, Konstantinos

AU - Morencey, Louis–Philippe

AU - Pantic, Maja

N1 - 10.1109/FG.2011.5771341

PY - 2011/3

Y1 - 2011/3

N2 - This paper attempts to recognize spontaneous agreement and disagreement based only on nonverbal multi-modal cues. Related work has mainly used verbal and prosodic cues. We demonstrate that it is possible to correctly recognize agreement and disagreement without the use of verbal context (i.e. words, syntax). We propose to explicitly model the complex hidden dynamics of the multimodal cues using a sequential discriminative model, the Hidden Conditional Random Field (HCRF). In this paper, we show that the HCRF model is able to capture what makes each of these social attitudes unique. We present an efficient technique to analyze the concepts learned by the HCRF model and show that these coincide with the findings from social psychology regarding which cues are most prevalent in agreement and disagreement. Our experiments are performed on a spontaneous dataset of real televised debates. The HCRF model outperforms conventional approaches such as Hidden Markov Models and Support Vector Machines.

AB - This paper attempts to recognize spontaneous agreement and disagreement based only on nonverbal multi-modal cues. Related work has mainly used verbal and prosodic cues. We demonstrate that it is possible to correctly recognize agreement and disagreement without the use of verbal context (i.e. words, syntax). We propose to explicitly model the complex hidden dynamics of the multimodal cues using a sequential discriminative model, the Hidden Conditional Random Field (HCRF). In this paper, we show that the HCRF model is able to capture what makes each of these social attitudes unique. We present an efficient technique to analyze the concepts learned by the HCRF model and show that these coincide with the findings from social psychology regarding which cues are most prevalent in agreement and disagreement. Our experiments are performed on a spontaneous dataset of real televised debates. The HCRF model outperforms conventional approaches such as Hidden Markov Models and Support Vector Machines.

KW - METIS-285042

KW - IR-79505

KW - Eyebrows

KW - Analytical models

KW - Psychology

KW - HMI-MI: MULTIMODAL INTERACTIONS

KW - Speech

KW - Training

KW - Visualization

KW - EC Grant Agreement nr.: FP7/231287

KW - EWI-21350

KW - Hidden Markov models

U2 - 10.1109/FG.2011.5771341

DO - 10.1109/FG.2011.5771341

M3 - Conference contribution

SN - 978-1-4244-9140-7

SP - 746

EP - 752

BT - IEEE International Conference on Automatic Face & Gesture Recognition and Workshops (FG 2011)

PB - IEEE Computer Society

CY - USA

ER -

Bousmalis K, Morencey LP, Pantic M. Modeling hidden dynamics of multimodal cues for spontaneous agreement and disagreement recognition. In IEEE International Conference on Automatic Face & Gesture Recognition and Workshops (FG 2011). USA: IEEE Computer Society. 2011. p. 746-752 https://doi.org/10.1109/FG.2011.5771341