Predicting Listener Backchannels: A Probabilistic Multimodal Approach

Louis-Philippe Morency; I.A. de Kok; Jonathan Gratch

doi:10.1007/978-3-540-85483-8_18

Predicting Listener Backchannels: A Probabilistic Multimodal Approach

Louis-Philippe Morency, I.A. de Kok, Jonathan Gratch

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

61 Citations (Scopus)

131 Downloads (Pure)

Abstract

During face-to-face interactions, listeners use backchannel feedback such as head nods as a signal to the speaker that the communication is working and that they should continue speaking. Predicting these backchannel opportunities is an important milestone for building engaging and natural virtual humans. In this paper we show how sequential probabilistic models (e.g., Hidden Markov Model (HMM) or Conditional Random Fields (CRF)) can automatically learn from a database of human-to-human interactions to predict listener backchannels using the speaker multimodal output features (e.g., prosody, spoken words and eye gaze). The main challenges addressed in this paper are automatic selection of the relevant features and optimal feature representation for probabilistic models. For prediction of visual backchannel cues (i.e., head nods), our prediction model shows a statistically significant improvement over a previously published approach based on hand-crafted rules.

Original language	Undefined
Title of host publication	Proceedings of the Eight International Conference on Intelligent Virtual Agents 2008
Editors	Helmut Prendinger, James Lester, Mitsuru Ishizuka
Place of Publication	Berlin
Publisher	Springer
Pages	176-190
Number of pages	15
ISBN (Print)	978-3-540-85482-1
DOIs	https://doi.org/10.1007/978-3-540-85483-8_18
Publication status	Published - 2008
Event	8th International Conference on Intelligent Virtual Agents, IVA 2008 - Tokyo, Japan Duration: 1 Sep 2008 → 3 Sep 2008 Conference number: 8

Publication series

Name	Lecture Notes in Computer Science
Publisher	Springer Verlag
Volume	5208/2008

Conference

Conference	8th International Conference on Intelligent Virtual Agents, IVA 2008
Abbreviated title	IVA
Country/Territory	Japan
City	Tokyo
Period	1/09/08 → 3/09/08

Keywords

METIS-264251
IR-68949
HMI-HF: Human Factors
HMI-IA: Intelligent Agents
EWI-17025

Access to Document

10.1007/978-3-540-85483-8_18

morency_2008_predicting

http://eprints.eemcs.utwente.nl/secure2/17025/01/morency_2008_predicting.pdf

Cite this

Morency, L-P., de Kok, I. A., & Gratch, J. (2008). Predicting Listener Backchannels: A Probabilistic Multimodal Approach. In H. Prendinger, J. Lester, & M. Ishizuka (Eds.), Proceedings of the Eight International Conference on Intelligent Virtual Agents 2008 (pp. 176-190). [10.1007/978-3-540-85483-8_18] (Lecture Notes in Computer Science; Vol. 5208/2008). Springer. https://doi.org/10.1007/978-3-540-85483-8_18

@inproceedings{1310761d379f47d6b0f42194ff7b42d2,

title = "Predicting Listener Backchannels: A Probabilistic Multimodal Approach",

abstract = "During face-to-face interactions, listeners use backchannel feedback such as head nods as a signal to the speaker that the communication is working and that they should continue speaking. Predicting these backchannel opportunities is an important milestone for building engaging and natural virtual humans. In this paper we show how sequential probabilistic models (e.g., Hidden Markov Model (HMM) or Conditional Random Fields (CRF)) can automatically learn from a database of human-to-human interactions to predict listener backchannels using the speaker multimodal output features (e.g., prosody, spoken words and eye gaze). The main challenges addressed in this paper are automatic selection of the relevant features and optimal feature representation for probabilistic models. For prediction of visual backchannel cues (i.e., head nods), our prediction model shows a statistically significant improvement over a previously published approach based on hand-crafted rules.",

keywords = "METIS-264251, IR-68949, HMI-HF: Human Factors, HMI-IA: Intelligent Agents, EWI-17025",

author = "Louis-Philippe Morency and {de Kok}, I.A. and Jonathan Gratch",

note = "10.1007/978-3-540-85483-8_18 ; null ; Conference date: 01-09-2008 Through 03-09-2008",

year = "2008",

doi = "10.1007/978-3-540-85483-8_18",

language = "Undefined",

isbn = "978-3-540-85482-1",

series = "Lecture Notes in Computer Science",

publisher = "Springer",

pages = "176--190",

editor = "Helmut Prendinger and James Lester and Mitsuru Ishizuka",

booktitle = "Proceedings of the Eight International Conference on Intelligent Virtual Agents 2008",

address = "Netherlands",

}

Morency, L-P, de Kok, IA & Gratch, J 2008, Predicting Listener Backchannels: A Probabilistic Multimodal Approach. in H Prendinger, J Lester & M Ishizuka (eds), Proceedings of the Eight International Conference on Intelligent Virtual Agents 2008., 10.1007/978-3-540-85483-8_18, Lecture Notes in Computer Science, vol. 5208/2008, Springer, Berlin, pp. 176-190, 8th International Conference on Intelligent Virtual Agents, IVA 2008, Tokyo, Japan, 1/09/08. https://doi.org/10.1007/978-3-540-85483-8_18

Predicting Listener Backchannels: A Probabilistic Multimodal Approach. / Morency, Louis-Philippe; de Kok, I.A.; Gratch, Jonathan.

Proceedings of the Eight International Conference on Intelligent Virtual Agents 2008. ed. / Helmut Prendinger; James Lester; Mitsuru Ishizuka. Berlin : Springer, 2008. p. 176-190 10.1007/978-3-540-85483-8_18 (Lecture Notes in Computer Science; Vol. 5208/2008).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

TY - GEN

T1 - Predicting Listener Backchannels: A Probabilistic Multimodal Approach

AU - Morency, Louis-Philippe

AU - de Kok, I.A.

AU - Gratch, Jonathan

N1 - Conference code: 8

PY - 2008

Y1 - 2008

N2 - During face-to-face interactions, listeners use backchannel feedback such as head nods as a signal to the speaker that the communication is working and that they should continue speaking. Predicting these backchannel opportunities is an important milestone for building engaging and natural virtual humans. In this paper we show how sequential probabilistic models (e.g., Hidden Markov Model (HMM) or Conditional Random Fields (CRF)) can automatically learn from a database of human-to-human interactions to predict listener backchannels using the speaker multimodal output features (e.g., prosody, spoken words and eye gaze). The main challenges addressed in this paper are automatic selection of the relevant features and optimal feature representation for probabilistic models. For prediction of visual backchannel cues (i.e., head nods), our prediction model shows a statistically significant improvement over a previously published approach based on hand-crafted rules.

AB - During face-to-face interactions, listeners use backchannel feedback such as head nods as a signal to the speaker that the communication is working and that they should continue speaking. Predicting these backchannel opportunities is an important milestone for building engaging and natural virtual humans. In this paper we show how sequential probabilistic models (e.g., Hidden Markov Model (HMM) or Conditional Random Fields (CRF)) can automatically learn from a database of human-to-human interactions to predict listener backchannels using the speaker multimodal output features (e.g., prosody, spoken words and eye gaze). The main challenges addressed in this paper are automatic selection of the relevant features and optimal feature representation for probabilistic models. For prediction of visual backchannel cues (i.e., head nods), our prediction model shows a statistically significant improvement over a previously published approach based on hand-crafted rules.

KW - METIS-264251

KW - IR-68949

KW - HMI-HF: Human Factors

KW - HMI-IA: Intelligent Agents

KW - EWI-17025

U2 - 10.1007/978-3-540-85483-8_18

DO - 10.1007/978-3-540-85483-8_18

M3 - Conference contribution

SN - 978-3-540-85482-1

T3 - Lecture Notes in Computer Science

SP - 176

EP - 190

BT - Proceedings of the Eight International Conference on Intelligent Virtual Agents 2008

A2 - Prendinger, Helmut

A2 - Lester, James

A2 - Ishizuka, Mitsuru

PB - Springer

CY - Berlin

Y2 - 1 September 2008 through 3 September 2008

ER -

Morency L-P, de Kok IA, Gratch J. Predicting Listener Backchannels: A Probabilistic Multimodal Approach. In Prendinger H, Lester J, Ishizuka M, editors, Proceedings of the Eight International Conference on Intelligent Virtual Agents 2008. Berlin: Springer. 2008. p. 176-190. 10.1007/978-3-540-85483-8_18. (Lecture Notes in Computer Science). https://doi.org/10.1007/978-3-540-85483-8_18