A Multimodal Analysis of Vocal and Visual Backchannels in Spontaneous Dialogs

Khiet Phuong Truong; Ronald Walter Poppe; I.A. de Kok; Dirk K.J. Heylen

A Multimodal Analysis of Vocal and Visual Backchannels in Spontaneous Dialogs

Khiet Phuong Truong, Ronald Walter Poppe, I.A. de Kok, Dirk K.J. Heylen

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

23 Citations (Scopus)

232 Downloads (Pure)

Abstract

Backchannels (BCs) are short vocal and visual listener responses that signal attention, interest, and understanding to the speaker. Previous studies have investigated BC prediction in telephone-style dialogs from prosodic cues. In contrast, we consider spontaneous face-to-face dialogs. The additional visual modality allows speaker and listener to monitor each other's attention continuously, and we hypothesize that this affects the BC-inviting cues. In this study, we investigate how gaze, in addition to prosody, can cue BCs. Moreover, we focus on the type of BC performed, with the aim to find out whether vocal and visual BCs are invited by similar cues. In contrast to telephone-style dialogs, we do not find rising/falling pitch to be a BC-inviting cue. However, in a face-to-face setting, gaze appears to cue BCs. In addition, we find that mutual gaze occurs significantly more often during visual BCs. Moreover, vocal BCs are more likely to be timed during pauses in the speaker's speech.

Original language	English
Title of host publication	Proceedings of Interspeech 2011
Place of Publication	France
Publisher	International Speech Communication Association
Pages	2973-2976
Number of pages	4
Publication status	Published - Aug 2011
Event	12th Annual Conference of the International Speech Communication Association, INTERSPEECH 2011 - Florence, Italy Duration: 28 Aug 2011 → 31 Aug 2011 Conference number: 12

Publication series

Name
Publisher	International Speech Communication Association
ISSN (Print)	1990-9772

Conference

Conference	12th Annual Conference of the International Speech Communication Association, INTERSPEECH 2011
Abbreviated title	INTERSPEECH
Country/Territory	Italy
City	Florence
Period	28/08/11 → 31/08/11

Keywords

METIS-279669
IR-78349
EWI-20721
EC Grant Agreement nr.: FP7/231287

Access to Document

http://www.isca-speech.org/archive/interspeech_2011/i11_2973.html

Cite this

@inproceedings{134571f498344afa85df3d9e996b9210,

title = "A Multimodal Analysis of Vocal and Visual Backchannels in Spontaneous Dialogs",

abstract = "Backchannels (BCs) are short vocal and visual listener responses that signal attention, interest, and understanding to the speaker. Previous studies have investigated BC prediction in telephone-style dialogs from prosodic cues. In contrast, we consider spontaneous face-to-face dialogs. The additional visual modality allows speaker and listener to monitor each other's attention continuously, and we hypothesize that this affects the BC-inviting cues. In this study, we investigate how gaze, in addition to prosody, can cue BCs. Moreover, we focus on the type of BC performed, with the aim to find out whether vocal and visual BCs are invited by similar cues. In contrast to telephone-style dialogs, we do not find rising/falling pitch to be a BC-inviting cue. However, in a face-to-face setting, gaze appears to cue BCs. In addition, we find that mutual gaze occurs significantly more often during visual BCs. Moreover, vocal BCs are more likely to be timed during pauses in the speaker's speech.",

keywords = "METIS-279669, IR-78349, EWI-20721, EC Grant Agreement nr.: FP7/231287",

author = "Truong, {Khiet Phuong} and Poppe, {Ronald Walter} and {de Kok}, I.A. and Heylen, {Dirk K.J.}",

note = "eemcs-eprint-20721 ; 12th Annual Conference of the International Speech Communication Association, INTERSPEECH 2011, INTERSPEECH ; Conference date: 28-08-2011 Through 31-08-2011",

year = "2011",

month = aug,

language = "English",

publisher = "International Speech Communication Association",

pages = "2973--2976",

booktitle = "Proceedings of Interspeech 2011",

address = "France",

}

Truong, KP, Poppe, RW, de Kok, IA & Heylen, DKJ 2011, A Multimodal Analysis of Vocal and Visual Backchannels in Spontaneous Dialogs. in Proceedings of Interspeech 2011. International Speech Communication Association, France, pp. 2973-2976, 12th Annual Conference of the International Speech Communication Association, INTERSPEECH 2011, Florence, Italy, 28/08/11. <http://www.isca-speech.org/archive/interspeech_2011/i11_2973.html>

A Multimodal Analysis of Vocal and Visual Backchannels in Spontaneous Dialogs. / Truong, Khiet Phuong; Poppe, Ronald Walter; de Kok, I.A. et al.
Proceedings of Interspeech 2011. France: International Speech Communication Association, 2011. p. 2973-2976.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

TY - GEN

T1 - A Multimodal Analysis of Vocal and Visual Backchannels in Spontaneous Dialogs

AU - Truong, Khiet Phuong

AU - Poppe, Ronald Walter

AU - de Kok, I.A.

AU - Heylen, Dirk K.J.

N1 - Conference code: 12

PY - 2011/8

Y1 - 2011/8

N2 - Backchannels (BCs) are short vocal and visual listener responses that signal attention, interest, and understanding to the speaker. Previous studies have investigated BC prediction in telephone-style dialogs from prosodic cues. In contrast, we consider spontaneous face-to-face dialogs. The additional visual modality allows speaker and listener to monitor each other's attention continuously, and we hypothesize that this affects the BC-inviting cues. In this study, we investigate how gaze, in addition to prosody, can cue BCs. Moreover, we focus on the type of BC performed, with the aim to find out whether vocal and visual BCs are invited by similar cues. In contrast to telephone-style dialogs, we do not find rising/falling pitch to be a BC-inviting cue. However, in a face-to-face setting, gaze appears to cue BCs. In addition, we find that mutual gaze occurs significantly more often during visual BCs. Moreover, vocal BCs are more likely to be timed during pauses in the speaker's speech.

AB - Backchannels (BCs) are short vocal and visual listener responses that signal attention, interest, and understanding to the speaker. Previous studies have investigated BC prediction in telephone-style dialogs from prosodic cues. In contrast, we consider spontaneous face-to-face dialogs. The additional visual modality allows speaker and listener to monitor each other's attention continuously, and we hypothesize that this affects the BC-inviting cues. In this study, we investigate how gaze, in addition to prosody, can cue BCs. Moreover, we focus on the type of BC performed, with the aim to find out whether vocal and visual BCs are invited by similar cues. In contrast to telephone-style dialogs, we do not find rising/falling pitch to be a BC-inviting cue. However, in a face-to-face setting, gaze appears to cue BCs. In addition, we find that mutual gaze occurs significantly more often during visual BCs. Moreover, vocal BCs are more likely to be timed during pauses in the speaker's speech.

KW - METIS-279669

KW - IR-78349

KW - EWI-20721

KW - EC Grant Agreement nr.: FP7/231287

M3 - Conference contribution

SP - 2973

EP - 2976

BT - Proceedings of Interspeech 2011

PB - International Speech Communication Association

CY - France

T2 - 12th Annual Conference of the International Speech Communication Association, INTERSPEECH 2011

Y2 - 28 August 2011 through 31 August 2011

ER -

A Multimodal Analysis of Vocal and Visual Backchannels in Spontaneous Dialogs

Abstract

Publication series

Conference

Keywords

Access to Document

Fingerprint

Cite this