The effect of multiple modalities on the perception of a listening agent

Ronald Walter Poppe, Mark ter Maat, Dirk K.J. Heylen

Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

6 Downloads (Pure)

Abstract

Listening agents are IVAs that display attentive listening behavior to a human speaker. The research into listening agents has mainly focused on (1) automatically timing listener responses; and (2) investigating the perceptual quality of listening behavior. Both issues have predominantly been addressed in an offline fashion, e.g. based on controlled animations that were rated by human observers. This allows for the systematic investigation of variables such as the quantity, type and timing of listening behaviors. However, there is a trade-off between the control and the realism of the stimuli. The display of head movement and facial expressions makes the animated listening behavior more realistic but hinders the investigation of specific behavior such as the timing of a backchannel. To migitate these problems, the Switching Wizard of Oz (SWOZ) framework was introduced in [1]. In online speaker-listener dialogs, a human listener and a behavior synthesis algorithm simultaneously generate backchannel timings. The listening agent is animated based on one of the two sources, which is switched at random time intervals. Speakers are asked to press a button whenever they think the behavior is not human-like. As both human and algorithm have the same limited means of expression, these judgements can solely be based on aspects of the behavior such as the quantity and timing of backchannels. In [1], the listening agent only showed head nods. In the current experiment, we investigate the effect of adding facial expressions. Facial expressions such as smiles and frowns are known to function as backchannels as they can be regarded as a signal of understanding and attention.
Original languageUndefined
Title of host publicationProceedings of the International Conference on Intelligent Virtual Agents (IVA 2013)
Place of PublicationBerlin
PublisherSpringer
Pages470-471
Number of pages2
ISBN (Print)978-3-642-40414-6
Publication statusPublished - Aug 2013
Event13th International Conference on Intelligent Virtual Agents, IVA 2013 - Edinburgh, United Kingdom
Duration: 29 Aug 201331 Aug 2013
Conference number: 13

Publication series

NameLecture Notes in Computer Science
PublisherSpringer Verlag
Volume8108
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Conference

Conference13th International Conference on Intelligent Virtual Agents, IVA 2013
Abbreviated titleIVA
CountryUnited Kingdom
CityEdinburgh
Period29/08/1331/08/13

Keywords

  • HMI-IA: Intelligent Agents
  • HMI-MI: MULTIMODAL INTERACTIONS
  • EWI-23574
  • Facial Expression
  • Listening agent
  • METIS-297772
  • IR-87110
  • Backchannels
  • Experiment
  • Modality
  • Virtual listener

Cite this

Poppe, R. W., ter Maat, M., & Heylen, D. K. J. (2013). The effect of multiple modalities on the perception of a listening agent. In Proceedings of the International Conference on Intelligent Virtual Agents (IVA 2013) (pp. 470-471). (Lecture Notes in Computer Science; Vol. 8108). Berlin: Springer.
Poppe, Ronald Walter ; ter Maat, Mark ; Heylen, Dirk K.J. / The effect of multiple modalities on the perception of a listening agent. Proceedings of the International Conference on Intelligent Virtual Agents (IVA 2013). Berlin : Springer, 2013. pp. 470-471 (Lecture Notes in Computer Science).
@inproceedings{178824f4d57443df9d1a45ac8294aa6c,
title = "The effect of multiple modalities on the perception of a listening agent",
abstract = "Listening agents are IVAs that display attentive listening behavior to a human speaker. The research into listening agents has mainly focused on (1) automatically timing listener responses; and (2) investigating the perceptual quality of listening behavior. Both issues have predominantly been addressed in an offline fashion, e.g. based on controlled animations that were rated by human observers. This allows for the systematic investigation of variables such as the quantity, type and timing of listening behaviors. However, there is a trade-off between the control and the realism of the stimuli. The display of head movement and facial expressions makes the animated listening behavior more realistic but hinders the investigation of specific behavior such as the timing of a backchannel. To migitate these problems, the Switching Wizard of Oz (SWOZ) framework was introduced in [1]. In online speaker-listener dialogs, a human listener and a behavior synthesis algorithm simultaneously generate backchannel timings. The listening agent is animated based on one of the two sources, which is switched at random time intervals. Speakers are asked to press a button whenever they think the behavior is not human-like. As both human and algorithm have the same limited means of expression, these judgements can solely be based on aspects of the behavior such as the quantity and timing of backchannels. In [1], the listening agent only showed head nods. In the current experiment, we investigate the effect of adding facial expressions. Facial expressions such as smiles and frowns are known to function as backchannels as they can be regarded as a signal of understanding and attention.",
keywords = "HMI-IA: Intelligent Agents, HMI-MI: MULTIMODAL INTERACTIONS, EWI-23574, Facial Expression, Listening agent, METIS-297772, IR-87110, Backchannels, Experiment, Modality, Virtual listener",
author = "Poppe, {Ronald Walter} and {ter Maat}, Mark and Heylen, {Dirk K.J.}",
year = "2013",
month = "8",
language = "Undefined",
isbn = "978-3-642-40414-6",
series = "Lecture Notes in Computer Science",
publisher = "Springer",
pages = "470--471",
booktitle = "Proceedings of the International Conference on Intelligent Virtual Agents (IVA 2013)",

}

Poppe, RW, ter Maat, M & Heylen, DKJ 2013, The effect of multiple modalities on the perception of a listening agent. in Proceedings of the International Conference on Intelligent Virtual Agents (IVA 2013). Lecture Notes in Computer Science, vol. 8108, Springer, Berlin, pp. 470-471, 13th International Conference on Intelligent Virtual Agents, IVA 2013, Edinburgh, United Kingdom, 29/08/13.

The effect of multiple modalities on the perception of a listening agent. / Poppe, Ronald Walter; ter Maat, Mark; Heylen, Dirk K.J.

Proceedings of the International Conference on Intelligent Virtual Agents (IVA 2013). Berlin : Springer, 2013. p. 470-471 (Lecture Notes in Computer Science; Vol. 8108).

Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

TY - GEN

T1 - The effect of multiple modalities on the perception of a listening agent

AU - Poppe, Ronald Walter

AU - ter Maat, Mark

AU - Heylen, Dirk K.J.

PY - 2013/8

Y1 - 2013/8

N2 - Listening agents are IVAs that display attentive listening behavior to a human speaker. The research into listening agents has mainly focused on (1) automatically timing listener responses; and (2) investigating the perceptual quality of listening behavior. Both issues have predominantly been addressed in an offline fashion, e.g. based on controlled animations that were rated by human observers. This allows for the systematic investigation of variables such as the quantity, type and timing of listening behaviors. However, there is a trade-off between the control and the realism of the stimuli. The display of head movement and facial expressions makes the animated listening behavior more realistic but hinders the investigation of specific behavior such as the timing of a backchannel. To migitate these problems, the Switching Wizard of Oz (SWOZ) framework was introduced in [1]. In online speaker-listener dialogs, a human listener and a behavior synthesis algorithm simultaneously generate backchannel timings. The listening agent is animated based on one of the two sources, which is switched at random time intervals. Speakers are asked to press a button whenever they think the behavior is not human-like. As both human and algorithm have the same limited means of expression, these judgements can solely be based on aspects of the behavior such as the quantity and timing of backchannels. In [1], the listening agent only showed head nods. In the current experiment, we investigate the effect of adding facial expressions. Facial expressions such as smiles and frowns are known to function as backchannels as they can be regarded as a signal of understanding and attention.

AB - Listening agents are IVAs that display attentive listening behavior to a human speaker. The research into listening agents has mainly focused on (1) automatically timing listener responses; and (2) investigating the perceptual quality of listening behavior. Both issues have predominantly been addressed in an offline fashion, e.g. based on controlled animations that were rated by human observers. This allows for the systematic investigation of variables such as the quantity, type and timing of listening behaviors. However, there is a trade-off between the control and the realism of the stimuli. The display of head movement and facial expressions makes the animated listening behavior more realistic but hinders the investigation of specific behavior such as the timing of a backchannel. To migitate these problems, the Switching Wizard of Oz (SWOZ) framework was introduced in [1]. In online speaker-listener dialogs, a human listener and a behavior synthesis algorithm simultaneously generate backchannel timings. The listening agent is animated based on one of the two sources, which is switched at random time intervals. Speakers are asked to press a button whenever they think the behavior is not human-like. As both human and algorithm have the same limited means of expression, these judgements can solely be based on aspects of the behavior such as the quantity and timing of backchannels. In [1], the listening agent only showed head nods. In the current experiment, we investigate the effect of adding facial expressions. Facial expressions such as smiles and frowns are known to function as backchannels as they can be regarded as a signal of understanding and attention.

KW - HMI-IA: Intelligent Agents

KW - HMI-MI: MULTIMODAL INTERACTIONS

KW - EWI-23574

KW - Facial Expression

KW - Listening agent

KW - METIS-297772

KW - IR-87110

KW - Backchannels

KW - Experiment

KW - Modality

KW - Virtual listener

M3 - Conference contribution

SN - 978-3-642-40414-6

T3 - Lecture Notes in Computer Science

SP - 470

EP - 471

BT - Proceedings of the International Conference on Intelligent Virtual Agents (IVA 2013)

PB - Springer

CY - Berlin

ER -

Poppe RW, ter Maat M, Heylen DKJ. The effect of multiple modalities on the perception of a listening agent. In Proceedings of the International Conference on Intelligent Virtual Agents (IVA 2013). Berlin: Springer. 2013. p. 470-471. (Lecture Notes in Computer Science).