The effect of multiple modalities on the perception of a listening agent

Ronald Walter Poppe, Mark ter Maat, Dirk K.J. Heylen

    Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

    15 Downloads (Pure)


    Listening agents are IVAs that display attentive listening behavior to a human speaker. The research into listening agents has mainly focused on (1) automatically timing listener responses; and (2) investigating the perceptual quality of listening behavior. Both issues have predominantly been addressed in an offline fashion, e.g. based on controlled animations that were rated by human observers. This allows for the systematic investigation of variables such as the quantity, type and timing of listening behaviors. However, there is a trade-off between the control and the realism of the stimuli. The display of head movement and facial expressions makes the animated listening behavior more realistic but hinders the investigation of specific behavior such as the timing of a backchannel. To migitate these problems, the Switching Wizard of Oz (SWOZ) framework was introduced in [1]. In online speaker-listener dialogs, a human listener and a behavior synthesis algorithm simultaneously generate backchannel timings. The listening agent is animated based on one of the two sources, which is switched at random time intervals. Speakers are asked to press a button whenever they think the behavior is not human-like. As both human and algorithm have the same limited means of expression, these judgements can solely be based on aspects of the behavior such as the quantity and timing of backchannels. In [1], the listening agent only showed head nods. In the current experiment, we investigate the effect of adding facial expressions. Facial expressions such as smiles and frowns are known to function as backchannels as they can be regarded as a signal of understanding and attention.
    Original languageUndefined
    Title of host publicationProceedings of the International Conference on Intelligent Virtual Agents (IVA 2013)
    Place of PublicationBerlin
    Number of pages2
    ISBN (Print)978-3-642-40414-6
    Publication statusPublished - Aug 2013
    Event13th International Conference on Intelligent Virtual Agents, IVA 2013 - Edinburgh, United Kingdom
    Duration: 29 Aug 201331 Aug 2013
    Conference number: 13

    Publication series

    NameLecture Notes in Computer Science
    PublisherSpringer Verlag
    ISSN (Print)0302-9743
    ISSN (Electronic)1611-3349


    Conference13th International Conference on Intelligent Virtual Agents, IVA 2013
    Abbreviated titleIVA
    Country/TerritoryUnited Kingdom


    • HMI-IA: Intelligent Agents
    • EWI-23574
    • Facial Expression
    • Listening agent
    • METIS-297772
    • IR-87110
    • Backchannels
    • Experiment
    • Modality
    • Virtual listener

    Cite this