Building Autonomous Sensitive Artificial Listeners

Marc Schroeder, Elisabetta Bevacqua, Roddy Cowie, Florian Eyben, Hatice Gunes, Dirk K.J. Heylen, Mark ter Maat, Gary McKeown, Sathish Pammi, Maja Pantic, Catherine Pelachaud, Björn Schuller, Etienne de Sevin, Michel Valstar, Martin Wöllmer

    Research output: Contribution to journalArticleAcademicpeer-review

    106 Citations (Scopus)

    Abstract

    This paper describes a substantial effort to build a real-time interactive multimodal dialogue system with a focus on emotional and nonverbal interaction capabilities. The work is motivated by the aim to provide technology with competences in perceiving and producing the emotional and nonverbal behaviors required to sustain a conversational dialogue. We present the Sensitive Artificial Listener (SAL) scenario as a setting which seems particularly suited for the study of emotional and nonverbal behavior since it requires only very limited verbal understanding on the part of the machine. This scenario allows us to concentrate on nonverbal capabilities without having to address at the same time the challenges of spoken language understanding, task modeling, etc. We first report on three prototype versions of the SAL scenario in which the behavior of the Sensitive Artificial Listener characters was determined by a human operator. These prototypes served the purpose of verifying the effectiveness of the SAL scenario and allowed us to collect data required for building system components for analyzing and synthesizing the respective behaviors. We then describe the fully autonomous integrated real-time system we created, which combines incremental analysis of user behavior, dialogue management, and synthesis of speaker and listener behavior of a SAL character displayed as a virtual agent. We discuss principles that should underlie the evaluation of SAL-type systems. Since the system is designed for modularity and reuse and since it is publicly available, the SAL system has potential as a joint research tool in the affective computing research community.
    Original languageUndefined
    Pages (from-to)165-183
    Number of pages19
    JournalIEEE transactions on affective computing
    Volume3
    Issue number2
    DOIs
    Publication statusPublished - Apr 2012

    Keywords

    • HMI-MI: MULTIMODAL INTERACTIONS
    • listener behavior
    • real-time dialogue
    • EWI-22932
    • Embodied Conversational Agents
    • IR-84214
    • Turn Taking
    • Rapport agents
    • emotion synthesis
    • METIS-296236
    • Emotion Recognition

    Cite this

    Schroeder, M., Bevacqua, E., Cowie, R., Eyben, F., Gunes, H., Heylen, D. K. J., ... Wöllmer, M. (2012). Building Autonomous Sensitive Artificial Listeners. IEEE transactions on affective computing, 3(2), 165-183. https://doi.org/10.1109/T-AFFC.2011.34
    Schroeder, Marc ; Bevacqua, Elisabetta ; Cowie, Roddy ; Eyben, Florian ; Gunes, Hatice ; Heylen, Dirk K.J. ; ter Maat, Mark ; McKeown, Gary ; Pammi, Sathish ; Pantic, Maja ; Pelachaud, Catherine ; Schuller, Björn ; de Sevin, Etienne ; Valstar, Michel ; Wöllmer, Martin. / Building Autonomous Sensitive Artificial Listeners. In: IEEE transactions on affective computing. 2012 ; Vol. 3, No. 2. pp. 165-183.
    @article{056c2c15c78141839654353097807f3e,
    title = "Building Autonomous Sensitive Artificial Listeners",
    abstract = "This paper describes a substantial effort to build a real-time interactive multimodal dialogue system with a focus on emotional and nonverbal interaction capabilities. The work is motivated by the aim to provide technology with competences in perceiving and producing the emotional and nonverbal behaviors required to sustain a conversational dialogue. We present the Sensitive Artificial Listener (SAL) scenario as a setting which seems particularly suited for the study of emotional and nonverbal behavior since it requires only very limited verbal understanding on the part of the machine. This scenario allows us to concentrate on nonverbal capabilities without having to address at the same time the challenges of spoken language understanding, task modeling, etc. We first report on three prototype versions of the SAL scenario in which the behavior of the Sensitive Artificial Listener characters was determined by a human operator. These prototypes served the purpose of verifying the effectiveness of the SAL scenario and allowed us to collect data required for building system components for analyzing and synthesizing the respective behaviors. We then describe the fully autonomous integrated real-time system we created, which combines incremental analysis of user behavior, dialogue management, and synthesis of speaker and listener behavior of a SAL character displayed as a virtual agent. We discuss principles that should underlie the evaluation of SAL-type systems. Since the system is designed for modularity and reuse and since it is publicly available, the SAL system has potential as a joint research tool in the affective computing research community.",
    keywords = "HMI-MI: MULTIMODAL INTERACTIONS, listener behavior, real-time dialogue, EWI-22932, Embodied Conversational Agents, IR-84214, Turn Taking, Rapport agents, emotion synthesis, METIS-296236, Emotion Recognition",
    author = "Marc Schroeder and Elisabetta Bevacqua and Roddy Cowie and Florian Eyben and Hatice Gunes and Heylen, {Dirk K.J.} and {ter Maat}, Mark and Gary McKeown and Sathish Pammi and Maja Pantic and Catherine Pelachaud and Bj{\"o}rn Schuller and {de Sevin}, Etienne and Michel Valstar and Martin W{\"o}llmer",
    note = "eemcs-eprint-22932",
    year = "2012",
    month = "4",
    doi = "10.1109/T-AFFC.2011.34",
    language = "Undefined",
    volume = "3",
    pages = "165--183",
    journal = "IEEE transactions on affective computing",
    issn = "1949-3045",
    publisher = "IEEE",
    number = "2",

    }

    Schroeder, M, Bevacqua, E, Cowie, R, Eyben, F, Gunes, H, Heylen, DKJ, ter Maat, M, McKeown, G, Pammi, S, Pantic, M, Pelachaud, C, Schuller, B, de Sevin, E, Valstar, M & Wöllmer, M 2012, 'Building Autonomous Sensitive Artificial Listeners', IEEE transactions on affective computing, vol. 3, no. 2, pp. 165-183. https://doi.org/10.1109/T-AFFC.2011.34

    Building Autonomous Sensitive Artificial Listeners. / Schroeder, Marc; Bevacqua, Elisabetta; Cowie, Roddy; Eyben, Florian; Gunes, Hatice; Heylen, Dirk K.J.; ter Maat, Mark; McKeown, Gary; Pammi, Sathish; Pantic, Maja; Pelachaud, Catherine; Schuller, Björn; de Sevin, Etienne; Valstar, Michel; Wöllmer, Martin.

    In: IEEE transactions on affective computing, Vol. 3, No. 2, 04.2012, p. 165-183.

    Research output: Contribution to journalArticleAcademicpeer-review

    TY - JOUR

    T1 - Building Autonomous Sensitive Artificial Listeners

    AU - Schroeder, Marc

    AU - Bevacqua, Elisabetta

    AU - Cowie, Roddy

    AU - Eyben, Florian

    AU - Gunes, Hatice

    AU - Heylen, Dirk K.J.

    AU - ter Maat, Mark

    AU - McKeown, Gary

    AU - Pammi, Sathish

    AU - Pantic, Maja

    AU - Pelachaud, Catherine

    AU - Schuller, Björn

    AU - de Sevin, Etienne

    AU - Valstar, Michel

    AU - Wöllmer, Martin

    N1 - eemcs-eprint-22932

    PY - 2012/4

    Y1 - 2012/4

    N2 - This paper describes a substantial effort to build a real-time interactive multimodal dialogue system with a focus on emotional and nonverbal interaction capabilities. The work is motivated by the aim to provide technology with competences in perceiving and producing the emotional and nonverbal behaviors required to sustain a conversational dialogue. We present the Sensitive Artificial Listener (SAL) scenario as a setting which seems particularly suited for the study of emotional and nonverbal behavior since it requires only very limited verbal understanding on the part of the machine. This scenario allows us to concentrate on nonverbal capabilities without having to address at the same time the challenges of spoken language understanding, task modeling, etc. We first report on three prototype versions of the SAL scenario in which the behavior of the Sensitive Artificial Listener characters was determined by a human operator. These prototypes served the purpose of verifying the effectiveness of the SAL scenario and allowed us to collect data required for building system components for analyzing and synthesizing the respective behaviors. We then describe the fully autonomous integrated real-time system we created, which combines incremental analysis of user behavior, dialogue management, and synthesis of speaker and listener behavior of a SAL character displayed as a virtual agent. We discuss principles that should underlie the evaluation of SAL-type systems. Since the system is designed for modularity and reuse and since it is publicly available, the SAL system has potential as a joint research tool in the affective computing research community.

    AB - This paper describes a substantial effort to build a real-time interactive multimodal dialogue system with a focus on emotional and nonverbal interaction capabilities. The work is motivated by the aim to provide technology with competences in perceiving and producing the emotional and nonverbal behaviors required to sustain a conversational dialogue. We present the Sensitive Artificial Listener (SAL) scenario as a setting which seems particularly suited for the study of emotional and nonverbal behavior since it requires only very limited verbal understanding on the part of the machine. This scenario allows us to concentrate on nonverbal capabilities without having to address at the same time the challenges of spoken language understanding, task modeling, etc. We first report on three prototype versions of the SAL scenario in which the behavior of the Sensitive Artificial Listener characters was determined by a human operator. These prototypes served the purpose of verifying the effectiveness of the SAL scenario and allowed us to collect data required for building system components for analyzing and synthesizing the respective behaviors. We then describe the fully autonomous integrated real-time system we created, which combines incremental analysis of user behavior, dialogue management, and synthesis of speaker and listener behavior of a SAL character displayed as a virtual agent. We discuss principles that should underlie the evaluation of SAL-type systems. Since the system is designed for modularity and reuse and since it is publicly available, the SAL system has potential as a joint research tool in the affective computing research community.

    KW - HMI-MI: MULTIMODAL INTERACTIONS

    KW - listener behavior

    KW - real-time dialogue

    KW - EWI-22932

    KW - Embodied Conversational Agents

    KW - IR-84214

    KW - Turn Taking

    KW - Rapport agents

    KW - emotion synthesis

    KW - METIS-296236

    KW - Emotion Recognition

    U2 - 10.1109/T-AFFC.2011.34

    DO - 10.1109/T-AFFC.2011.34

    M3 - Article

    VL - 3

    SP - 165

    EP - 183

    JO - IEEE transactions on affective computing

    JF - IEEE transactions on affective computing

    SN - 1949-3045

    IS - 2

    ER -