Abstract
| Original language | Undefined |
|---|---|
| Pages (from-to) | 165-183 |
| Number of pages | 19 |
| Journal | IEEE transactions on affective computing |
| Volume | 3 |
| Issue number | 2 |
| DOIs | |
| Publication status | Published - Apr 2012 |
Keywords
- HMI-MI: MULTIMODAL INTERACTIONS
- listener behavior
- real-time dialogue
- EWI-22932
- Embodied Conversational Agents
- IR-84214
- Turn Taking
- Rapport agents
- emotion synthesis
- METIS-296236
- Emotion Recognition
Cite this
}
Building Autonomous Sensitive Artificial Listeners. / Schroeder, Marc; Bevacqua, Elisabetta; Cowie, Roddy; Eyben, Florian; Gunes, Hatice; Heylen, Dirk K.J.; ter Maat, Mark; McKeown, Gary; Pammi, Sathish; Pantic, Maja; Pelachaud, Catherine; Schuller, Björn; de Sevin, Etienne; Valstar, Michel; Wöllmer, Martin.
In: IEEE transactions on affective computing, Vol. 3, No. 2, 04.2012, p. 165-183.Research output: Contribution to journal › Article › Academic › peer-review
TY - JOUR
T1 - Building Autonomous Sensitive Artificial Listeners
AU - Schroeder, Marc
AU - Bevacqua, Elisabetta
AU - Cowie, Roddy
AU - Eyben, Florian
AU - Gunes, Hatice
AU - Heylen, Dirk K.J.
AU - ter Maat, Mark
AU - McKeown, Gary
AU - Pammi, Sathish
AU - Pantic, Maja
AU - Pelachaud, Catherine
AU - Schuller, Björn
AU - de Sevin, Etienne
AU - Valstar, Michel
AU - Wöllmer, Martin
N1 - eemcs-eprint-22932
PY - 2012/4
Y1 - 2012/4
N2 - This paper describes a substantial effort to build a real-time interactive multimodal dialogue system with a focus on emotional and nonverbal interaction capabilities. The work is motivated by the aim to provide technology with competences in perceiving and producing the emotional and nonverbal behaviors required to sustain a conversational dialogue. We present the Sensitive Artificial Listener (SAL) scenario as a setting which seems particularly suited for the study of emotional and nonverbal behavior since it requires only very limited verbal understanding on the part of the machine. This scenario allows us to concentrate on nonverbal capabilities without having to address at the same time the challenges of spoken language understanding, task modeling, etc. We first report on three prototype versions of the SAL scenario in which the behavior of the Sensitive Artificial Listener characters was determined by a human operator. These prototypes served the purpose of verifying the effectiveness of the SAL scenario and allowed us to collect data required for building system components for analyzing and synthesizing the respective behaviors. We then describe the fully autonomous integrated real-time system we created, which combines incremental analysis of user behavior, dialogue management, and synthesis of speaker and listener behavior of a SAL character displayed as a virtual agent. We discuss principles that should underlie the evaluation of SAL-type systems. Since the system is designed for modularity and reuse and since it is publicly available, the SAL system has potential as a joint research tool in the affective computing research community.
AB - This paper describes a substantial effort to build a real-time interactive multimodal dialogue system with a focus on emotional and nonverbal interaction capabilities. The work is motivated by the aim to provide technology with competences in perceiving and producing the emotional and nonverbal behaviors required to sustain a conversational dialogue. We present the Sensitive Artificial Listener (SAL) scenario as a setting which seems particularly suited for the study of emotional and nonverbal behavior since it requires only very limited verbal understanding on the part of the machine. This scenario allows us to concentrate on nonverbal capabilities without having to address at the same time the challenges of spoken language understanding, task modeling, etc. We first report on three prototype versions of the SAL scenario in which the behavior of the Sensitive Artificial Listener characters was determined by a human operator. These prototypes served the purpose of verifying the effectiveness of the SAL scenario and allowed us to collect data required for building system components for analyzing and synthesizing the respective behaviors. We then describe the fully autonomous integrated real-time system we created, which combines incremental analysis of user behavior, dialogue management, and synthesis of speaker and listener behavior of a SAL character displayed as a virtual agent. We discuss principles that should underlie the evaluation of SAL-type systems. Since the system is designed for modularity and reuse and since it is publicly available, the SAL system has potential as a joint research tool in the affective computing research community.
KW - HMI-MI: MULTIMODAL INTERACTIONS
KW - listener behavior
KW - real-time dialogue
KW - EWI-22932
KW - Embodied Conversational Agents
KW - IR-84214
KW - Turn Taking
KW - Rapport agents
KW - emotion synthesis
KW - METIS-296236
KW - Emotion Recognition
U2 - 10.1109/T-AFFC.2011.34
DO - 10.1109/T-AFFC.2011.34
M3 - Article
VL - 3
SP - 165
EP - 183
JO - IEEE transactions on affective computing
JF - IEEE transactions on affective computing
SN - 1949-3045
IS - 2
ER -