Computational models of social and emotional turn-taking for embodied conversational agents: a review

Abstract

The emotional involvement of participants in a conversation not only shows in the words they speak and in the way they speak and gesture but also in their turn-taking behavior. This paper reviews research into computational models of embodied conversational agents. We focus on models for turn-taking management and (social) emotions. We are particularly interested in how in these models emotions of the agent itself and those of the others in uence the agent's turn-taking behavior and vice versa how turn-taking behavior of the partner is perceived by the agent itself. The system of turn-taking rules presented by Sacks, Schegloff and Jefferson (1974) is often a starting point for computational turn-taking models of conversational agents. But emotions have their own rules besides the "one-at-a-time" paradigm of the SSJ system. It turns out that almost without exception computational models of turn-taking behavior that allow "continuous interaction" and "natural turntaking" do not model the underlying psychological, affective, attentional and cognitive processes. They are restricted to rules in terms of a number of supercially observable cues. On the other hand computational models for virtual humans that are based on a functional theory of social emotion do not contain explicit rules on how social emotions affect turn-taking behavior or how the emotional state of the agent is affected by turn-taking behavior of its interlocutors. We conclude with some preliminary ideas on what an architecture for emotional turn-taking should look like and we discuss the challenges in building believable emotional turn-taking agents.
Original languageUndefined
Place of PublicationEnschede
PublisherCentre for Telematics and Information Technology (CTIT)
Number of pages62
StatePublished - 23 May 2012

Publication series

NameCTIT Technical Report Series
PublisherUniversity of Twente, Centre for Telematics and Information Technology (CTIT)
No.TR-CTIT-12-13
ISSN (Print)1381-3625

Keywords

  • Emotion
  • IR-80451
  • computational models turn taking emotion agents
  • Computational Models
  • METIS-286360
  • Agents
  • EWI-21867
  • Turn Taking

Cite this

op den Akker, H. J. A., & Bruijnes, M. (2012). Computational models of social and emotional turn-taking for embodied conversational agents: a review. (CTIT Technical Report Series; No. TR-CTIT-12-13). Enschede: Centre for Telematics and Information Technology (CTIT).

op den Akker, Hendrikus J.A.; Bruijnes, Merijn / Computational models of social and emotional turn-taking for embodied conversational agents: a review.

Enschede : Centre for Telematics and Information Technology (CTIT), 2012. 62 p. (CTIT Technical Report Series; No. TR-CTIT-12-13).

Research output: ProfessionalReport

@book{c0fa743b007743e5a9c1e71923dbc0b6,
title = "Computational models of social and emotional turn-taking for embodied conversational agents: a review",
abstract = "The emotional involvement of participants in a conversation not only shows in the words they speak and in the way they speak and gesture but also in their turn-taking behavior. This paper reviews research into computational models of embodied conversational agents. We focus on models for turn-taking management and (social) emotions. We are particularly interested in how in these models emotions of the agent itself and those of the others in uence the agent's turn-taking behavior and vice versa how turn-taking behavior of the partner is perceived by the agent itself. The system of turn-taking rules presented by Sacks, Schegloff and Jefferson (1974) is often a starting point for computational turn-taking models of conversational agents. But emotions have their own rules besides the {"}one-at-a-time{"} paradigm of the SSJ system. It turns out that almost without exception computational models of turn-taking behavior that allow {"}continuous interaction{"} and {"}natural turntaking{"} do not model the underlying psychological, affective, attentional and cognitive processes. They are restricted to rules in terms of a number of supercially observable cues. On the other hand computational models for virtual humans that are based on a functional theory of social emotion do not contain explicit rules on how social emotions affect turn-taking behavior or how the emotional state of the agent is affected by turn-taking behavior of its interlocutors. We conclude with some preliminary ideas on what an architecture for emotional turn-taking should look like and we discuss the challenges in building believable emotional turn-taking agents.",
keywords = "Emotion, IR-80451, computational models turn taking emotion agents, Computational Models, METIS-286360, Agents, EWI-21867, Turn Taking",
author = "{op den Akker}, {Hendrikus J.A.} and Merijn Bruijnes",
year = "2012",
month = "5",
series = "CTIT Technical Report Series",
publisher = "Centre for Telematics and Information Technology (CTIT)",
number = "TR-CTIT-12-13",
address = "Netherlands",

}

op den Akker, HJA & Bruijnes, M 2012, Computational models of social and emotional turn-taking for embodied conversational agents: a review. CTIT Technical Report Series, no. TR-CTIT-12-13, Centre for Telematics and Information Technology (CTIT), Enschede.

Computational models of social and emotional turn-taking for embodied conversational agents: a review. / op den Akker, Hendrikus J.A.; Bruijnes, Merijn.

Enschede : Centre for Telematics and Information Technology (CTIT), 2012. 62 p. (CTIT Technical Report Series; No. TR-CTIT-12-13).

Research output: ProfessionalReport

TY - BOOK

T1 - Computational models of social and emotional turn-taking for embodied conversational agents: a review

AU - op den Akker,Hendrikus J.A.

AU - Bruijnes,Merijn

PY - 2012/5/23

Y1 - 2012/5/23

N2 - The emotional involvement of participants in a conversation not only shows in the words they speak and in the way they speak and gesture but also in their turn-taking behavior. This paper reviews research into computational models of embodied conversational agents. We focus on models for turn-taking management and (social) emotions. We are particularly interested in how in these models emotions of the agent itself and those of the others in uence the agent's turn-taking behavior and vice versa how turn-taking behavior of the partner is perceived by the agent itself. The system of turn-taking rules presented by Sacks, Schegloff and Jefferson (1974) is often a starting point for computational turn-taking models of conversational agents. But emotions have their own rules besides the "one-at-a-time" paradigm of the SSJ system. It turns out that almost without exception computational models of turn-taking behavior that allow "continuous interaction" and "natural turntaking" do not model the underlying psychological, affective, attentional and cognitive processes. They are restricted to rules in terms of a number of supercially observable cues. On the other hand computational models for virtual humans that are based on a functional theory of social emotion do not contain explicit rules on how social emotions affect turn-taking behavior or how the emotional state of the agent is affected by turn-taking behavior of its interlocutors. We conclude with some preliminary ideas on what an architecture for emotional turn-taking should look like and we discuss the challenges in building believable emotional turn-taking agents.

AB - The emotional involvement of participants in a conversation not only shows in the words they speak and in the way they speak and gesture but also in their turn-taking behavior. This paper reviews research into computational models of embodied conversational agents. We focus on models for turn-taking management and (social) emotions. We are particularly interested in how in these models emotions of the agent itself and those of the others in uence the agent's turn-taking behavior and vice versa how turn-taking behavior of the partner is perceived by the agent itself. The system of turn-taking rules presented by Sacks, Schegloff and Jefferson (1974) is often a starting point for computational turn-taking models of conversational agents. But emotions have their own rules besides the "one-at-a-time" paradigm of the SSJ system. It turns out that almost without exception computational models of turn-taking behavior that allow "continuous interaction" and "natural turntaking" do not model the underlying psychological, affective, attentional and cognitive processes. They are restricted to rules in terms of a number of supercially observable cues. On the other hand computational models for virtual humans that are based on a functional theory of social emotion do not contain explicit rules on how social emotions affect turn-taking behavior or how the emotional state of the agent is affected by turn-taking behavior of its interlocutors. We conclude with some preliminary ideas on what an architecture for emotional turn-taking should look like and we discuss the challenges in building believable emotional turn-taking agents.

KW - Emotion

KW - IR-80451

KW - computational models turn taking emotion agents

KW - Computational Models

KW - METIS-286360

KW - Agents

KW - EWI-21867

KW - Turn Taking

M3 - Report

T3 - CTIT Technical Report Series

BT - Computational models of social and emotional turn-taking for embodied conversational agents: a review

PB - Centre for Telematics and Information Technology (CTIT)

ER -

op den Akker HJA, Bruijnes M. Computational models of social and emotional turn-taking for embodied conversational agents: a review. Enschede: Centre for Telematics and Information Technology (CTIT), 2012. 62 p. (CTIT Technical Report Series; TR-CTIT-12-13).