Computational models of social and emotional turn-taking for embodied conversational agents: a review

Hendrikus J.A. op den Akker, Merijn Bruijnes

    Research output: Book/ReportReportProfessional

    69 Downloads (Pure)


    The emotional involvement of participants in a conversation not only shows in the words they speak and in the way they speak and gesture but also in their turn-taking behavior. This paper reviews research into computational models of embodied conversational agents. We focus on models for turn-taking management and (social) emotions. We are particularly interested in how in these models emotions of the agent itself and those of the others in uence the agent's turn-taking behavior and vice versa how turn-taking behavior of the partner is perceived by the agent itself. The system of turn-taking rules presented by Sacks, Schegloff and Jefferson (1974) is often a starting point for computational turn-taking models of conversational agents. But emotions have their own rules besides the "one-at-a-time" paradigm of the SSJ system. It turns out that almost without exception computational models of turn-taking behavior that allow "continuous interaction" and "natural turntaking" do not model the underlying psychological, affective, attentional and cognitive processes. They are restricted to rules in terms of a number of supercially observable cues. On the other hand computational models for virtual humans that are based on a functional theory of social emotion do not contain explicit rules on how social emotions affect turn-taking behavior or how the emotional state of the agent is affected by turn-taking behavior of its interlocutors. We conclude with some preliminary ideas on what an architecture for emotional turn-taking should look like and we discuss the challenges in building believable emotional turn-taking agents.
    Original languageUndefined
    Place of PublicationEnschede
    PublisherCentre for Telematics and Information Technology (CTIT)
    Number of pages62
    Publication statusPublished - 23 May 2012

    Publication series

    NameCTIT Technical Report Series
    PublisherUniversity of Twente, Centre for Telematics and Information Technology (CTIT)
    ISSN (Print)1381-3625


    • Emotion
    • IR-80451
    • computational models turn taking emotion agents
    • Computational Models
    • METIS-286360
    • Agents
    • EWI-21867
    • Turn Taking

    Cite this

    op den Akker, H. J. A., & Bruijnes, M. (2012). Computational models of social and emotional turn-taking for embodied conversational agents: a review. (CTIT Technical Report Series; No. TR-CTIT-12-13). Enschede: Centre for Telematics and Information Technology (CTIT).