Human and Virtual Agents Interacting in the Virtuality Continuum

In this paper we take a multi-party interaction point of view on our research on multimodal interactions between agents in various virtual environments: an educational environment, a meeting environment, and a storytelling environment. These environments are quite different. All these environments require the modeling of multimodal interaction: interactions between human users, the environments and objects represented in the environments, and embodied conversational agents that represent human users or that have been designed to play particular roles in the environment. Rather than interacting with one particular user agents need to interact with different human and synthetic agents, they need to know about the properties (personalities, intelligence, emotions, capabilities, etc.) of these different agents, they need to know who is aware of what they are saying or doing, and they need to maintain a model of the multi-party dialogue between the different agents.. Rather than introduce new research results we choose to give an overview of the multi-party interaction projects we work on, with the aim, both for ourselves and the readers, to make our intuitively available framework more explicit and to give it a more fundamental basis by comparing it with research projects performed by others.