Temporal Interaction between an Artificial Orchestra Conductor and Human Musicians

One interest of our research group is in (1) developing Ambient Entertainment technologies and applications that (2) interact in a coordinated way with human partners using (3) a multitude of different sensors observing many characteristics of the partner and using (4) many different channels of expression (such as sound, visuals, speech and embodiment with gestures and facial expressions). The Virtual Conductor project [1, 2] concerns the development of the first properly interactive virtual orchestra conductor that can conduct a piece of music through interaction with musicians, leading and following them while they are playing. Its observations consist of different manners of musical processing of the incoming signal from a microphone. The forms of its expressions are defined by the possible conducting patterns and by the timing, speed, amplitude and smoothness with which those patterns are expressed. The interaction is focused on the tempo of the musicians and includes a correction module that interactively corrects the tempo when the musicians are playing too slow or too fast.
We describe our motivation for developing such a system; related work in the areas of ambient entertainment and coordinated timing, automatic music processing, virtual humans and conducting; the design, implementation and evaluation of the Virtual Conductor; and, finally, contains a discussion of the resulting system, general applicability of the ideas and technology developed in this project, and expected developments in the (ongoing) Virtual Conductor project.