Learning and Evaluating Response Prediction Models using Parallel Listener Consensus

I.A. de Kok, Derya Ozkan, Dirk K.J. Heylen, Louis-Philippe Morency

    Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

    11 Citations (Scopus)


    Traditionally listener response prediction models are learned from pre-recorded dyadic interactions. Because of individual differences in behavior, these recordings do not capture the complete ground truth. Where the recorded listener did not respond to an opportunity provided by the speaker, another listener would have responded or vice versa. In this paper, we introduce the concept of parallel listener consensus where the listener responses from multiple parallel interactions are combined to better capture differences and similarities between individuals. We show how parallel listener consensus can be used for both learning and evaluating probabilistic prediction models of listener responses. To improve the learning performance, the parallel consensus helps identifying better negative samples and reduces outliers in the positive samples. We propose a new error measurement called Fconsensus which exploits the parallel consensus to better define the concepts of exactness (mislabels) and completeness (missed labels) for prediction models. We present a series of experiments using the MultiLis Corpus where three listeners were tricked into believing that they had a one-on-one conversation with a speaker, while in fact they were recorded in parallel in interaction with the same speaker. In this paper we show that using parallel listener consensus can improve learning performance and represent better evaluation criteria for predictive models.
    Original languageUndefined
    Title of host publicationInternational Conference on Multimodal Interfaces and the Workshop on Machine Learning for Multimodal Interaction, ICMI-MLMI '10
    Place of PublicationNew York
    PublisherAssociation for Computing Machinery (ACM)
    Number of pages8
    ISBN (Print)978-1-4503-0414-6
    Publication statusPublished - Nov 2010
    Event10th International Conference on Multimodal Interfaces, ICMI 2008 - Chania, Crete, Greece
    Duration: 20 Oct 200822 Oct 2008
    Conference number: 10

    Publication series



    Conference10th International Conference on Multimodal Interfaces, ICMI 2008
    Abbreviated titleICMI
    CityChania, Crete


    • IR-75323
    • METIS-276234
    • EWI-19108
    • HMI-IA: Intelligent Agents
    • EC Grant Agreement nr.: FP7/211486

    Cite this