Multimodal Detection of Engagement in Groups of Children Using Rank Learning

Jaebok Kim, Khiet Phuong Truong, Vasiliki Charisi, Cristina Zaga, Vanessa Evers, Mohamed Chetouani

    Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

    4 Citations (Scopus)
    119 Downloads (Pure)


    In collaborative play, children exhibit different levels of engagement. Some children are engaged with other children while some play alone. In this study, we investigated multimodal detection of individual levels of engagement using a ranking method and non-verbal features: turn-taking and body movement. Firstly, we automatically extracted turn-taking and body movement features in naturalistic and challenging settings. Secondly, we used an ordinal annotation scheme and employed a ranking method considering the great heterogeneity and temporal dynamics of engagement that exist in interactions. We showed that levels of engagement can be characterised by relative levels between children. In particular, a ranking method, Ranking SVM, outperformed a conventional method, SVM classification. While either turn-taking or body movement features alone did not achieve promising results, combining the two features yielded significant error reduction, showing their complementary power.
    Original languageEnglish
    Title of host publicationHuman Behavior Understanding
    Subtitle of host publication7th International Workshop, HBU 2016, Amsterdam, The Netherlands, October 16, 2016, Proceedings
    EditorsMohamed Chetouani, Jeffrey Cohn, Albert Ali Salah
    Place of PublicationLondon
    Number of pages14
    ISBN (Electronic)978-3-319-46843-3
    ISBN (Print)978-3-319-46842-6
    Publication statusPublished - Oct 2016
    Event7th International Workshop on Human Behavior Understanding, HBU 2016 - Amsterdam, The Netherlands
    Duration: 16 Oct 201616 Oct 2016

    Publication series

    NameLecture Notes in Computer Science
    PublisherSpringer Verlag
    ISSN (Print)0302-9743
    ISSN (Electronic)1611-3349


    Workshop7th International Workshop on Human Behavior Understanding, HBU 2016
    OtherOctober 16, 2016


    • EC Grant Agreement nr.: FP7/610532
    • HMI-SLT: Speech and Language Technology
    • Non-verbal behaviours
    • Social signal processing
    • Children
    • Engagement

    Cite this