Static vs. Dynamic Modeling of Human Nonverbal Behavior from Multiple Cues and Modalities

Stavros Petridis, Hatice Gunes, Sebastian Kaltwang, Maja Pantic

    Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

    26 Citations (Scopus)

    Abstract

    Human nonverbal behavior recognition from multiple cues and modalities has attracted a lot of interest in recent years. Despite the interest, many research questions, including the type of feature representation, choice of static vs. dynamic classification schemes, the number and type of cues or modalities to use, and the optimal way of fusing these, remain open research questions. This paper compares frame-based vs window-based feature representation and employs static vs. dynamic classification schemes for two distinct problems in the field of automatic human nonverbal behavior analysis: multicue discrimination between posed and spontaneous smiles from facial expressions, head and shoulder movements, and audio-visual discrimination between laughter and speech. Single cue and single modality results are compared to multicue and multimodal results by employing Neural Networks, Hidden Markov Models (HMMs), and 2- and 3-chain coupled HMMs. Subject independent experimental evaluation shows that: 1) both for static and dynamic classification, fusing data coming from multiple cues and modalities proves useful to the overall task of recognition, 2) the type of feature representation appears to have a direct impact on the classification performance, and 3) static classification is comparable to dynamic classification both for multicue discrimination between posed and spontaneous smiles, and audio-visual discrimination between laughter and speech.
    Original languageUndefined
    Title of host publicationProceedings of the 2009 International Conference on Multimodal Interfaces (ICMI’09)
    Place of PublicationNew York
    PublisherAssociation for Computing Machinery (ACM)
    Pages23-30
    Number of pages8
    ISBN (Print)978-1-60558-772-1
    DOIs
    Publication statusPublished - 2009
    Event11th International Conference on Multimodal Interfaces, ICMI 2009 - Boston, United States
    Duration: 2 Nov 20094 Nov 2009
    Conference number: 11

    Publication series

    Name
    PublisherACM

    Conference

    Conference11th International Conference on Multimodal Interfaces, ICMI 2009
    Abbreviated titleICMI
    CountryUnited States
    CityBoston
    Period2/11/094/11/09

    Keywords

    • METIS-264302
    • IR-69478
    • Multi-modal Fusion
    • HMI-HF: Human Factors
    • EC Grant Agreement nr.: FP7/211486
    • assimilation and processing
    • HMI-MI: MULTIMODAL INTERACTIONS
    • EWI-17130

    Cite this