Towards visual and vocal mimicry recognition in human-human interactions

X. Sun, Khiet Phuong Truong, Maja Pantic, Antinus Nijholt

    Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

    20 Citations (Scopus)

    Abstract

    During face-to-face interpersonal interaction, people have a tendency to mimic each other. People not only mimic postures, mannerisms, moods or emotions, but they also mimic several speech-related behaviors. In this paper we describe how visual and vocal behavioral information expressed between two interlocutors can be used to detect and identify visual and vocal mimicry. We investigate expressions of mimicry and aim to learn more about in which situation and to what extent mimicry occurs. The observable effects of mimicry can be explored by representing and recognizing mimicry using visual and vocal features. In order to automatically analyze how to extract and integrate this behavioral information into a multimodal mimicry detection framework for improving affective computing, this paper addresses the main challenge: mimicry representation in terms of optimal behavioral feature extraction and automatic integration in both audio and video modalities.
    Original languageEnglish
    Title of host publicationIEEE SMC 2011
    Subtitle of host publication2011 IEEE International Conference on Systems, Man and Cybernetics October 9-12, 2011
    EditorsE. Tunstel, S. Nahavandi, A. Stoica
    Place of PublicationPiscataway, NJ
    PublisherIEEE Computer Society
    Pages367-373
    Number of pages7
    ISBN (Print)978-1-4577-0652-3
    DOIs
    Publication statusPublished - 21 Nov 2011
    EventIEEE International Conference on Systems, Man, and Cybernetics, SMC 2011 - Hilton Anchorage, Anchorage, United States
    Duration: 9 Oct 201112 Oct 2011

    Conference

    ConferenceIEEE International Conference on Systems, Man, and Cybernetics, SMC 2011
    Abbreviated titleSMC
    CountryUnited States
    CityAnchorage
    Period9/10/1112/10/11

    Fingerprint

    Feature extraction

    Keywords

    • METIS-281573
    • Human behavior analysis
    • Mimicry representation
    • IR-78613
    • human- human interaction
    • vocal mimicry
    • Visual mimicry
    • HMI-MI: MULTIMODAL INTERACTIONS
    • EWI-20822
    • EC Grant Agreement nr.: FP7/231287
    • EC Grant Agreement nr.: ERC/203143
    • interpersonal synchrony

    Cite this

    Sun, X., Truong, K. P., Pantic, M., & Nijholt, A. (2011). Towards visual and vocal mimicry recognition in human-human interactions. In E. Tunstel, S. Nahavandi, & A. Stoica (Eds.), IEEE SMC 2011: 2011 IEEE International Conference on Systems, Man and Cybernetics October 9-12, 2011 (pp. 367-373). Piscataway, NJ: IEEE Computer Society. https://doi.org/10.1109/ICSMC.2011.6083693
    Sun, X. ; Truong, Khiet Phuong ; Pantic, Maja ; Nijholt, Antinus. / Towards visual and vocal mimicry recognition in human-human interactions. IEEE SMC 2011: 2011 IEEE International Conference on Systems, Man and Cybernetics October 9-12, 2011. editor / E. Tunstel ; S. Nahavandi ; A. Stoica. Piscataway, NJ : IEEE Computer Society, 2011. pp. 367-373
    @inproceedings{ae9c2c79e0e349bbbf3b281282336682,
    title = "Towards visual and vocal mimicry recognition in human-human interactions",
    abstract = "During face-to-face interpersonal interaction, people have a tendency to mimic each other. People not only mimic postures, mannerisms, moods or emotions, but they also mimic several speech-related behaviors. In this paper we describe how visual and vocal behavioral information expressed between two interlocutors can be used to detect and identify visual and vocal mimicry. We investigate expressions of mimicry and aim to learn more about in which situation and to what extent mimicry occurs. The observable effects of mimicry can be explored by representing and recognizing mimicry using visual and vocal features. In order to automatically analyze how to extract and integrate this behavioral information into a multimodal mimicry detection framework for improving affective computing, this paper addresses the main challenge: mimicry representation in terms of optimal behavioral feature extraction and automatic integration in both audio and video modalities.",
    keywords = "METIS-281573, Human behavior analysis, Mimicry representation, IR-78613, human- human interaction, vocal mimicry, Visual mimicry, HMI-MI: MULTIMODAL INTERACTIONS, EWI-20822, EC Grant Agreement nr.: FP7/231287, EC Grant Agreement nr.: ERC/203143, interpersonal synchrony",
    author = "X. Sun and Truong, {Khiet Phuong} and Maja Pantic and Antinus Nijholt",
    note = "Special Session on Social Signal Processing",
    year = "2011",
    month = "11",
    day = "21",
    doi = "10.1109/ICSMC.2011.6083693",
    language = "English",
    isbn = "978-1-4577-0652-3",
    pages = "367--373",
    editor = "E. Tunstel and S. Nahavandi and A. Stoica",
    booktitle = "IEEE SMC 2011",
    publisher = "IEEE Computer Society",
    address = "United States",

    }

    Sun, X, Truong, KP, Pantic, M & Nijholt, A 2011, Towards visual and vocal mimicry recognition in human-human interactions. in E Tunstel, S Nahavandi & A Stoica (eds), IEEE SMC 2011: 2011 IEEE International Conference on Systems, Man and Cybernetics October 9-12, 2011. IEEE Computer Society, Piscataway, NJ, pp. 367-373, IEEE International Conference on Systems, Man, and Cybernetics, SMC 2011, Anchorage, United States, 9/10/11. https://doi.org/10.1109/ICSMC.2011.6083693

    Towards visual and vocal mimicry recognition in human-human interactions. / Sun, X.; Truong, Khiet Phuong; Pantic, Maja; Nijholt, Antinus.

    IEEE SMC 2011: 2011 IEEE International Conference on Systems, Man and Cybernetics October 9-12, 2011. ed. / E. Tunstel; S. Nahavandi; A. Stoica. Piscataway, NJ : IEEE Computer Society, 2011. p. 367-373.

    Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

    TY - GEN

    T1 - Towards visual and vocal mimicry recognition in human-human interactions

    AU - Sun, X.

    AU - Truong, Khiet Phuong

    AU - Pantic, Maja

    AU - Nijholt, Antinus

    N1 - Special Session on Social Signal Processing

    PY - 2011/11/21

    Y1 - 2011/11/21

    N2 - During face-to-face interpersonal interaction, people have a tendency to mimic each other. People not only mimic postures, mannerisms, moods or emotions, but they also mimic several speech-related behaviors. In this paper we describe how visual and vocal behavioral information expressed between two interlocutors can be used to detect and identify visual and vocal mimicry. We investigate expressions of mimicry and aim to learn more about in which situation and to what extent mimicry occurs. The observable effects of mimicry can be explored by representing and recognizing mimicry using visual and vocal features. In order to automatically analyze how to extract and integrate this behavioral information into a multimodal mimicry detection framework for improving affective computing, this paper addresses the main challenge: mimicry representation in terms of optimal behavioral feature extraction and automatic integration in both audio and video modalities.

    AB - During face-to-face interpersonal interaction, people have a tendency to mimic each other. People not only mimic postures, mannerisms, moods or emotions, but they also mimic several speech-related behaviors. In this paper we describe how visual and vocal behavioral information expressed between two interlocutors can be used to detect and identify visual and vocal mimicry. We investigate expressions of mimicry and aim to learn more about in which situation and to what extent mimicry occurs. The observable effects of mimicry can be explored by representing and recognizing mimicry using visual and vocal features. In order to automatically analyze how to extract and integrate this behavioral information into a multimodal mimicry detection framework for improving affective computing, this paper addresses the main challenge: mimicry representation in terms of optimal behavioral feature extraction and automatic integration in both audio and video modalities.

    KW - METIS-281573

    KW - Human behavior analysis

    KW - Mimicry representation

    KW - IR-78613

    KW - human- human interaction

    KW - vocal mimicry

    KW - Visual mimicry

    KW - HMI-MI: MULTIMODAL INTERACTIONS

    KW - EWI-20822

    KW - EC Grant Agreement nr.: FP7/231287

    KW - EC Grant Agreement nr.: ERC/203143

    KW - interpersonal synchrony

    U2 - 10.1109/ICSMC.2011.6083693

    DO - 10.1109/ICSMC.2011.6083693

    M3 - Conference contribution

    SN - 978-1-4577-0652-3

    SP - 367

    EP - 373

    BT - IEEE SMC 2011

    A2 - Tunstel, E.

    A2 - Nahavandi, S.

    A2 - Stoica, A.

    PB - IEEE Computer Society

    CY - Piscataway, NJ

    ER -

    Sun X, Truong KP, Pantic M, Nijholt A. Towards visual and vocal mimicry recognition in human-human interactions. In Tunstel E, Nahavandi S, Stoica A, editors, IEEE SMC 2011: 2011 IEEE International Conference on Systems, Man and Cybernetics October 9-12, 2011. Piscataway, NJ: IEEE Computer Society. 2011. p. 367-373 https://doi.org/10.1109/ICSMC.2011.6083693