Multi-Modal Neural Conditional Ordinal Random Fields for Agreement Level Estimation

N. Rakicevic, O. Rudovic, S. Petridis, Maja Pantic

    Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

    2 Citations (Scopus)
    22 Downloads (Pure)


    The ability to automatically detect the extent of agreement or disagreement a person expresses is an important indicator of inter-personal relations and emotion expression. Most of existing methods for automated analysis of human agreement from audio-visual data perform agreement detection using either audio or visual modality of human interactions. However, this is suboptimal as expression of different agreement levels is composed of various facial and vocal cues specific to the target level. To this end, we propose the first approach for multi-modal estimation of agreement intensity levels. Specifically, our model leverages the feature representation power of Multimodal Neural Networks (NN) and discriminative power of Conditional Ordinal Random Fields (CORF) to achieve dynamic classification of agreement levels from videos. We show on the MAHNOB-Mimicry database of dyadic human interactions that the proposed approach outperforms its uni-modal and linear counterparts, and related models that can be applied to the target task.
    Original languageEnglish
    Title of host publication2016 23rd International Conference on Pattern Recognition (ICPR)
    Place of PublicationUSA
    Number of pages6
    ISBN (Electronic)978-1-5090-4847-2
    ISBN (Print)978-1-5090-4848-9
    Publication statusPublished - Apr 2017
    Event23rd International Conference on Pattern Recognition 2016 - Cancun, Mexico
    Duration: 4 Dec 20168 Dec 2016
    Conference number: 23


    Conference23rd International Conference on Pattern Recognition 2016
    Abbreviated titleICPR 2016
    Internet address


    • Automatic speech and speaker recognition
    • Affective Computing
    • EWI-27585
    • HMI-HF: Human Factors


    Dive into the research topics of 'Multi-Modal Neural Conditional Ordinal Random Fields for Agreement Level Estimation'. Together they form a unique fingerprint.

    Cite this