Multi-Instance Dynamic Ordinal Random Fields for Weakly Supervised Facial Behavior Analysis

Adria Ruiz*, Ognjen Rudovic, Xavier Binefa, Maja Pantic

*Corresponding author for this work

    Research output: Contribution to journalArticleAcademicpeer-review

    14 Citations (Scopus)


    We propose a multi-instance-learning (MIL) approach for weakly supervised learning problems, where a training set is formed by bags (sets of feature vectors or instances) and only labels at bag-level are provided. Specifically, we consider the multi-instance dynamic-ordinal-regression (MI-DOR) setting, where the instance labels are naturally represented as ordinal variables and bags are structured as temporal sequences. To this end, we propose MI dynamic ordinal random fields (MI-DORF). In this paper, we treat instance-labels as temporally dependent latent variables in an undirected graphical model. Different MIL assumptions are modelled via newly introduced high-order potentials relating bag and instance-labels within the energy function of the model. We also extend our framework to address the partially observed MI-DOR problem, where a subset of instance labels is also available during training. We show on the tasks of weakly supervised facial action unit and pain intensity estimation, that the proposed framework outperforms alternative learning approaches. Furthermore, we show that MI-DORF can be employed to reduce the data annotation efforts in this context by large-scale.

    Original languageEnglish
    Pages (from-to)3969-3982
    Number of pages14
    JournalIEEE transactions on image processing
    Issue number8
    Publication statusPublished - Aug 2018


    • action units
    • facial behavior analysis
    • Mutiple instance learning
    • pain intensity
    • undirected graphical models
    • n/a OA procedure


    Dive into the research topics of 'Multi-Instance Dynamic Ordinal Random Fields for Weakly Supervised Facial Behavior Analysis'. Together they form a unique fingerprint.

    Cite this