Context-sensitive Dynamic Ordinal Regression for Intensity Estimation of Facial Action Units

Ognjen Rudovic, Vladimir Pavlovic, Maja Pantic

    Research output: Contribution to journalArticleAcademicpeer-review

    91 Citations (Scopus)


    Modeling intensity of facial action units from spontaneously displayed facial expressions is challenging mainly because of high variability in subject-specific facial expressiveness, head-movements, illumination changes, etc. These factors make the target problem highly context-sensitive. However, existing methods usually ignore this context-sensitivity of the target problem. We propose a novel Conditional Ordinal Random Field (CORF) model for context-sensitive modeling of the facial action unit intensity, where the W5+ (who, when, what, where, why and how) definition of the context is used. While the proposed model is general enough to handle all six context questions, in this paper we focus on the context questions: who (the observed subject), how (the changes in facial expressions), and when (the timing of facial expressions and their intensity). The context questions who and howare modeled by means of the newly introduced context-dependent covariate effects, and the context question when is modeled in terms of temporal correlation between the ordinal outputs, i.e., intensity levels of action units. We also introduce a weighted softmax-margin learning of CRFs from data with skewed distribution of the intensity levels, which is commonly encountered in spontaneous facial data. The proposed model is evaluated on intensity estimation of pain and facial action units using two recently published datasets (UNBC Shoulder Pain and DISFA) of spontaneously displayed facial expressions. Our experiments show that the proposed model performs significantly better on the target tasks compared to the state-of-the-art approaches. Furthermore, compared to traditional learning of CRFs, we show that the proposed weighted learning results in more robust parameter estimation from the imbalanced intensity data.
    Original languageUndefined
    Pages (from-to)944-958
    Number of pages15
    JournalIEEE transactions on pattern analysis and machine intelligence
    Issue number5
    Publication statusPublished - 1 May 2015


    • action unit intensity
    • spontaneous facial behavior
    • FACS
    • HMI-HF: Human Factors
    • EC Grant Agreement nr.: FP7/611153
    • EC Grant Agreement nr.: FP7/2007-2013
    • IR-100359
    • Facial Expression Analysis
    • ordinal regres- sion
    • Context modeling
    • Conditional random fields
    • METIS-316027
    • EWI-26769

    Cite this