A dynamic texture based approach to recognition of facial actions and their temporal models

Sander Koelstra, Maja Pantic, Ioannis (Yannis) Patras

Research output: Contribution to journalArticleAcademicpeer-review

179 Citations (Scopus)
10 Downloads (Pure)

Abstract

In this work, we propose a dynamic texture-based approach to the recognition of facial Action Units (AUs, atomic facial gestures) and their temporal models (i.e., sequences of temporal segments: neutral, onset, apex, and offset) in near-frontal-view face videos. Two approaches to modeling the dynamics and the appearance in the face region of an input video are compared: an extended version of Motion History Images and a novel method based on Nonrigid Registration using Free-Form Deformations (FFDs). The extracted motion representation is used to derive motion orientation histogram descriptors in both the spatial and temporal domain. Per AU, a combination of discriminative, frame-based GentleBoost ensemble learners and dynamic, generative Hidden Markov Models detects the presence of the AU in question and its temporal segments in an input image sequence. When tested for recognition of all 27 lower and upper face AUs, occurring alone or in combination in 264 sequences from the MMI facial expression database, the proposed method achieved an average event recognition accuracy of 89.2 percent for the MHI method and 94.3 percent for the FFD method. The generalization performance of the FFD method has been tested using the Cohn-Kanade database. Finally, we also explored the performance on spontaneous expressions in the Sensitive Artificial Listener data set.
Original languageUndefined
Pages (from-to)1940-1954
Number of pages15
JournalIEEE transactions on pattern analysis and machine intelligence
Volume32
Issue number11
DOIs
Publication statusPublished - Nov 2010

Keywords

  • dynamic texture
  • EC Grant Agreement nr.: FP7/211486
  • EC Grant Agreement nr.: FP7/216444
  • EWI-19457
  • IR-75886
  • Motion
  • Facial Expression
  • Facial image analysis
  • METIS-276325
  • HMI-MI: MULTIMODAL INTERACTIONS

Cite this

Koelstra, Sander ; Pantic, Maja ; Patras, Ioannis (Yannis). / A dynamic texture based approach to recognition of facial actions and their temporal models. In: IEEE transactions on pattern analysis and machine intelligence. 2010 ; Vol. 32, No. 11. pp. 1940-1954.
@article{c7aa6834c49640d6baf845ce659e4c54,
title = "A dynamic texture based approach to recognition of facial actions and their temporal models",
abstract = "In this work, we propose a dynamic texture-based approach to the recognition of facial Action Units (AUs, atomic facial gestures) and their temporal models (i.e., sequences of temporal segments: neutral, onset, apex, and offset) in near-frontal-view face videos. Two approaches to modeling the dynamics and the appearance in the face region of an input video are compared: an extended version of Motion History Images and a novel method based on Nonrigid Registration using Free-Form Deformations (FFDs). The extracted motion representation is used to derive motion orientation histogram descriptors in both the spatial and temporal domain. Per AU, a combination of discriminative, frame-based GentleBoost ensemble learners and dynamic, generative Hidden Markov Models detects the presence of the AU in question and its temporal segments in an input image sequence. When tested for recognition of all 27 lower and upper face AUs, occurring alone or in combination in 264 sequences from the MMI facial expression database, the proposed method achieved an average event recognition accuracy of 89.2 percent for the MHI method and 94.3 percent for the FFD method. The generalization performance of the FFD method has been tested using the Cohn-Kanade database. Finally, we also explored the performance on spontaneous expressions in the Sensitive Artificial Listener data set.",
keywords = "dynamic texture, EC Grant Agreement nr.: FP7/211486, EC Grant Agreement nr.: FP7/216444, EWI-19457, IR-75886, Motion, Facial Expression, Facial image analysis, METIS-276325, HMI-MI: MULTIMODAL INTERACTIONS",
author = "Sander Koelstra and Maja Pantic and Patras, {Ioannis (Yannis)}",
note = "eemcs-eprint-19457",
year = "2010",
month = "11",
doi = "10.1109/TPAMI.2010.50",
language = "Undefined",
volume = "32",
pages = "1940--1954",
journal = "IEEE transactions on pattern analysis and machine intelligence",
issn = "0162-8828",
publisher = "IEEE Computer Society",
number = "11",

}

A dynamic texture based approach to recognition of facial actions and their temporal models. / Koelstra, Sander; Pantic, Maja; Patras, Ioannis (Yannis).

In: IEEE transactions on pattern analysis and machine intelligence, Vol. 32, No. 11, 11.2010, p. 1940-1954.

Research output: Contribution to journalArticleAcademicpeer-review

TY - JOUR

T1 - A dynamic texture based approach to recognition of facial actions and their temporal models

AU - Koelstra, Sander

AU - Pantic, Maja

AU - Patras, Ioannis (Yannis)

N1 - eemcs-eprint-19457

PY - 2010/11

Y1 - 2010/11

N2 - In this work, we propose a dynamic texture-based approach to the recognition of facial Action Units (AUs, atomic facial gestures) and their temporal models (i.e., sequences of temporal segments: neutral, onset, apex, and offset) in near-frontal-view face videos. Two approaches to modeling the dynamics and the appearance in the face region of an input video are compared: an extended version of Motion History Images and a novel method based on Nonrigid Registration using Free-Form Deformations (FFDs). The extracted motion representation is used to derive motion orientation histogram descriptors in both the spatial and temporal domain. Per AU, a combination of discriminative, frame-based GentleBoost ensemble learners and dynamic, generative Hidden Markov Models detects the presence of the AU in question and its temporal segments in an input image sequence. When tested for recognition of all 27 lower and upper face AUs, occurring alone or in combination in 264 sequences from the MMI facial expression database, the proposed method achieved an average event recognition accuracy of 89.2 percent for the MHI method and 94.3 percent for the FFD method. The generalization performance of the FFD method has been tested using the Cohn-Kanade database. Finally, we also explored the performance on spontaneous expressions in the Sensitive Artificial Listener data set.

AB - In this work, we propose a dynamic texture-based approach to the recognition of facial Action Units (AUs, atomic facial gestures) and their temporal models (i.e., sequences of temporal segments: neutral, onset, apex, and offset) in near-frontal-view face videos. Two approaches to modeling the dynamics and the appearance in the face region of an input video are compared: an extended version of Motion History Images and a novel method based on Nonrigid Registration using Free-Form Deformations (FFDs). The extracted motion representation is used to derive motion orientation histogram descriptors in both the spatial and temporal domain. Per AU, a combination of discriminative, frame-based GentleBoost ensemble learners and dynamic, generative Hidden Markov Models detects the presence of the AU in question and its temporal segments in an input image sequence. When tested for recognition of all 27 lower and upper face AUs, occurring alone or in combination in 264 sequences from the MMI facial expression database, the proposed method achieved an average event recognition accuracy of 89.2 percent for the MHI method and 94.3 percent for the FFD method. The generalization performance of the FFD method has been tested using the Cohn-Kanade database. Finally, we also explored the performance on spontaneous expressions in the Sensitive Artificial Listener data set.

KW - dynamic texture

KW - EC Grant Agreement nr.: FP7/211486

KW - EC Grant Agreement nr.: FP7/216444

KW - EWI-19457

KW - IR-75886

KW - Motion

KW - Facial Expression

KW - Facial image analysis

KW - METIS-276325

KW - HMI-MI: MULTIMODAL INTERACTIONS

U2 - 10.1109/TPAMI.2010.50

DO - 10.1109/TPAMI.2010.50

M3 - Article

VL - 32

SP - 1940

EP - 1954

JO - IEEE transactions on pattern analysis and machine intelligence

JF - IEEE transactions on pattern analysis and machine intelligence

SN - 0162-8828

IS - 11

ER -