Spontaneous vs. posed facial behavior: Automatic analysis of brow actions

M.F. Valstar, Maja Pantic, Z. Ambadar, J.F. Cohn

Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

118 Citations (Scopus)

Abstract

Past research on automatic facial expression analysis has focused mostly on the recognition of prototypic expressions of discrete emotions rather than on the analysis of dynamic changes over time, although the importance of temporal dynamics of facial expressions for interpretation of the observed facial behavior has been acknowledged for over 20 years. For instance, it has been shown that the temporal dynamics of spontaneous and volitional smiles are fundamentally different from each other. In this work, we argue that the same holds for the temporal dynamics of brow actions and show that velocity, duration, and order of occurrence of brow actions are highly relevant parameters for distinguishing posed from spontaneous brow actions. The proposed system for discrimination between volitional and spontaneous brow actions is based on automatic detection of Action Units (AUs) and their temporal segments (onset, apex, offset) produced by movements of the eyebrows. For each temporal segment of an activated AU, we compute a number of mid-level feature parameters including the maximal intensity, duration, and order of occurrence. We use Gentle Boost to select the most important of these parameters. The selected parameters are used further to train Relevance Vector Machines to determine per temporal segment of an activated AU whether the action was displayed spontaneously or volitionally. Finally, a probabilistic decision function determines the class (spontaneous or posed) for the entire brow action. When tested on 189 samples taken from three different sets of spontaneous and volitional facial data, we attain a 90.7% correct recognition rate.
Original languageUndefined
Title of host publicationProceedings of ACM Int'l Conf. Multimodal Interfaces (ICMI'06)
Place of PublicationNew York, NY, USA
PublisherAssociation for Computing Machinery (ACM)
Pages162-170
Number of pages9
ISBN (Print)1-59593-541-X
DOIs
Publication statusPublished - 3 Nov 2006
Event8th International Conference on Multimodal Interfaces, ICMI 2006 - Banff, Canada
Duration: 2 Nov 20064 Nov 2006
Conference number: 8

Publication series

Name
PublisherACM
Number10

Conference

Conference8th International Conference on Multimodal Interfaces, ICMI 2006
Abbreviated titleICMI
CountryCanada
CityBanff
Period2/11/064/11/06

Keywords

  • HMI-MI: MULTIMODAL INTERACTIONS
  • METIS-248237
  • EWI-11628
  • IR-62076

Cite this

Valstar, M. F., Pantic, M., Ambadar, Z., & Cohn, J. F. (2006). Spontaneous vs. posed facial behavior: Automatic analysis of brow actions. In Proceedings of ACM Int'l Conf. Multimodal Interfaces (ICMI'06) (pp. 162-170). [10.1145/1180995.1181031] New York, NY, USA: Association for Computing Machinery (ACM). https://doi.org/10.1145/1180995.1181031
Valstar, M.F. ; Pantic, Maja ; Ambadar, Z. ; Cohn, J.F. / Spontaneous vs. posed facial behavior: Automatic analysis of brow actions. Proceedings of ACM Int'l Conf. Multimodal Interfaces (ICMI'06). New York, NY, USA : Association for Computing Machinery (ACM), 2006. pp. 162-170
@inproceedings{8d236232a3334b2196927633a1d1d52c,
title = "Spontaneous vs. posed facial behavior: Automatic analysis of brow actions",
abstract = "Past research on automatic facial expression analysis has focused mostly on the recognition of prototypic expressions of discrete emotions rather than on the analysis of dynamic changes over time, although the importance of temporal dynamics of facial expressions for interpretation of the observed facial behavior has been acknowledged for over 20 years. For instance, it has been shown that the temporal dynamics of spontaneous and volitional smiles are fundamentally different from each other. In this work, we argue that the same holds for the temporal dynamics of brow actions and show that velocity, duration, and order of occurrence of brow actions are highly relevant parameters for distinguishing posed from spontaneous brow actions. The proposed system for discrimination between volitional and spontaneous brow actions is based on automatic detection of Action Units (AUs) and their temporal segments (onset, apex, offset) produced by movements of the eyebrows. For each temporal segment of an activated AU, we compute a number of mid-level feature parameters including the maximal intensity, duration, and order of occurrence. We use Gentle Boost to select the most important of these parameters. The selected parameters are used further to train Relevance Vector Machines to determine per temporal segment of an activated AU whether the action was displayed spontaneously or volitionally. Finally, a probabilistic decision function determines the class (spontaneous or posed) for the entire brow action. When tested on 189 samples taken from three different sets of spontaneous and volitional facial data, we attain a 90.7{\%} correct recognition rate.",
keywords = "HMI-MI: MULTIMODAL INTERACTIONS, METIS-248237, EWI-11628, IR-62076",
author = "M.F. Valstar and Maja Pantic and Z. Ambadar and J.F. Cohn",
note = "10.1145/1180995.1181031",
year = "2006",
month = "11",
day = "3",
doi = "10.1145/1180995.1181031",
language = "Undefined",
isbn = "1-59593-541-X",
publisher = "Association for Computing Machinery (ACM)",
number = "10",
pages = "162--170",
booktitle = "Proceedings of ACM Int'l Conf. Multimodal Interfaces (ICMI'06)",
address = "United States",

}

Valstar, MF, Pantic, M, Ambadar, Z & Cohn, JF 2006, Spontaneous vs. posed facial behavior: Automatic analysis of brow actions. in Proceedings of ACM Int'l Conf. Multimodal Interfaces (ICMI'06)., 10.1145/1180995.1181031, Association for Computing Machinery (ACM), New York, NY, USA, pp. 162-170, 8th International Conference on Multimodal Interfaces, ICMI 2006, Banff, Canada, 2/11/06. https://doi.org/10.1145/1180995.1181031

Spontaneous vs. posed facial behavior: Automatic analysis of brow actions. / Valstar, M.F.; Pantic, Maja; Ambadar, Z.; Cohn, J.F.

Proceedings of ACM Int'l Conf. Multimodal Interfaces (ICMI'06). New York, NY, USA : Association for Computing Machinery (ACM), 2006. p. 162-170 10.1145/1180995.1181031.

Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

TY - GEN

T1 - Spontaneous vs. posed facial behavior: Automatic analysis of brow actions

AU - Valstar, M.F.

AU - Pantic, Maja

AU - Ambadar, Z.

AU - Cohn, J.F.

N1 - 10.1145/1180995.1181031

PY - 2006/11/3

Y1 - 2006/11/3

N2 - Past research on automatic facial expression analysis has focused mostly on the recognition of prototypic expressions of discrete emotions rather than on the analysis of dynamic changes over time, although the importance of temporal dynamics of facial expressions for interpretation of the observed facial behavior has been acknowledged for over 20 years. For instance, it has been shown that the temporal dynamics of spontaneous and volitional smiles are fundamentally different from each other. In this work, we argue that the same holds for the temporal dynamics of brow actions and show that velocity, duration, and order of occurrence of brow actions are highly relevant parameters for distinguishing posed from spontaneous brow actions. The proposed system for discrimination between volitional and spontaneous brow actions is based on automatic detection of Action Units (AUs) and their temporal segments (onset, apex, offset) produced by movements of the eyebrows. For each temporal segment of an activated AU, we compute a number of mid-level feature parameters including the maximal intensity, duration, and order of occurrence. We use Gentle Boost to select the most important of these parameters. The selected parameters are used further to train Relevance Vector Machines to determine per temporal segment of an activated AU whether the action was displayed spontaneously or volitionally. Finally, a probabilistic decision function determines the class (spontaneous or posed) for the entire brow action. When tested on 189 samples taken from three different sets of spontaneous and volitional facial data, we attain a 90.7% correct recognition rate.

AB - Past research on automatic facial expression analysis has focused mostly on the recognition of prototypic expressions of discrete emotions rather than on the analysis of dynamic changes over time, although the importance of temporal dynamics of facial expressions for interpretation of the observed facial behavior has been acknowledged for over 20 years. For instance, it has been shown that the temporal dynamics of spontaneous and volitional smiles are fundamentally different from each other. In this work, we argue that the same holds for the temporal dynamics of brow actions and show that velocity, duration, and order of occurrence of brow actions are highly relevant parameters for distinguishing posed from spontaneous brow actions. The proposed system for discrimination between volitional and spontaneous brow actions is based on automatic detection of Action Units (AUs) and their temporal segments (onset, apex, offset) produced by movements of the eyebrows. For each temporal segment of an activated AU, we compute a number of mid-level feature parameters including the maximal intensity, duration, and order of occurrence. We use Gentle Boost to select the most important of these parameters. The selected parameters are used further to train Relevance Vector Machines to determine per temporal segment of an activated AU whether the action was displayed spontaneously or volitionally. Finally, a probabilistic decision function determines the class (spontaneous or posed) for the entire brow action. When tested on 189 samples taken from three different sets of spontaneous and volitional facial data, we attain a 90.7% correct recognition rate.

KW - HMI-MI: MULTIMODAL INTERACTIONS

KW - METIS-248237

KW - EWI-11628

KW - IR-62076

U2 - 10.1145/1180995.1181031

DO - 10.1145/1180995.1181031

M3 - Conference contribution

SN - 1-59593-541-X

SP - 162

EP - 170

BT - Proceedings of ACM Int'l Conf. Multimodal Interfaces (ICMI'06)

PB - Association for Computing Machinery (ACM)

CY - New York, NY, USA

ER -

Valstar MF, Pantic M, Ambadar Z, Cohn JF. Spontaneous vs. posed facial behavior: Automatic analysis of brow actions. In Proceedings of ACM Int'l Conf. Multimodal Interfaces (ICMI'06). New York, NY, USA: Association for Computing Machinery (ACM). 2006. p. 162-170. 10.1145/1180995.1181031 https://doi.org/10.1145/1180995.1181031