Output-associative RVM regression for dimensional and continuous emotion prediction

Mihalis A. Nicolaou, Hatice Gunes, Maja Pantic

Research output: Contribution to journalArticleAcademicpeer-review

64 Citations (Scopus)

Abstract

Many problems in machine learning and computer vision consist of predicting multi-dimensional output vectors given a specific set of input features. In many of these problems, there exist inherent temporal and spatial dependencies between the output vectors, as well as repeating output patterns and input–output associations, that can provide more robust and accurate predictors when modeled properly. With this intrinsic motivation, we propose a novel Output-Associative Relevance Vector Machine (OA-RVM) regression framework that augments the traditional RVM regression by being able to learn non-linear input and output dependencies. Instead of depending solely on the input patterns, OA-RVM models output covariances within a predefined temporal window, thus capturing past, current and future context. As a result, output patterns manifested in the training data are captured within a formal probabilistic framework, and subsequently used during inference. As a proof of concept, we target the highly challenging problem of dimensional and continuous prediction of emotions, and evaluate the proposed framework by focusing on the case of multiple nonverbal cues, namely facial expressions, shoulder movements and audio cues. We demonstrate the advantages of the proposed OA-RVM regression by performing subject-independent evaluation using the SAL database that constitutes naturalistic conversational interactions. The experimental results show that OA-RVM regression outperforms the traditional RVM and SVM regression approaches in terms of accuracy of the prediction (evaluated using the Root Mean Squared Error) and structure of the prediction (evaluated using the correlation coefficient), generating more accurate and robust prediction models.
Original languageUndefined
Pages (from-to)186-196
Number of pages11
JournalImage and vision computing
Volume30
Issue number10
DOIs
Publication statusPublished - Oct 2012

Keywords

  • HMI-MI: MULTIMODAL INTERACTIONS
  • EWI-22939
  • Audio cues
  • Dimensional and continuous emotion prediction
  • IR-84217
  • Shoulder movements
  • Output-associative RVM regression
  • METIS-296242
  • Facial expressions

Cite this

Nicolaou, Mihalis A. ; Gunes, Hatice ; Pantic, Maja. / Output-associative RVM regression for dimensional and continuous emotion prediction. In: Image and vision computing. 2012 ; Vol. 30, No. 10. pp. 186-196.
@article{d1846d6ace604a3586ef00f2b6f0ee5e,
title = "Output-associative RVM regression for dimensional and continuous emotion prediction",
abstract = "Many problems in machine learning and computer vision consist of predicting multi-dimensional output vectors given a specific set of input features. In many of these problems, there exist inherent temporal and spatial dependencies between the output vectors, as well as repeating output patterns and input–output associations, that can provide more robust and accurate predictors when modeled properly. With this intrinsic motivation, we propose a novel Output-Associative Relevance Vector Machine (OA-RVM) regression framework that augments the traditional RVM regression by being able to learn non-linear input and output dependencies. Instead of depending solely on the input patterns, OA-RVM models output covariances within a predefined temporal window, thus capturing past, current and future context. As a result, output patterns manifested in the training data are captured within a formal probabilistic framework, and subsequently used during inference. As a proof of concept, we target the highly challenging problem of dimensional and continuous prediction of emotions, and evaluate the proposed framework by focusing on the case of multiple nonverbal cues, namely facial expressions, shoulder movements and audio cues. We demonstrate the advantages of the proposed OA-RVM regression by performing subject-independent evaluation using the SAL database that constitutes naturalistic conversational interactions. The experimental results show that OA-RVM regression outperforms the traditional RVM and SVM regression approaches in terms of accuracy of the prediction (evaluated using the Root Mean Squared Error) and structure of the prediction (evaluated using the correlation coefficient), generating more accurate and robust prediction models.",
keywords = "HMI-MI: MULTIMODAL INTERACTIONS, EWI-22939, Audio cues, Dimensional and continuous emotion prediction, IR-84217, Shoulder movements, Output-associative RVM regression, METIS-296242, Facial expressions",
author = "Nicolaou, {Mihalis A.} and Hatice Gunes and Maja Pantic",
note = "eemcs-eprint-22939",
year = "2012",
month = "10",
doi = "10.1016/j.imavis.2011.12.005",
language = "Undefined",
volume = "30",
pages = "186--196",
journal = "Image and vision computing",
issn = "0262-8856",
publisher = "Elsevier",
number = "10",

}

Output-associative RVM regression for dimensional and continuous emotion prediction. / Nicolaou, Mihalis A.; Gunes, Hatice; Pantic, Maja.

In: Image and vision computing, Vol. 30, No. 10, 10.2012, p. 186-196.

Research output: Contribution to journalArticleAcademicpeer-review

TY - JOUR

T1 - Output-associative RVM regression for dimensional and continuous emotion prediction

AU - Nicolaou, Mihalis A.

AU - Gunes, Hatice

AU - Pantic, Maja

N1 - eemcs-eprint-22939

PY - 2012/10

Y1 - 2012/10

N2 - Many problems in machine learning and computer vision consist of predicting multi-dimensional output vectors given a specific set of input features. In many of these problems, there exist inherent temporal and spatial dependencies between the output vectors, as well as repeating output patterns and input–output associations, that can provide more robust and accurate predictors when modeled properly. With this intrinsic motivation, we propose a novel Output-Associative Relevance Vector Machine (OA-RVM) regression framework that augments the traditional RVM regression by being able to learn non-linear input and output dependencies. Instead of depending solely on the input patterns, OA-RVM models output covariances within a predefined temporal window, thus capturing past, current and future context. As a result, output patterns manifested in the training data are captured within a formal probabilistic framework, and subsequently used during inference. As a proof of concept, we target the highly challenging problem of dimensional and continuous prediction of emotions, and evaluate the proposed framework by focusing on the case of multiple nonverbal cues, namely facial expressions, shoulder movements and audio cues. We demonstrate the advantages of the proposed OA-RVM regression by performing subject-independent evaluation using the SAL database that constitutes naturalistic conversational interactions. The experimental results show that OA-RVM regression outperforms the traditional RVM and SVM regression approaches in terms of accuracy of the prediction (evaluated using the Root Mean Squared Error) and structure of the prediction (evaluated using the correlation coefficient), generating more accurate and robust prediction models.

AB - Many problems in machine learning and computer vision consist of predicting multi-dimensional output vectors given a specific set of input features. In many of these problems, there exist inherent temporal and spatial dependencies between the output vectors, as well as repeating output patterns and input–output associations, that can provide more robust and accurate predictors when modeled properly. With this intrinsic motivation, we propose a novel Output-Associative Relevance Vector Machine (OA-RVM) regression framework that augments the traditional RVM regression by being able to learn non-linear input and output dependencies. Instead of depending solely on the input patterns, OA-RVM models output covariances within a predefined temporal window, thus capturing past, current and future context. As a result, output patterns manifested in the training data are captured within a formal probabilistic framework, and subsequently used during inference. As a proof of concept, we target the highly challenging problem of dimensional and continuous prediction of emotions, and evaluate the proposed framework by focusing on the case of multiple nonverbal cues, namely facial expressions, shoulder movements and audio cues. We demonstrate the advantages of the proposed OA-RVM regression by performing subject-independent evaluation using the SAL database that constitutes naturalistic conversational interactions. The experimental results show that OA-RVM regression outperforms the traditional RVM and SVM regression approaches in terms of accuracy of the prediction (evaluated using the Root Mean Squared Error) and structure of the prediction (evaluated using the correlation coefficient), generating more accurate and robust prediction models.

KW - HMI-MI: MULTIMODAL INTERACTIONS

KW - EWI-22939

KW - Audio cues

KW - Dimensional and continuous emotion prediction

KW - IR-84217

KW - Shoulder movements

KW - Output-associative RVM regression

KW - METIS-296242

KW - Facial expressions

U2 - 10.1016/j.imavis.2011.12.005

DO - 10.1016/j.imavis.2011.12.005

M3 - Article

VL - 30

SP - 186

EP - 196

JO - Image and vision computing

JF - Image and vision computing

SN - 0262-8856

IS - 10

ER -