Implicit Image Tagging via Facial Information

Jun Jiao; Maja Pantic

doi:10.1145/1878116.1878133

Implicit Image Tagging via Facial Information

Jun Jiao, Maja Pantic

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

13 Citations (Scopus)

Abstract

Implicit Tagging is the technique to annotate multimedia data based on user’s spontaneous nonverbal reactions. In this paper, a study is conducted to test whether user’s facial expression can be used to predict the correctness of tags of images. The basic assumption behind this study is that users are likely to display certain kind of emotion due to the correctness of tags. The dataset used in this paper is users’ frontal face video collected during an implicit tagging experiment, in which participants were presented with tagged images and their facial reactions when viewing these images were recorded. Based on this dataset, facial points in video sequences are tracked by a facial point tracker. Geometric features are calculated from the positions of facial points to represent each video as a sequence of feature vectors, and Hidden Markov Models (HMM) are used to classify this information in terms of behavior typical for viewing a correctly or an incorrectly tagged image. Experimental results show that user’s facial expression can be used to help judge the correctness of tags. The proposed is effective in case of 16 out of 27 participants, the highest prediction accuracy for a single participant being 72.1%, and the highest overall accuracy being 77.98%.

Original language	Undefined
Title of host publication	Proceedings of the 2nd International Workshop on Social Signal Processing, SSPW 2010
Place of Publication	New York
Publisher	Association for Computing Machinery
Pages	59-64
Number of pages	6
ISBN (Print)	978-1-4503-0174-9
DOIs	https://doi.org/10.1145/1878116.1878133
Publication status	Published - 29 Oct 2010
Event	2nd International Workshop on Social Signal Processing, SSPW 2010 - Firenze, Italy Duration: 21 Oct 2010 → 21 Oct 2010

Publication series

Name
Publisher	ACM

Workshop

Workshop	2nd International Workshop on Social Signal Processing, SSPW 2010
Period	21/10/10 → 21/10/10
Other	21 October 2010

Keywords

IR-75890
METIS-276329
Implicit Tagging
ensemble learning
EWI-19466
EC Grant Agreement nr.: FP7/231287
image tagging
facial feature extraction
HMI-MI: MULTIMODAL INTERACTIONS
dynamic classification

Access to Document

10.1145/1878116.1878133

http://eprints.eemcs.utwente.nl/secure2/19466/01/pantic_implicit_image_tagging.pdf

Cite this

@inproceedings{88f39835e1f543f4a2d53359f8b8b567,

title = "Implicit Image Tagging via Facial Information",

abstract = "Implicit Tagging is the technique to annotate multimedia data based on user{\textquoteright}s spontaneous nonverbal reactions. In this paper, a study is conducted to test whether user{\textquoteright}s facial expression can be used to predict the correctness of tags of images. The basic assumption behind this study is that users are likely to display certain kind of emotion due to the correctness of tags. The dataset used in this paper is users{\textquoteright} frontal face video collected during an implicit tagging experiment, in which participants were presented with tagged images and their facial reactions when viewing these images were recorded. Based on this dataset, facial points in video sequences are tracked by a facial point tracker. Geometric features are calculated from the positions of facial points to represent each video as a sequence of feature vectors, and Hidden Markov Models (HMM) are used to classify this information in terms of behavior typical for viewing a correctly or an incorrectly tagged image. Experimental results show that user{\textquoteright}s facial expression can be used to help judge the correctness of tags. The proposed is effective in case of 16 out of 27 participants, the highest prediction accuracy for a single participant being 72.1%, and the highest overall accuracy being 77.98%.",

keywords = "IR-75890, METIS-276329, Implicit Tagging, ensemble learning, EWI-19466, EC Grant Agreement nr.: FP7/231287, image tagging, facial feature extraction, HMI-MI: MULTIMODAL INTERACTIONS, dynamic classification",

author = "Jun Jiao and Maja Pantic",

note = "10.1145/1878116.1878133 ; null ; Conference date: 21-10-2010 Through 21-10-2010",

year = "2010",

month = oct,

day = "29",

doi = "10.1145/1878116.1878133",

language = "Undefined",

isbn = "978-1-4503-0174-9",

publisher = "Association for Computing Machinery",

pages = "59--64",

booktitle = "Proceedings of the 2nd International Workshop on Social Signal Processing, SSPW 2010",

}

TY - GEN

T1 - Implicit Image Tagging via Facial Information

AU - Jiao, Jun

AU - Pantic, Maja

N1 - 10.1145/1878116.1878133

PY - 2010/10/29

Y1 - 2010/10/29

N2 - Implicit Tagging is the technique to annotate multimedia data based on user’s spontaneous nonverbal reactions. In this paper, a study is conducted to test whether user’s facial expression can be used to predict the correctness of tags of images. The basic assumption behind this study is that users are likely to display certain kind of emotion due to the correctness of tags. The dataset used in this paper is users’ frontal face video collected during an implicit tagging experiment, in which participants were presented with tagged images and their facial reactions when viewing these images were recorded. Based on this dataset, facial points in video sequences are tracked by a facial point tracker. Geometric features are calculated from the positions of facial points to represent each video as a sequence of feature vectors, and Hidden Markov Models (HMM) are used to classify this information in terms of behavior typical for viewing a correctly or an incorrectly tagged image. Experimental results show that user’s facial expression can be used to help judge the correctness of tags. The proposed is effective in case of 16 out of 27 participants, the highest prediction accuracy for a single participant being 72.1%, and the highest overall accuracy being 77.98%.

AB - Implicit Tagging is the technique to annotate multimedia data based on user’s spontaneous nonverbal reactions. In this paper, a study is conducted to test whether user’s facial expression can be used to predict the correctness of tags of images. The basic assumption behind this study is that users are likely to display certain kind of emotion due to the correctness of tags. The dataset used in this paper is users’ frontal face video collected during an implicit tagging experiment, in which participants were presented with tagged images and their facial reactions when viewing these images were recorded. Based on this dataset, facial points in video sequences are tracked by a facial point tracker. Geometric features are calculated from the positions of facial points to represent each video as a sequence of feature vectors, and Hidden Markov Models (HMM) are used to classify this information in terms of behavior typical for viewing a correctly or an incorrectly tagged image. Experimental results show that user’s facial expression can be used to help judge the correctness of tags. The proposed is effective in case of 16 out of 27 participants, the highest prediction accuracy for a single participant being 72.1%, and the highest overall accuracy being 77.98%.

KW - IR-75890

KW - METIS-276329

KW - Implicit Tagging

KW - ensemble learning

KW - EWI-19466

KW - EC Grant Agreement nr.: FP7/231287

KW - image tagging

KW - facial feature extraction

KW - HMI-MI: MULTIMODAL INTERACTIONS

KW - dynamic classification

U2 - 10.1145/1878116.1878133

DO - 10.1145/1878116.1878133

M3 - Conference contribution

SN - 978-1-4503-0174-9

SP - 59

EP - 64

BT - Proceedings of the 2nd International Workshop on Social Signal Processing, SSPW 2010

PB - Association for Computing Machinery

CY - New York

Y2 - 21 October 2010 through 21 October 2010

ER -