Implicit Image Tagging via Facial Information

Jun Jiao, Maja Pantic

    Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

    13 Citations (Scopus)

    Abstract

    Implicit Tagging is the technique to annotate multimedia data based on user’s spontaneous nonverbal reactions. In this paper, a study is conducted to test whether user’s facial expression can be used to predict the correctness of tags of images. The basic assumption behind this study is that users are likely to display certain kind of emotion due to the correctness of tags. The dataset used in this paper is users’ frontal face video collected during an implicit tagging experiment, in which participants were presented with tagged images and their facial reactions when viewing these images were recorded. Based on this dataset, facial points in video sequences are tracked by a facial point tracker. Geometric features are calculated from the positions of facial points to represent each video as a sequence of feature vectors, and Hidden Markov Models (HMM) are used to classify this information in terms of behavior typical for viewing a correctly or an incorrectly tagged image. Experimental results show that user’s facial expression can be used to help judge the correctness of tags. The proposed is effective in case of 16 out of 27 participants, the highest prediction accuracy for a single participant being 72.1%, and the highest overall accuracy being 77.98%.
    Original languageUndefined
    Title of host publicationProceedings of the 2nd International Workshop on Social Signal Processing, SSPW 2010
    Place of PublicationNew York
    PublisherAssociation for Computing Machinery (ACM)
    Pages59-64
    Number of pages6
    ISBN (Print)978-1-4503-0174-9
    DOIs
    Publication statusPublished - 29 Oct 2010

    Publication series

    Name
    PublisherACM

    Keywords

    • IR-75890
    • METIS-276329
    • Implicit Tagging
    • ensemble learning
    • EWI-19466
    • EC Grant Agreement nr.: FP7/231287
    • image tagging
    • facial feature extraction
    • HMI-MI: MULTIMODAL INTERACTIONS
    • dynamic classification

    Cite this

    Jiao, J., & Pantic, M. (2010). Implicit Image Tagging via Facial Information. In Proceedings of the 2nd International Workshop on Social Signal Processing, SSPW 2010 (pp. 59-64). New York: Association for Computing Machinery (ACM). https://doi.org/10.1145/1878116.1878133