Implicit Image Tagging via Facial Information

Jun Jiao, Maja Pantic

    Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

    13 Citations (Scopus)


    Implicit Tagging is the technique to annotate multimedia data based on user’s spontaneous nonverbal reactions. In this paper, a study is conducted to test whether user’s facial expression can be used to predict the correctness of tags of images. The basic assumption behind this study is that users are likely to display certain kind of emotion due to the correctness of tags. The dataset used in this paper is users’ frontal face video collected during an implicit tagging experiment, in which participants were presented with tagged images and their facial reactions when viewing these images were recorded. Based on this dataset, facial points in video sequences are tracked by a facial point tracker. Geometric features are calculated from the positions of facial points to represent each video as a sequence of feature vectors, and Hidden Markov Models (HMM) are used to classify this information in terms of behavior typical for viewing a correctly or an incorrectly tagged image. Experimental results show that user’s facial expression can be used to help judge the correctness of tags. The proposed is effective in case of 16 out of 27 participants, the highest prediction accuracy for a single participant being 72.1%, and the highest overall accuracy being 77.98%.
    Original languageUndefined
    Title of host publicationProceedings of the 2nd International Workshop on Social Signal Processing, SSPW 2010
    Place of PublicationNew York
    PublisherAssociation for Computing Machinery
    Number of pages6
    ISBN (Print)978-1-4503-0174-9
    Publication statusPublished - 29 Oct 2010
    Event2nd International Workshop on Social Signal Processing, SSPW 2010 - Firenze, Italy
    Duration: 21 Oct 201021 Oct 2010

    Publication series



    Workshop2nd International Workshop on Social Signal Processing, SSPW 2010
    Other21 October 2010


    • IR-75890
    • METIS-276329
    • Implicit Tagging
    • ensemble learning
    • EWI-19466
    • EC Grant Agreement nr.: FP7/231287
    • image tagging
    • facial feature extraction
    • dynamic classification

    Cite this