Human behavior sensing for tag relevance assessment

M. Soleymani, S. Kaltwang, Maja Pantic

    Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

    3 Citations (Scopus)
    172 Downloads (Pure)


    Users react differently to non-relevant and relevant tags associated with content. These spontaneous reactions can be used for labeling large multimedia databases. We present a method to assess tag relevance to images using the non-verbal bodily responses, namely, electroencephalogram (EEG), facial expressions, and eye gaze. We conducted experiments in which 28 images were shown to 28 subjects once with correct and another time with incorrect tags. The goal of our system is to detect the responses to non-relevant tags and consequently filter them out. Therefore, we trained classifiers to detect the tag relevance from bodily responses. We evaluated the performance of our system using a subject independent approach. The precision at top 5% and top 10% detections were calculated and results of different modalities and different classifiers were compared. The results show that eye gaze outperforms the other modalities in tag relevance detection both overall and for top ranked results.
    Original languageUndefined
    Title of host publicationProceedings of the 21st ACM international conference on Multimedia, MM 2013
    Place of PublicationNew York
    PublisherAssociation for Computing Machinery
    Number of pages4
    ISBN (Print)978-1-4503-2404-5
    Publication statusPublished - Oct 2013
    Event21st ACM Multimedia Conference, MM 2013 - Barcelona, Spain
    Duration: 21 Oct 201325 Oct 2013
    Conference number: 21

    Publication series



    Conference21st ACM Multimedia Conference, MM 2013
    Abbreviated titleMM
    Internet address


    • EWI-24344
    • HMI-HF: Human Factors
    • EEG
    • METIS-302663
    • IR-89374
    • Implicit Tagging
    • Facial expressions
    • eye gaze

    Cite this