Enhancing emotion recognition in VIPs with haptic feedback

Hendrik Buimer, Marian Bittner, Tjerk Kostelijk, Thea van der Geest, Richard Jack Anton van Wezel, Yan Zhao

    Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

    12 Citations (Scopus)


    The rise of smart technologies has created new opportunities to support blind and visually impaired persons (VIPs). One of the biggest problems we identified in our previous research on problems VIPs face during activities of daily life concerned the recognition of persons and their facial expressions. In this study we developed a system to detect faces, recognize their emotions, and provide vibrotactile feedback about the emotions expressed. The prototype system was tested to determine whether vibrotactile feedback through a haptic belt is capable of enhancing social interactions for VIPs. The system consisted of commercially available technologies. A Logitech C920 webcam mounted on a cap, a Microsoft Surface Pro 4 carried in a mesh backpack, an Elitac tactile belt worn around the waist, and the VicarVision FaceReader software application, which recognizes facial expressions. In preliminary tests with the systems both visually impaired and sighted persons were presented with sets of stimuli consisting of actors displaying six emotions (e.g. joy, surprise, anger, sadness, fear, and disgust) derived from the validated Amsterdam Dynamic Facial Expression Set and Warsaw Set of Emotional Facial Expression Pictures with matching audio by using nonlinguistic affect bursts. Subjects had to determine the emotions expressed in the videos without and, after a training period, with haptic feedback. An exit survey was conducted aimed to gain insights into the opinion of the users, on the perceived usefulness and benefits of the emotional feedback, and their willingness of using the prototype as assistive technology in daily life. Haptic feedback about facial expressions may improve the ability of VIPs to determine emotions expressed by others and, as a result, increase the confidence of VIPs during social interactions. More studies are needed to determine whether this is a viable method to convey information and enhance social interactions in the daily life of VIPs.
    Original languageEnglish
    Title of host publicationInternational Conference on Human-Computer Interaction, HCI 2016
    EditorsConstantine Stephanidis
    Place of PublicationSwitzerland
    Number of pages7
    ISBN (Electronic)978-3-319-40542-1
    ISBN (Print)978-3-319-40541-4
    Publication statusPublished - 22 Jun 2016
    Event18th International Conference on Human-Computer Interaction, HCI International 2016 - Toronto, Canada
    Duration: 17 Jul 201622 Jul 2016
    Conference number: 18

    Publication series

    NameCommunications in Computer and Information Science
    ISSN (Print)1865-0929


    Conference18th International Conference on Human-Computer Interaction, HCI International 2016
    Abbreviated titleHCI International
    Internet address


    • EWI-27374
    • User centered design
    • IR-104022
    • Wearables
    • METIS-317691
    • Sensory substitution


    Dive into the research topics of 'Enhancing emotion recognition in VIPs with haptic feedback'. Together they form a unique fingerprint.

    Cite this