Conveying facial expressions to blind and visually impaired persons through a wearable vibrotactile device

Hendrik P. Buimer, Marian Bittner, Tjerk Kostelijk, Thea M. Van Der Geest, Abdellatif Nemri, Richard J.A. Van Wezel, Yan Zhao

Research output: Contribution to journalArticle

Abstract

In face-to-face social interactions, blind and visually impaired persons (VIPs) lack access to nonverbal cues like facial expressions, body posture, and gestures, which may lead to impaired interpersonal communication. In this study, a wearable sensory substitution device (SSD) consisting of a head mounted camera and a haptic belt was evaluated to determine whether vibrotactile cues around the waist could be used to convey facial expressions to users and whether such a device is desired by VIPs for use in daily living situations. Ten VIPs (mean age: 38.8, SD: 14.4) and 10 sighted persons (SPs) (mean age: 44.5, SD: 19.6) participated in the study, in which validated sets of pictures, silent videos, and videos with audio of facial expressions were presented to the participant. A control measurement was first performed to determine how accurately participants could identify facial expressions while relying on their functional senses. After a short training, participants were asked to determine facial expressions while wearing the emotion feedback system. VIPs using the device showed significant improvements in their ability to determine which facial expressions were shown. A significant increase in accuracy of 44.4% was found across all types of stimuli when comparing the scores of the control (mean±SEM: 35.0±2.5%) and supported (mean±SEM: 79.4±2.1%) phases. The greatest improvements achieved with the support of the SSD were found for silent stimuli (68.3% for pictures and 50.8% for silent videos). SPs also showed consistent, though not statistically significant, improvements while supported. Overall, our study shows that vibrotactile cues are well suited to convey facial expressions to VIPs in real-time. Participants became skilled with the device after a short training session. Further testing and development of the SSD is required to improve its accuracy and aesthetics for potential daily use.

LanguageEnglish
Article numbere0194737
JournalPLoS ONE
Volume13
Issue number3
DOIs
StatePublished - 1 Mar 2018

Fingerprint

Visually Impaired Persons
Facial Expression
Conveying
Substitution reactions
Equipment and Supplies
Scanning electron microscopy
Cues
Cameras
Feedback
communication skills
Communication
Testing
Gestures
Aptitude
waist
aesthetics
Interpersonal Relations
emotions
posture
Posture

Cite this

Buimer, H. P., Bittner, M., Kostelijk, T., Van Der Geest, T. M., Nemri, A., Van Wezel, R. J. A., & Zhao, Y. (2018). Conveying facial expressions to blind and visually impaired persons through a wearable vibrotactile device. PLoS ONE, 13(3), [e0194737]. DOI: 10.1371/journal.pone.0194737
Buimer, Hendrik P. ; Bittner, Marian ; Kostelijk, Tjerk ; Van Der Geest, Thea M. ; Nemri, Abdellatif ; Van Wezel, Richard J.A. ; Zhao, Yan. / Conveying facial expressions to blind and visually impaired persons through a wearable vibrotactile device. In: PLoS ONE. 2018 ; Vol. 13, No. 3.
@article{96b2067856d94f279a46ebc0037a6ea4,
title = "Conveying facial expressions to blind and visually impaired persons through a wearable vibrotactile device",
abstract = "In face-to-face social interactions, blind and visually impaired persons (VIPs) lack access to nonverbal cues like facial expressions, body posture, and gestures, which may lead to impaired interpersonal communication. In this study, a wearable sensory substitution device (SSD) consisting of a head mounted camera and a haptic belt was evaluated to determine whether vibrotactile cues around the waist could be used to convey facial expressions to users and whether such a device is desired by VIPs for use in daily living situations. Ten VIPs (mean age: 38.8, SD: 14.4) and 10 sighted persons (SPs) (mean age: 44.5, SD: 19.6) participated in the study, in which validated sets of pictures, silent videos, and videos with audio of facial expressions were presented to the participant. A control measurement was first performed to determine how accurately participants could identify facial expressions while relying on their functional senses. After a short training, participants were asked to determine facial expressions while wearing the emotion feedback system. VIPs using the device showed significant improvements in their ability to determine which facial expressions were shown. A significant increase in accuracy of 44.4{\%} was found across all types of stimuli when comparing the scores of the control (mean±SEM: 35.0±2.5{\%}) and supported (mean±SEM: 79.4±2.1{\%}) phases. The greatest improvements achieved with the support of the SSD were found for silent stimuli (68.3{\%} for pictures and 50.8{\%} for silent videos). SPs also showed consistent, though not statistically significant, improvements while supported. Overall, our study shows that vibrotactile cues are well suited to convey facial expressions to VIPs in real-time. Participants became skilled with the device after a short training session. Further testing and development of the SSD is required to improve its accuracy and aesthetics for potential daily use.",
author = "Buimer, {Hendrik P.} and Marian Bittner and Tjerk Kostelijk and {Van Der Geest}, {Thea M.} and Abdellatif Nemri and {Van Wezel}, {Richard J.A.} and Yan Zhao",
year = "2018",
month = "3",
day = "1",
doi = "10.1371/journal.pone.0194737",
language = "English",
volume = "13",
journal = "PLoS ONE",
issn = "1932-6203",
publisher = "Public Library of Science",
number = "3",

}

Buimer, HP, Bittner, M, Kostelijk, T, Van Der Geest, TM, Nemri, A, Van Wezel, RJA & Zhao, Y 2018, 'Conveying facial expressions to blind and visually impaired persons through a wearable vibrotactile device' PLoS ONE, vol 13, no. 3, e0194737. DOI: 10.1371/journal.pone.0194737

Conveying facial expressions to blind and visually impaired persons through a wearable vibrotactile device. / Buimer, Hendrik P.; Bittner, Marian; Kostelijk, Tjerk; Van Der Geest, Thea M.; Nemri, Abdellatif; Van Wezel, Richard J.A.; Zhao, Yan.

In: PLoS ONE, Vol. 13, No. 3, e0194737, 01.03.2018.

Research output: Contribution to journalArticle

TY - JOUR

T1 - Conveying facial expressions to blind and visually impaired persons through a wearable vibrotactile device

AU - Buimer,Hendrik P.

AU - Bittner,Marian

AU - Kostelijk,Tjerk

AU - Van Der Geest,Thea M.

AU - Nemri,Abdellatif

AU - Van Wezel,Richard J.A.

AU - Zhao,Yan

PY - 2018/3/1

Y1 - 2018/3/1

N2 - In face-to-face social interactions, blind and visually impaired persons (VIPs) lack access to nonverbal cues like facial expressions, body posture, and gestures, which may lead to impaired interpersonal communication. In this study, a wearable sensory substitution device (SSD) consisting of a head mounted camera and a haptic belt was evaluated to determine whether vibrotactile cues around the waist could be used to convey facial expressions to users and whether such a device is desired by VIPs for use in daily living situations. Ten VIPs (mean age: 38.8, SD: 14.4) and 10 sighted persons (SPs) (mean age: 44.5, SD: 19.6) participated in the study, in which validated sets of pictures, silent videos, and videos with audio of facial expressions were presented to the participant. A control measurement was first performed to determine how accurately participants could identify facial expressions while relying on their functional senses. After a short training, participants were asked to determine facial expressions while wearing the emotion feedback system. VIPs using the device showed significant improvements in their ability to determine which facial expressions were shown. A significant increase in accuracy of 44.4% was found across all types of stimuli when comparing the scores of the control (mean±SEM: 35.0±2.5%) and supported (mean±SEM: 79.4±2.1%) phases. The greatest improvements achieved with the support of the SSD were found for silent stimuli (68.3% for pictures and 50.8% for silent videos). SPs also showed consistent, though not statistically significant, improvements while supported. Overall, our study shows that vibrotactile cues are well suited to convey facial expressions to VIPs in real-time. Participants became skilled with the device after a short training session. Further testing and development of the SSD is required to improve its accuracy and aesthetics for potential daily use.

AB - In face-to-face social interactions, blind and visually impaired persons (VIPs) lack access to nonverbal cues like facial expressions, body posture, and gestures, which may lead to impaired interpersonal communication. In this study, a wearable sensory substitution device (SSD) consisting of a head mounted camera and a haptic belt was evaluated to determine whether vibrotactile cues around the waist could be used to convey facial expressions to users and whether such a device is desired by VIPs for use in daily living situations. Ten VIPs (mean age: 38.8, SD: 14.4) and 10 sighted persons (SPs) (mean age: 44.5, SD: 19.6) participated in the study, in which validated sets of pictures, silent videos, and videos with audio of facial expressions were presented to the participant. A control measurement was first performed to determine how accurately participants could identify facial expressions while relying on their functional senses. After a short training, participants were asked to determine facial expressions while wearing the emotion feedback system. VIPs using the device showed significant improvements in their ability to determine which facial expressions were shown. A significant increase in accuracy of 44.4% was found across all types of stimuli when comparing the scores of the control (mean±SEM: 35.0±2.5%) and supported (mean±SEM: 79.4±2.1%) phases. The greatest improvements achieved with the support of the SSD were found for silent stimuli (68.3% for pictures and 50.8% for silent videos). SPs also showed consistent, though not statistically significant, improvements while supported. Overall, our study shows that vibrotactile cues are well suited to convey facial expressions to VIPs in real-time. Participants became skilled with the device after a short training session. Further testing and development of the SSD is required to improve its accuracy and aesthetics for potential daily use.

UR - http://www.scopus.com/inward/record.url?scp=85044538054&partnerID=8YFLogxK

U2 - 10.1371/journal.pone.0194737

DO - 10.1371/journal.pone.0194737

M3 - Article

VL - 13

JO - PLoS ONE

T2 - PLoS ONE

JF - PLoS ONE

SN - 1932-6203

IS - 3

M1 - e0194737

ER -

Buimer HP, Bittner M, Kostelijk T, Van Der Geest TM, Nemri A, Van Wezel RJA et al. Conveying facial expressions to blind and visually impaired persons through a wearable vibrotactile device. PLoS ONE. 2018 Mar 1;13(3). e0194737. Available from, DOI: 10.1371/journal.pone.0194737