Emotion recognition with simulated phosphene vision

Caroline J.M. Bollen, Richard J.A. van Wezel, Marcel A.J. van Gerven, Yaǧmur Güçlütürk

Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

2 Citations (Scopus)


Electrical stimulation of retina, optic nerve or cortex is found to elicit visual sensations, known as phosphenes. This allows visual prosthetics to partially restore vision by representing the visual field as a phosphene pattern. Since the resolution and performance of visual prostheses are limited, only a fraction of the information in a visual scene can be represented by phosphenes. Here, we propose a simple yet powerful image processing strategy for recognizing facial expressions with prosthetic vision, supporting communication and social interaction in the blind. A psychophysical study was conducted to investigate whether a landmark-based representation of facial expressions could improve emotion detection with prosthetic vision. Our approach was compared to edge detection, which is commonly used in current retinal prosthetic devices. Additionally, the relationship between the number of phosphenes and accuracy of emotion recognition was studied. The landmark model improved accuracy of emotion recognition, regardless of the number of phosphenes. Secondly, the accuracy improved with an increasing number of phosphenes up to a saturation point. The performance saturated with fewer phosphenes with the landmark model than with edge detection. These results suggest that landmark-based image pre-processing allows for a more efficient use of the limited information that can be stored in a phosphene pattern, providing a route towards more meaningful and higher-quality perceptual experience in subjects with prosthetic vision.

Original languageEnglish
Title of host publicationMAHCI 2019
Subtitle of host publicationProceedings of the 2nd Workshop on Multimedia for Accessible Human Computer Interfaces, co-located with MM 2019
PublisherAssociation for Computing Machinery
Number of pages8
ISBN (Electronic)9781450369121
Publication statusPublished - 15 Oct 2019
Externally publishedYes
Event2nd Workshop on Multimedia for Accessible Human Computer Interfaces, MAHCI 2019 - Nice, France
Duration: 25 Oct 201925 Oct 2019
Conference number: 2


Workshop2nd Workshop on Multimedia for Accessible Human Computer Interfaces, MAHCI 2019
Abbreviated titleMAHCI


  • Emotion recognition
  • Facial landmark detection
  • Phosphene vision
  • Visual neuroprosthetics


Dive into the research topics of 'Emotion recognition with simulated phosphene vision'. Together they form a unique fingerprint.

Cite this