Brain-Computer Interfaces for Multimodal Interaction: A Survey and Principles

Hayrettin Gürkök, Antinus Nijholt

  • 37 Citations

Abstract

For decades, brain–computer interfaces (BCIs) have been used for restoring the communication and mobility of disabled people through applications such as spellers, web browsers, and wheelchair controls. In parallel to advances in computational intelligence and the production of consumer BCI products, BCIs have recently started to be considered as alternative modalities in human–computer interaction (HCI). One of the popular topics in HCI is multimodal interaction (MMI), which deals with combining multiple modalities in order to provide powerful, flexible, adaptable, and natural interfaces. This article discusses the situation of BCI as a modality within MMI research. State-of-the-art, real-time multimodal BCI applications are surveyed in order to demonstrate how BCI can be helpful as a modality in MMI. It is shown that multimodal use of BCIs can improve error handling, task performance, and user experience and that they can broaden the user spectrum. The techniques for employing BCI in MMI are described, and the experimental and technical challenges with some guidelines to overcome these are shown. Issues in input fusion, output fission, integration architectures, and data collection are covered.
Original languageUndefined
Pages (from-to)292-307
Number of pages16
JournalInternational journal of human-computer interaction
Volume28
Issue number5
DOIs
StatePublished - Apr 2012

Fingerprint

Brain computer interface
Human computer interaction
Wheelchairs
Web browsers
Artificial intelligence
Fusion reactions
Communication

Keywords

  • HMI-CI: Computational Intelligence
  • HMI-IA: Intelligent Agents
  • IR-80120
  • EWI-21740
  • METIS-286312
  • HMI-MI: MULTIMODAL INTERACTIONS

Cite this

Gürkök, Hayrettin; Nijholt, Antinus / Brain-Computer Interfaces for Multimodal Interaction: A Survey and Principles.

In: International journal of human-computer interaction, Vol. 28, No. 5, 04.2012, p. 292-307.

Research output: Scientific - peer-reviewArticle

@article{5919382b30b148f69c7faa1288c35279,
title = "Brain-Computer Interfaces for Multimodal Interaction: A Survey and Principles",
abstract = "For decades, brain–computer interfaces (BCIs) have been used for restoring the communication and mobility of disabled people through applications such as spellers, web browsers, and wheelchair controls. In parallel to advances in computational intelligence and the production of consumer BCI products, BCIs have recently started to be considered as alternative modalities in human–computer interaction (HCI). One of the popular topics in HCI is multimodal interaction (MMI), which deals with combining multiple modalities in order to provide powerful, flexible, adaptable, and natural interfaces. This article discusses the situation of BCI as a modality within MMI research. State-of-the-art, real-time multimodal BCI applications are surveyed in order to demonstrate how BCI can be helpful as a modality in MMI. It is shown that multimodal use of BCIs can improve error handling, task performance, and user experience and that they can broaden the user spectrum. The techniques for employing BCI in MMI are described, and the experimental and technical challenges with some guidelines to overcome these are shown. Issues in input fusion, output fission, integration architectures, and data collection are covered.",
keywords = "HMI-CI: Computational Intelligence, HMI-IA: Intelligent Agents, IR-80120, EWI-21740, METIS-286312, HMI-MI: MULTIMODAL INTERACTIONS",
author = "Hayrettin Gürkök and Antinus Nijholt",
note = "eemcs-eprint-21740",
year = "2012",
month = "4",
doi = "10.1080/10447318.2011.582022",
volume = "28",
pages = "292--307",
journal = "International journal of human-computer interaction",
issn = "1044-7318",
publisher = "Taylor & Francis",
number = "5",

}

Brain-Computer Interfaces for Multimodal Interaction: A Survey and Principles. / Gürkök, Hayrettin; Nijholt, Antinus.

In: International journal of human-computer interaction, Vol. 28, No. 5, 04.2012, p. 292-307.

Research output: Scientific - peer-reviewArticle

TY - JOUR

T1 - Brain-Computer Interfaces for Multimodal Interaction: A Survey and Principles

AU - Gürkök,Hayrettin

AU - Nijholt,Antinus

N1 - eemcs-eprint-21740

PY - 2012/4

Y1 - 2012/4

N2 - For decades, brain–computer interfaces (BCIs) have been used for restoring the communication and mobility of disabled people through applications such as spellers, web browsers, and wheelchair controls. In parallel to advances in computational intelligence and the production of consumer BCI products, BCIs have recently started to be considered as alternative modalities in human–computer interaction (HCI). One of the popular topics in HCI is multimodal interaction (MMI), which deals with combining multiple modalities in order to provide powerful, flexible, adaptable, and natural interfaces. This article discusses the situation of BCI as a modality within MMI research. State-of-the-art, real-time multimodal BCI applications are surveyed in order to demonstrate how BCI can be helpful as a modality in MMI. It is shown that multimodal use of BCIs can improve error handling, task performance, and user experience and that they can broaden the user spectrum. The techniques for employing BCI in MMI are described, and the experimental and technical challenges with some guidelines to overcome these are shown. Issues in input fusion, output fission, integration architectures, and data collection are covered.

AB - For decades, brain–computer interfaces (BCIs) have been used for restoring the communication and mobility of disabled people through applications such as spellers, web browsers, and wheelchair controls. In parallel to advances in computational intelligence and the production of consumer BCI products, BCIs have recently started to be considered as alternative modalities in human–computer interaction (HCI). One of the popular topics in HCI is multimodal interaction (MMI), which deals with combining multiple modalities in order to provide powerful, flexible, adaptable, and natural interfaces. This article discusses the situation of BCI as a modality within MMI research. State-of-the-art, real-time multimodal BCI applications are surveyed in order to demonstrate how BCI can be helpful as a modality in MMI. It is shown that multimodal use of BCIs can improve error handling, task performance, and user experience and that they can broaden the user spectrum. The techniques for employing BCI in MMI are described, and the experimental and technical challenges with some guidelines to overcome these are shown. Issues in input fusion, output fission, integration architectures, and data collection are covered.

KW - HMI-CI: Computational Intelligence

KW - HMI-IA: Intelligent Agents

KW - IR-80120

KW - EWI-21740

KW - METIS-286312

KW - HMI-MI: MULTIMODAL INTERACTIONS

U2 - 10.1080/10447318.2011.582022

DO - 10.1080/10447318.2011.582022

M3 - Article

VL - 28

SP - 292

EP - 307

JO - International journal of human-computer interaction

T2 - International journal of human-computer interaction

JF - International journal of human-computer interaction

SN - 1044-7318

IS - 5

ER -