Modality switching and performance in a thought and speech controlled computer game

Hayrettin Gürkök; Gido Hakvoort; Mannes Poel

doi:10.1145/2070481.2070491

Modality switching and performance in a thought and speech controlled computer game

Hayrettin Gürkök, Gido Hakvoort, Mannes Poel

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

14 Citations (Scopus)

11 Downloads (Pure)

Abstract

Providing multiple modalities to users is known to improve the overall performance of an interface. Weakness of one modality can be overcome by the strength of another one. Moreover, with respect to their abilities, users can choose between the modalities to use the one that is the best for them. In this paper we explored whether this holds for direct control of a computer game which can be played using a brain-computer interface (BCI) and an automatic speech recogniser (ASR). Participants played the games in unimodal mode (i.e. ASR-only and BCI-only) and multimodal mode where they could switch between the two modalities. The majority of the participants switched modality during the multimodal game but for the most of the time they stayed in ASR control. Therefore multimodality did not provide a significant performance improvement over unimodal control in our particular setup. We also investigated the factors which influence modality switching. We found that performance and peformance-related factors were prominently effective in modality switching.

Original language	Undefined
Title of host publication	Proceedings of the 13th International Conference on Multimodal Interfaces (ICMI 2011)
Editors	Hervé Bourlard, Thomas S. Huang, Enrique Vidal, Daniel Gatica-Perez, Louis-Philippe Morency, Nicu Sebe
Place of Publication	New York
Publisher	Association for Computing Machinery
Pages	41-48
Number of pages	8
ISBN (Print)	978-1-4503-0641-6
DOIs	https://doi.org/10.1145/2070481.2070491
Publication status	Published - Nov 2011
Event	13th International Conference on Multimodal Interfaces, ICMI 2011 - Alicante, Spain Duration: 14 Nov 2011 → 18 Nov 2011 Conference number: 13

Publication series

Name
Publisher	ACM

Conference

Conference	13th International Conference on Multimodal Interfaces, ICMI 2011
Abbreviated title	ICMI
Country/Territory	Spain
City	Alicante
Period	14/11/11 → 18/11/11

Keywords

IR-78612
METIS-281572
EWI-20821
HMI-MI: MULTIMODAL INTERACTIONS
HMI-CI: Computational Intelligence
HMI-SLT: Speech and Language Technology

Access to Document

10.1145/2070481.2070491

http://eprints.eemcs.utwente.nl/secure2/20821/01/p41-gurkok.pdf

Cite this

Gürkök, H., Hakvoort, G., & Poel, M. (2011). Modality switching and performance in a thought and speech controlled computer game. In H. Bourlard, T. S. Huang, E. Vidal, D. Gatica-Perez, L.-P. Morency, & N. Sebe (Eds.), Proceedings of the 13th International Conference on Multimodal Interfaces (ICMI 2011) (pp. 41-48). Association for Computing Machinery. https://doi.org/10.1145/2070481.2070491

Gürkök, Hayrettin ; Hakvoort, Gido ; Poel, Mannes. / Modality switching and performance in a thought and speech controlled computer game. Proceedings of the 13th International Conference on Multimodal Interfaces (ICMI 2011). editor / Hervé Bourlard ; Thomas S. Huang ; Enrique Vidal ; Daniel Gatica-Perez ; Louis-Philippe Morency ; Nicu Sebe. New York : Association for Computing Machinery, 2011. pp. 41-48

@inproceedings{f5f5fb476cae44c8802720b84032e6f1,

title = "Modality switching and performance in a thought and speech controlled computer game",

abstract = "Providing multiple modalities to users is known to improve the overall performance of an interface. Weakness of one modality can be overcome by the strength of another one. Moreover, with respect to their abilities, users can choose between the modalities to use the one that is the best for them. In this paper we explored whether this holds for direct control of a computer game which can be played using a brain-computer interface (BCI) and an automatic speech recogniser (ASR). Participants played the games in unimodal mode (i.e. ASR-only and BCI-only) and multimodal mode where they could switch between the two modalities. The majority of the participants switched modality during the multimodal game but for the most of the time they stayed in ASR control. Therefore multimodality did not provide a significant performance improvement over unimodal control in our particular setup. We also investigated the factors which influence modality switching. We found that performance and peformance-related factors were prominently effective in modality switching.",

keywords = "IR-78612, METIS-281572, EWI-20821, HMI-MI: MULTIMODAL INTERACTIONS, HMI-CI: Computational Intelligence, HMI-SLT: Speech and Language Technology",

author = "Hayrettin G{\"u}rk{\"o}k and Gido Hakvoort and Mannes Poel",

note = "10.1145/2070481.2070491 ; 13th International Conference on Multimodal Interfaces, ICMI 2011 ; Conference date: 14-11-2011 Through 18-11-2011",

year = "2011",

month = nov,

doi = "10.1145/2070481.2070491",

language = "Undefined",

isbn = "978-1-4503-0641-6",

publisher = "Association for Computing Machinery",

pages = "41--48",

editor = "Herv{\'e} Bourlard and Huang, {Thomas S.} and Enrique Vidal and Daniel Gatica-Perez and Louis-Philippe Morency and Nicu Sebe",

booktitle = "Proceedings of the 13th International Conference on Multimodal Interfaces (ICMI 2011)",

address = "United States",

}

Gürkök, H, Hakvoort, G & Poel, M 2011, Modality switching and performance in a thought and speech controlled computer game. in H Bourlard, TS Huang, E Vidal, D Gatica-Perez, L-P Morency & N Sebe (eds), Proceedings of the 13th International Conference on Multimodal Interfaces (ICMI 2011). Association for Computing Machinery, New York, pp. 41-48, 13th International Conference on Multimodal Interfaces, ICMI 2011, Alicante, Spain, 14/11/11. https://doi.org/10.1145/2070481.2070491

Modality switching and performance in a thought and speech controlled computer game. / Gürkök, Hayrettin; Hakvoort, Gido; Poel, Mannes.
Proceedings of the 13th International Conference on Multimodal Interfaces (ICMI 2011). ed. / Hervé Bourlard; Thomas S. Huang; Enrique Vidal; Daniel Gatica-Perez; Louis-Philippe Morency; Nicu Sebe. New York: Association for Computing Machinery, 2011. p. 41-48.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

TY - GEN

T1 - Modality switching and performance in a thought and speech controlled computer game

AU - Gürkök, Hayrettin

AU - Hakvoort, Gido

AU - Poel, Mannes

N1 - Conference code: 13

PY - 2011/11

Y1 - 2011/11

N2 - Providing multiple modalities to users is known to improve the overall performance of an interface. Weakness of one modality can be overcome by the strength of another one. Moreover, with respect to their abilities, users can choose between the modalities to use the one that is the best for them. In this paper we explored whether this holds for direct control of a computer game which can be played using a brain-computer interface (BCI) and an automatic speech recogniser (ASR). Participants played the games in unimodal mode (i.e. ASR-only and BCI-only) and multimodal mode where they could switch between the two modalities. The majority of the participants switched modality during the multimodal game but for the most of the time they stayed in ASR control. Therefore multimodality did not provide a significant performance improvement over unimodal control in our particular setup. We also investigated the factors which influence modality switching. We found that performance and peformance-related factors were prominently effective in modality switching.

AB - Providing multiple modalities to users is known to improve the overall performance of an interface. Weakness of one modality can be overcome by the strength of another one. Moreover, with respect to their abilities, users can choose between the modalities to use the one that is the best for them. In this paper we explored whether this holds for direct control of a computer game which can be played using a brain-computer interface (BCI) and an automatic speech recogniser (ASR). Participants played the games in unimodal mode (i.e. ASR-only and BCI-only) and multimodal mode where they could switch between the two modalities. The majority of the participants switched modality during the multimodal game but for the most of the time they stayed in ASR control. Therefore multimodality did not provide a significant performance improvement over unimodal control in our particular setup. We also investigated the factors which influence modality switching. We found that performance and peformance-related factors were prominently effective in modality switching.

KW - IR-78612

KW - METIS-281572

KW - EWI-20821

KW - HMI-MI: MULTIMODAL INTERACTIONS

KW - HMI-CI: Computational Intelligence

KW - HMI-SLT: Speech and Language Technology

U2 - 10.1145/2070481.2070491

DO - 10.1145/2070481.2070491

M3 - Conference contribution

SN - 978-1-4503-0641-6

SP - 41

EP - 48

BT - Proceedings of the 13th International Conference on Multimodal Interfaces (ICMI 2011)

A2 - Bourlard, Hervé

A2 - Huang, Thomas S.

A2 - Vidal, Enrique

A2 - Gatica-Perez, Daniel

A2 - Morency, Louis-Philippe

A2 - Sebe, Nicu

PB - Association for Computing Machinery

CY - New York

T2 - 13th International Conference on Multimodal Interfaces, ICMI 2011

Y2 - 14 November 2011 through 18 November 2011

ER -

Gürkök H, Hakvoort G, Poel M. Modality switching and performance in a thought and speech controlled computer game. In Bourlard H, Huang TS, Vidal E, Gatica-Perez D, Morency LP, Sebe N, editors, Proceedings of the 13th International Conference on Multimodal Interfaces (ICMI 2011). New York: Association for Computing Machinery. 2011. p. 41-48 doi: 10.1145/2070481.2070491