Modality switching and performance in a thought and speech controlled computer game

Hayrettin Gürkök, Gido Hakvoort, Mannes Poel

    Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

    11 Citations (Scopus)

    Abstract

    Providing multiple modalities to users is known to improve the overall performance of an interface. Weakness of one modality can be overcome by the strength of another one. Moreover, with respect to their abilities, users can choose between the modalities to use the one that is the best for them. In this paper we explored whether this holds for direct control of a computer game which can be played using a brain-computer interface (BCI) and an automatic speech recogniser (ASR). Participants played the games in unimodal mode (i.e. ASR-only and BCI-only) and multimodal mode where they could switch between the two modalities. The majority of the participants switched modality during the multimodal game but for the most of the time they stayed in ASR control. Therefore multimodality did not provide a significant performance improvement over unimodal control in our particular setup. We also investigated the factors which influence modality switching. We found that performance and peformance-related factors were prominently effective in modality switching.
    Original languageUndefined
    Title of host publicationProceedings of the 13th International Conference on Multimodal Interfaces (ICMI 2011)
    EditorsHervé Bourlard, Thomas S. Huang, Enrique Vidal, Daniel Gatica-Perez, Louis-Philippe Morency, Nicu Sebe
    Place of PublicationNew York
    PublisherAssociation for Computing Machinery (ACM)
    Pages41-48
    Number of pages8
    ISBN (Print)978-1-4503-0641-6
    DOIs
    Publication statusPublished - Nov 2011
    Event13th International Conference on Multimodal Interfaces, ICMI 2011 - Alicante, Spain
    Duration: 14 Nov 201118 Nov 2011
    Conference number: 13

    Publication series

    Name
    PublisherACM

    Conference

    Conference13th International Conference on Multimodal Interfaces, ICMI 2011
    Abbreviated titleICMI
    CountrySpain
    CityAlicante
    Period14/11/1118/11/11

    Keywords

    • IR-78612
    • METIS-281572
    • EWI-20821
    • HMI-MI: MULTIMODAL INTERACTIONS
    • HMI-CI: Computational Intelligence
    • HMI-SLT: Speech and Language Technology

    Cite this

    Gürkök, H., Hakvoort, G., & Poel, M. (2011). Modality switching and performance in a thought and speech controlled computer game. In H. Bourlard, T. S. Huang, E. Vidal, D. Gatica-Perez, L-P. Morency, & N. Sebe (Eds.), Proceedings of the 13th International Conference on Multimodal Interfaces (ICMI 2011) (pp. 41-48). New York: Association for Computing Machinery (ACM). https://doi.org/10.1145/2070481.2070491
    Gürkök, Hayrettin ; Hakvoort, Gido ; Poel, Mannes. / Modality switching and performance in a thought and speech controlled computer game. Proceedings of the 13th International Conference on Multimodal Interfaces (ICMI 2011). editor / Hervé Bourlard ; Thomas S. Huang ; Enrique Vidal ; Daniel Gatica-Perez ; Louis-Philippe Morency ; Nicu Sebe. New York : Association for Computing Machinery (ACM), 2011. pp. 41-48
    @inproceedings{f5f5fb476cae44c8802720b84032e6f1,
    title = "Modality switching and performance in a thought and speech controlled computer game",
    abstract = "Providing multiple modalities to users is known to improve the overall performance of an interface. Weakness of one modality can be overcome by the strength of another one. Moreover, with respect to their abilities, users can choose between the modalities to use the one that is the best for them. In this paper we explored whether this holds for direct control of a computer game which can be played using a brain-computer interface (BCI) and an automatic speech recogniser (ASR). Participants played the games in unimodal mode (i.e. ASR-only and BCI-only) and multimodal mode where they could switch between the two modalities. The majority of the participants switched modality during the multimodal game but for the most of the time they stayed in ASR control. Therefore multimodality did not provide a significant performance improvement over unimodal control in our particular setup. We also investigated the factors which influence modality switching. We found that performance and peformance-related factors were prominently effective in modality switching.",
    keywords = "IR-78612, METIS-281572, EWI-20821, HMI-MI: MULTIMODAL INTERACTIONS, HMI-CI: Computational Intelligence, HMI-SLT: Speech and Language Technology",
    author = "Hayrettin G{\"u}rk{\"o}k and Gido Hakvoort and Mannes Poel",
    note = "10.1145/2070481.2070491",
    year = "2011",
    month = "11",
    doi = "10.1145/2070481.2070491",
    language = "Undefined",
    isbn = "978-1-4503-0641-6",
    publisher = "Association for Computing Machinery (ACM)",
    pages = "41--48",
    editor = "Herv{\'e} Bourlard and Huang, {Thomas S.} and Enrique Vidal and Daniel Gatica-Perez and Louis-Philippe Morency and Nicu Sebe",
    booktitle = "Proceedings of the 13th International Conference on Multimodal Interfaces (ICMI 2011)",
    address = "United States",

    }

    Gürkök, H, Hakvoort, G & Poel, M 2011, Modality switching and performance in a thought and speech controlled computer game. in H Bourlard, TS Huang, E Vidal, D Gatica-Perez, L-P Morency & N Sebe (eds), Proceedings of the 13th International Conference on Multimodal Interfaces (ICMI 2011). Association for Computing Machinery (ACM), New York, pp. 41-48, 13th International Conference on Multimodal Interfaces, ICMI 2011, Alicante, Spain, 14/11/11. https://doi.org/10.1145/2070481.2070491

    Modality switching and performance in a thought and speech controlled computer game. / Gürkök, Hayrettin; Hakvoort, Gido; Poel, Mannes.

    Proceedings of the 13th International Conference on Multimodal Interfaces (ICMI 2011). ed. / Hervé Bourlard; Thomas S. Huang; Enrique Vidal; Daniel Gatica-Perez; Louis-Philippe Morency; Nicu Sebe. New York : Association for Computing Machinery (ACM), 2011. p. 41-48.

    Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

    TY - GEN

    T1 - Modality switching and performance in a thought and speech controlled computer game

    AU - Gürkök, Hayrettin

    AU - Hakvoort, Gido

    AU - Poel, Mannes

    N1 - 10.1145/2070481.2070491

    PY - 2011/11

    Y1 - 2011/11

    N2 - Providing multiple modalities to users is known to improve the overall performance of an interface. Weakness of one modality can be overcome by the strength of another one. Moreover, with respect to their abilities, users can choose between the modalities to use the one that is the best for them. In this paper we explored whether this holds for direct control of a computer game which can be played using a brain-computer interface (BCI) and an automatic speech recogniser (ASR). Participants played the games in unimodal mode (i.e. ASR-only and BCI-only) and multimodal mode where they could switch between the two modalities. The majority of the participants switched modality during the multimodal game but for the most of the time they stayed in ASR control. Therefore multimodality did not provide a significant performance improvement over unimodal control in our particular setup. We also investigated the factors which influence modality switching. We found that performance and peformance-related factors were prominently effective in modality switching.

    AB - Providing multiple modalities to users is known to improve the overall performance of an interface. Weakness of one modality can be overcome by the strength of another one. Moreover, with respect to their abilities, users can choose between the modalities to use the one that is the best for them. In this paper we explored whether this holds for direct control of a computer game which can be played using a brain-computer interface (BCI) and an automatic speech recogniser (ASR). Participants played the games in unimodal mode (i.e. ASR-only and BCI-only) and multimodal mode where they could switch between the two modalities. The majority of the participants switched modality during the multimodal game but for the most of the time they stayed in ASR control. Therefore multimodality did not provide a significant performance improvement over unimodal control in our particular setup. We also investigated the factors which influence modality switching. We found that performance and peformance-related factors were prominently effective in modality switching.

    KW - IR-78612

    KW - METIS-281572

    KW - EWI-20821

    KW - HMI-MI: MULTIMODAL INTERACTIONS

    KW - HMI-CI: Computational Intelligence

    KW - HMI-SLT: Speech and Language Technology

    U2 - 10.1145/2070481.2070491

    DO - 10.1145/2070481.2070491

    M3 - Conference contribution

    SN - 978-1-4503-0641-6

    SP - 41

    EP - 48

    BT - Proceedings of the 13th International Conference on Multimodal Interfaces (ICMI 2011)

    A2 - Bourlard, Hervé

    A2 - Huang, Thomas S.

    A2 - Vidal, Enrique

    A2 - Gatica-Perez, Daniel

    A2 - Morency, Louis-Philippe

    A2 - Sebe, Nicu

    PB - Association for Computing Machinery (ACM)

    CY - New York

    ER -

    Gürkök H, Hakvoort G, Poel M. Modality switching and performance in a thought and speech controlled computer game. In Bourlard H, Huang TS, Vidal E, Gatica-Perez D, Morency L-P, Sebe N, editors, Proceedings of the 13th International Conference on Multimodal Interfaces (ICMI 2011). New York: Association for Computing Machinery (ACM). 2011. p. 41-48 https://doi.org/10.1145/2070481.2070491