Towards Cognitive-Aware Multimodal Presentation: The Modality Effects in High-Load HCI

Y. Cao; Mariet Theune; Antinus Nijholt

doi:10.1007/978-3-642-02728-4_1

Towards Cognitive-Aware Multimodal Presentation: The Modality Effects in High-Load HCI

Y. Cao, Mariet Theune, Antinus Nijholt

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

3 Citations (Scopus)

Abstract

In this study, we argue that multimodal presentations should be created in a cognitive-aware manner, especially in a high-load HCI situation where the user task challenges the full capacity of the human cognition. An experiment was conducted to investigate the cognitive effects of modalities, using a high-load task. The performance measurements and subjective reports consistently confirm a significant modality impact on cognitive workload, stress and performance. A relation between modality usage and physiological states was not found, due to the insufficient sensitivity and individual differences of the physiological measurements. The findings of this experiment can be well explained by several modality-related cognitive theories. We further integrate these theories into a suitability prediction model, which can systematically predict how suitable a certain modality usage is for this presentation task. The model demonstrates a possible approach towards cognitive-aware modality planning and can be modified for other applications.

Original language	Undefined
Title of host publication	Engineering Psychology and Cognitive Ergonomics. 8th International Conference, EPCE 2009, Held as part of HCI International 2009
Editors	D. Harris
Place of Publication	Berlin
Publisher	Springer
Pages	3-12
Number of pages	10
ISBN (Print)	978-3-642-02727-7
DOIs	https://doi.org/10.1007/978-3-642-02728-4_1
Publication status	Published - 19 Jul 2009
Event	8th International Conference on Engineering Psychology and Cognitive Ergonomics, EPCE 2009 - San Diego, United States Duration: 19 Jul 2009 → 24 Jul 2009 Conference number: 8

Publication series

Name	Lecture Notes in Computer Science
Publisher	Springer Verlag
Volume	5639
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	8th International Conference on Engineering Psychology and Cognitive Ergonomics, EPCE 2009
Abbreviated title	EPCE
Country/Territory	United States
City	San Diego
Period	19/07/09 → 24/07/09

Keywords

METIS-263935
Multimodal presentation
Cognitive load
Cognitive-aware
IR-67820
EWI-15744
high-load HCI
Performance
Stress
HMI-MI: MULTIMODAL INTERACTIONS
HMI-HF: Human Factors
modality planning

Access to Document

10.1007/978-3-642-02728-4_1

http://eprints.eemcs.utwente.nl/secure2/15744/01/hcii2009_yc.pdf

Cite this

Cao, Y., Theune, M., & Nijholt, A. (2009). Towards Cognitive-Aware Multimodal Presentation: The Modality Effects in High-Load HCI. In D. Harris (Ed.), Engineering Psychology and Cognitive Ergonomics. 8th International Conference, EPCE 2009, Held as part of HCI International 2009 (pp. 3-12). [10.1007/978-3-642-02728-4_1] (Lecture Notes in Computer Science; Vol. 5639). Springer. https://doi.org/10.1007/978-3-642-02728-4_1

@inproceedings{7f0697e26994448db852fcdd730e749d,

title = "Towards Cognitive-Aware Multimodal Presentation: The Modality Effects in High-Load HCI",

abstract = "In this study, we argue that multimodal presentations should be created in a cognitive-aware manner, especially in a high-load HCI situation where the user task challenges the full capacity of the human cognition. An experiment was conducted to investigate the cognitive effects of modalities, using a high-load task. The performance measurements and subjective reports consistently confirm a significant modality impact on cognitive workload, stress and performance. A relation between modality usage and physiological states was not found, due to the insufficient sensitivity and individual differences of the physiological measurements. The findings of this experiment can be well explained by several modality-related cognitive theories. We further integrate these theories into a suitability prediction model, which can systematically predict how suitable a certain modality usage is for this presentation task. The model demonstrates a possible approach towards cognitive-aware modality planning and can be modified for other applications.",

keywords = "METIS-263935, Multimodal presentation, Cognitive load, Cognitive-aware, IR-67820, EWI-15744, high-load HCI, Performance, Stress, HMI-MI: MULTIMODAL INTERACTIONS, HMI-HF: Human Factors, modality planning",

author = "Y. Cao and Mariet Theune and Antinus Nijholt",

note = "10.1007/978-3-642-02728-4_1 ; null ; Conference date: 19-07-2009 Through 24-07-2009",

year = "2009",

month = jul,

day = "19",

doi = "10.1007/978-3-642-02728-4_1",

language = "Undefined",

isbn = "978-3-642-02727-7",

series = "Lecture Notes in Computer Science",

publisher = "Springer",

pages = "3--12",

editor = "D. Harris",

booktitle = "Engineering Psychology and Cognitive Ergonomics. 8th International Conference, EPCE 2009, Held as part of HCI International 2009",

}

Cao, Y, Theune, M & Nijholt, A 2009, Towards Cognitive-Aware Multimodal Presentation: The Modality Effects in High-Load HCI. in D Harris (ed.), Engineering Psychology and Cognitive Ergonomics. 8th International Conference, EPCE 2009, Held as part of HCI International 2009., 10.1007/978-3-642-02728-4_1, Lecture Notes in Computer Science, vol. 5639, Springer, Berlin, pp. 3-12, 8th International Conference on Engineering Psychology and Cognitive Ergonomics, EPCE 2009, San Diego, California, United States, 19/07/09. https://doi.org/10.1007/978-3-642-02728-4_1

Towards Cognitive-Aware Multimodal Presentation: The Modality Effects in High-Load HCI. / Cao, Y.; Theune, Mariet ; Nijholt, Antinus.

Engineering Psychology and Cognitive Ergonomics. 8th International Conference, EPCE 2009, Held as part of HCI International 2009. ed. / D. Harris. Berlin : Springer, 2009. p. 3-12 10.1007/978-3-642-02728-4_1 (Lecture Notes in Computer Science; Vol. 5639).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

TY - GEN

T1 - Towards Cognitive-Aware Multimodal Presentation: The Modality Effects in High-Load HCI

AU - Cao, Y.

AU - Theune, Mariet

AU - Nijholt, Antinus

N1 - Conference code: 8

PY - 2009/7/19

Y1 - 2009/7/19

N2 - In this study, we argue that multimodal presentations should be created in a cognitive-aware manner, especially in a high-load HCI situation where the user task challenges the full capacity of the human cognition. An experiment was conducted to investigate the cognitive effects of modalities, using a high-load task. The performance measurements and subjective reports consistently confirm a significant modality impact on cognitive workload, stress and performance. A relation between modality usage and physiological states was not found, due to the insufficient sensitivity and individual differences of the physiological measurements. The findings of this experiment can be well explained by several modality-related cognitive theories. We further integrate these theories into a suitability prediction model, which can systematically predict how suitable a certain modality usage is for this presentation task. The model demonstrates a possible approach towards cognitive-aware modality planning and can be modified for other applications.

AB - In this study, we argue that multimodal presentations should be created in a cognitive-aware manner, especially in a high-load HCI situation where the user task challenges the full capacity of the human cognition. An experiment was conducted to investigate the cognitive effects of modalities, using a high-load task. The performance measurements and subjective reports consistently confirm a significant modality impact on cognitive workload, stress and performance. A relation between modality usage and physiological states was not found, due to the insufficient sensitivity and individual differences of the physiological measurements. The findings of this experiment can be well explained by several modality-related cognitive theories. We further integrate these theories into a suitability prediction model, which can systematically predict how suitable a certain modality usage is for this presentation task. The model demonstrates a possible approach towards cognitive-aware modality planning and can be modified for other applications.

KW - METIS-263935

KW - Multimodal presentation

KW - Cognitive load

KW - Cognitive-aware

KW - IR-67820

KW - EWI-15744

KW - high-load HCI

KW - Performance

KW - Stress

KW - HMI-MI: MULTIMODAL INTERACTIONS

KW - HMI-HF: Human Factors

KW - modality planning

U2 - 10.1007/978-3-642-02728-4_1

DO - 10.1007/978-3-642-02728-4_1

M3 - Conference contribution

SN - 978-3-642-02727-7

T3 - Lecture Notes in Computer Science

SP - 3

EP - 12

BT - Engineering Psychology and Cognitive Ergonomics. 8th International Conference, EPCE 2009, Held as part of HCI International 2009

A2 - Harris, D.

PB - Springer

CY - Berlin

Y2 - 19 July 2009 through 24 July 2009

ER -

Cao Y, Theune M , Nijholt A. Towards Cognitive-Aware Multimodal Presentation: The Modality Effects in High-Load HCI. In Harris D, editor, Engineering Psychology and Cognitive Ergonomics. 8th International Conference, EPCE 2009, Held as part of HCI International 2009. Berlin: Springer. 2009. p. 3-12. 10.1007/978-3-642-02728-4_1. (Lecture Notes in Computer Science). doi: 10.1007/978-3-642-02728-4_1