Cognitive-aware Modality Allocation in Intelligent Multimodal Information Presentation

    Research output: Chapter in Book/Report/Conference proceedingChapterAcademicpeer-review

    Abstract

    Intelligent multimodal presentation (IMMP) systems are able to generate multimodal presentations adaptively, based on the run-time requirements of user-computer interaction. Modality allocation in IMMP system needs to adapt the modality choice to changes in various relevant factors, such as the type of the information to be conveyed, the presentation goal, the characteristics of the available modalities, the user profile, the condition of the environment and the type of user task. In this study, we emphasize that modality allocation in IMMP systems should also take into account the cognitive impacts of modality on human information processing. We first describe several modality-related cognitive and neuropsychological findings. Then a user study is presented to demonstrate the effects of modality on performance, cognitive load and stress, using a high-load and time-critical user task. Finally, we show a possible way to integrate relevant cognitive theories into a computational model that can systematically predict the suitability of a modality choice for a given presentation task.
    Original languageUndefined
    Title of host publicationMultimedia Interaction and Intelligent User Interfaces: Principles, Methods and Applications
    EditorsLing Shao, Caifeng Shan, Jiebo Luo, Minoru Etoh
    Place of PublicationLondon
    PublisherSpringer
    Pages61-83
    Number of pages23
    ISBN (Print)978-1-84996-506-4
    DOIs
    Publication statusPublished - 16 Sep 2010

    Publication series

    NameAdvances in Pattern Recognition
    PublisherSpringer Verlag
    ISSN (Print)1617-7916

    Keywords

    • METIS-270882
    • Cognitive load
    • IR-73130
    • User Profile
    • EWI-18084
    • Modality choice
    • Human computer interaction
    • Stress
    • HMI-HF: Human Factors
    • Multimodal presentation

    Cite this