Multimodal Information Presentation for High-Load Human Computer Interaction

This dissertation addresses multimodal information presentation in human computer interaction. Information presentation refers to the manner in which computer systems/interfaces present information to human users. More specifically, the focus of our work is not on which information to present, but on how to present it, such as which modalities to use, how to spatially distribute items, et cetera. The notion ``computer'' is not limited to personal computers in their various forms. It also includes embedded computers in any devices or machines. Information presentation guides, constrains and even determines cognitive behavior. This is to say that the same information, when presented differently, can be processed differently by the human cognition system and may lead to different decisions and responses. Consequently, information presentation can influence the quality of human computer interaction, in terms of user performance, cognitive demand and user satisfaction. A good manner of presentation is particularly desired in high load interactions, because users may not have the spared cognitive capacity to cope with the unnecessary mental workload induced by a bad presentation. This dissertation work aims to investigate the effect of information presentation in high load human computer interactions, and to provide useful suggestions for the design of multimodal interfaces. The major presentation factor of interest is modality. First, a literature study has been conducted in the cognitive psychology domain, aiming to understand the role of modality in different stages of human information processing (see Chapter 3). At least three processing stages have modality-specific features, namely sensory processing, perception and working memory. Different modalities are sensed by different sensory receptors, such as the eyes for visual modalities, the ears for auditory modalities and the skin for tactile modalities. Attention needs to be directed to a raw sensory input in order to perceive its meaning, and modalities differ in their ability to attract attention. Complex cognitive activities take place in working memory, and different categories of modality (e.g. visual vs. auditory, verbal vs. nonverbal) consume separated cognitive resources to be processed. These cognitive findings provide theoretical guidance on how to allocate modality in a cognitively appropriate manner. We have conducted a series of user studies in two high load task domains, namely crisis management and driving. Crisis managers typically have to work under time pressure and stress, and they often have to deal with a large amount of information. One study in this domain investigated information presentation for a time limited visual search task (see Chapter 4). The task was to search for wounded earthquake victims on a map and send a doctor to rescue them. The location and type of victim were presented in five modality conditions. The results showed that modality significantly affected task performance, cognitive load and stress, especially when the task load was high. The experimental findings were well explained by several relevant cognitive theories. Further, we proposed a suitability prediction model that quantitatively evaluates modality options and selects the best option for a specific presentation task. The model was demonstrated by our task and suggestions on its generalization were given. A second study using a crisis scenario investigated information presentation for a time limited decision making task (see Chapter 5). Participants were given information about the injuries of two patients and were asked to decide which one needed treatment first. The presentation of the injury information was manipulated by two factors, modality and spatial structure. These two factors both significantly affected task performance and cognitive load. The results further indicated that the two factors influenced different stages of information processing -- modality mostly affected perception and spatial structure mostly affected cognition (cognitive activities in the working memory). Time pressure was found to heighten the effect of presentation factors on the quality of interaction. Driving is normally not a high load task for experienced drivers. However, in the case of emergent danger, drivers need to make quick decisions and respond correctly, otherwise consequences can be catastrophic. We conducted two studies investigating the presentation of local danger warnings (see Chapter 6). Drivers received warnings about emergent road obstacles when the obstacles were not yet visible to them. Based on the warnings, they needed to avoid the obstacles by either braking or changing lane. The presentation of the warnings was varied by modality and level of assistance. The results show that it is beneficial to present both visual and speech warnings, and to suggest appropriate actions to drivers. In addition, drivers may have different preferences on the modality of warnings in different driving conditions (e.g. low visibility, noise, etc). While driving, drivers very often engage in other activities simultaneously, such as interacting with in-vehicle information systems (IVIS), listening to the radio, talking with passengers, et cetera. Multitasking is a cause of high load and distraction during driving. To reduce harmful distractions by IVIS, we proposed to present informative interruption cues (IIC) in addition to IVIS messages. IIC not only convey the arrival of IVIS messages, but also convey their priority levels, helping drivers to decide whether and when to shift their attention to the messages. As a first step, a study was conducted to investigate the design and presentation of IIC (see Chapter 7). A set of vibration and sound cues were created to convey four levels of priority. The cues were evaluated in five task conditions that simulated the perceptional and cognitive load in real driving situations. The results showed that our design of cues was effective, because the cues could be easily learnt and accurately identified in all task conditions. Vibration and sound were both found to have advantages for presenting IIC. All of our studies consistently show that the manner of information presentation has a great influence on the quality of human computer interaction, especially in a high load task setting. Our results also provide useful suggestions on how to present information in a way that is compatible with the characteristics of human cognition. Although these outcomes are obtained in high load task settings, they can be applied to the design of multimodal computer interfaces in general.