A multimodal human-computer interaction framework for research into crisis management

S. Fitrianie, Ronald Walter Poppe, B.H.T. Bui Huu Trung, A.G. Chitu, D. Datcu, R. Dor, D.H.W. Hofs, P. Wiggers, D.J.M. Willems, Mannes Poel, L.J.M. Rothkrantz, L.G. Vuurpijl, Jakob Zwiers

  • 9 Citations

Abstract

Unreliable communication networks, chaotic environments and stressful conditions can make communication during crisis events difficult. The current practice in crisis management can be improved by introducing ICT systems in the process. However, much experimentation is needed to determine where and how ICT can aid. Therefore, we propose a framework in which predefined modules can be connected in an ad hoc fashion. Such a framework allows for rapid development and evaluation of such ICT systems. The framework offers recognition of various communication modalities including speech, lip movement, facial expression, handwriting and drawing, body gesture, text and visual symbols. It provides mechanisms to fuse these modalities into a context dependent interpretation of the current situation and generate appropriate the multimodal information responses. The proposed toolbox can be used as part of a disaster and rescue simulation. We propose evaluation methods, and focus on the technological aspects of our framework.
Original languageUndefined
Title of host publicationProceedings of the fourth international conference on information systems for crisis management, ISCRAM 2007
EditorsB. van der Walle, P. Burghardt, K. Nieuwenhuis
Place of PublicationBrussels
PublisherAcademic and Scientific Publishers NV
Pages149-158
Number of pages10
ISBN (Print)978-90-5487-417-1
StatePublished - 17 May 2007

Publication series

Name
PublisherAcademic and Scientific Publishers NV
Number06472

Fingerprint

Communication
Electric fuses
Disasters
Telecommunication networks

Keywords

  • EWI-9941
  • HMI-HF: Human Factors
  • HMI-MI: MULTIMODAL INTERACTIONS
  • Disaster and rescue simulation
  • IR-64045
  • Human computer interaction
  • Multimodal systems
  • METIS-241640

Cite this

Fitrianie, S., Poppe, R. W., Bui Huu Trung, B. H. T., Chitu, A. G., Datcu, D., Dor, R., ... Zwiers, J. (2007). A multimodal human-computer interaction framework for research into crisis management. In B. van der Walle, P. Burghardt, & K. Nieuwenhuis (Eds.), Proceedings of the fourth international conference on information systems for crisis management, ISCRAM 2007 (pp. 149-158). Brussels: Academic and Scientific Publishers NV.

Fitrianie, S.; Poppe, Ronald Walter; Bui Huu Trung, B.H.T.; Chitu, A.G.; Datcu, D.; Dor, R.; Hofs, D.H.W.; Wiggers, P.; Willems, D.J.M.; Poel, Mannes; Rothkrantz, L.J.M.; Vuurpijl, L.G.; Zwiers, Jakob / A multimodal human-computer interaction framework for research into crisis management.

Proceedings of the fourth international conference on information systems for crisis management, ISCRAM 2007. ed. / B. van der Walle; P. Burghardt; K. Nieuwenhuis. Brussels : Academic and Scientific Publishers NV, 2007. p. 149-158.

Research output: Scientific - peer-reviewConference contribution

@inbook{ab283549ac8d43168b22a11b69bbaa5b,
title = "A multimodal human-computer interaction framework for research into crisis management",
abstract = "Unreliable communication networks, chaotic environments and stressful conditions can make communication during crisis events difficult. The current practice in crisis management can be improved by introducing ICT systems in the process. However, much experimentation is needed to determine where and how ICT can aid. Therefore, we propose a framework in which predefined modules can be connected in an ad hoc fashion. Such a framework allows for rapid development and evaluation of such ICT systems. The framework offers recognition of various communication modalities including speech, lip movement, facial expression, handwriting and drawing, body gesture, text and visual symbols. It provides mechanisms to fuse these modalities into a context dependent interpretation of the current situation and generate appropriate the multimodal information responses. The proposed toolbox can be used as part of a disaster and rescue simulation. We propose evaluation methods, and focus on the technological aspects of our framework.",
keywords = "EWI-9941, HMI-HF: Human Factors, HMI-MI: MULTIMODAL INTERACTIONS, Disaster and rescue simulation, IR-64045, Human computer interaction, Multimodal systems, METIS-241640",
author = "S. Fitrianie and Poppe, {Ronald Walter} and {Bui Huu Trung}, B.H.T. and A.G. Chitu and D. Datcu and R. Dor and D.H.W. Hofs and P. Wiggers and D.J.M. Willems and Mannes Poel and L.J.M. Rothkrantz and L.G. Vuurpijl and Jakob Zwiers",
year = "2007",
month = "5",
isbn = "978-90-5487-417-1",
publisher = "Academic and Scientific Publishers NV",
number = "06472",
pages = "149--158",
editor = "{van der Walle}, B. and P. Burghardt and K. Nieuwenhuis",
booktitle = "Proceedings of the fourth international conference on information systems for crisis management, ISCRAM 2007",

}

Fitrianie, S, Poppe, RW, Bui Huu Trung, BHT, Chitu, AG, Datcu, D, Dor, R, Hofs, DHW, Wiggers, P, Willems, DJM, Poel, M, Rothkrantz, LJM, Vuurpijl, LG & Zwiers, J 2007, A multimodal human-computer interaction framework for research into crisis management. in B van der Walle, P Burghardt & K Nieuwenhuis (eds), Proceedings of the fourth international conference on information systems for crisis management, ISCRAM 2007. Academic and Scientific Publishers NV, Brussels, pp. 149-158.

A multimodal human-computer interaction framework for research into crisis management. / Fitrianie, S.; Poppe, Ronald Walter; Bui Huu Trung, B.H.T.; Chitu, A.G.; Datcu, D.; Dor, R.; Hofs, D.H.W.; Wiggers, P.; Willems, D.J.M.; Poel, Mannes; Rothkrantz, L.J.M.; Vuurpijl, L.G.; Zwiers, Jakob.

Proceedings of the fourth international conference on information systems for crisis management, ISCRAM 2007. ed. / B. van der Walle; P. Burghardt; K. Nieuwenhuis. Brussels : Academic and Scientific Publishers NV, 2007. p. 149-158.

Research output: Scientific - peer-reviewConference contribution

TY - CHAP

T1 - A multimodal human-computer interaction framework for research into crisis management

AU - Fitrianie,S.

AU - Poppe,Ronald Walter

AU - Bui Huu Trung,B.H.T.

AU - Chitu,A.G.

AU - Datcu,D.

AU - Dor,R.

AU - Hofs,D.H.W.

AU - Wiggers,P.

AU - Willems,D.J.M.

AU - Poel,Mannes

AU - Rothkrantz,L.J.M.

AU - Vuurpijl,L.G.

AU - Zwiers,Jakob

PY - 2007/5/17

Y1 - 2007/5/17

N2 - Unreliable communication networks, chaotic environments and stressful conditions can make communication during crisis events difficult. The current practice in crisis management can be improved by introducing ICT systems in the process. However, much experimentation is needed to determine where and how ICT can aid. Therefore, we propose a framework in which predefined modules can be connected in an ad hoc fashion. Such a framework allows for rapid development and evaluation of such ICT systems. The framework offers recognition of various communication modalities including speech, lip movement, facial expression, handwriting and drawing, body gesture, text and visual symbols. It provides mechanisms to fuse these modalities into a context dependent interpretation of the current situation and generate appropriate the multimodal information responses. The proposed toolbox can be used as part of a disaster and rescue simulation. We propose evaluation methods, and focus on the technological aspects of our framework.

AB - Unreliable communication networks, chaotic environments and stressful conditions can make communication during crisis events difficult. The current practice in crisis management can be improved by introducing ICT systems in the process. However, much experimentation is needed to determine where and how ICT can aid. Therefore, we propose a framework in which predefined modules can be connected in an ad hoc fashion. Such a framework allows for rapid development and evaluation of such ICT systems. The framework offers recognition of various communication modalities including speech, lip movement, facial expression, handwriting and drawing, body gesture, text and visual symbols. It provides mechanisms to fuse these modalities into a context dependent interpretation of the current situation and generate appropriate the multimodal information responses. The proposed toolbox can be used as part of a disaster and rescue simulation. We propose evaluation methods, and focus on the technological aspects of our framework.

KW - EWI-9941

KW - HMI-HF: Human Factors

KW - HMI-MI: MULTIMODAL INTERACTIONS

KW - Disaster and rescue simulation

KW - IR-64045

KW - Human computer interaction

KW - Multimodal systems

KW - METIS-241640

M3 - Conference contribution

SN - 978-90-5487-417-1

SP - 149

EP - 158

BT - Proceedings of the fourth international conference on information systems for crisis management, ISCRAM 2007

PB - Academic and Scientific Publishers NV

ER -

Fitrianie S, Poppe RW, Bui Huu Trung BHT, Chitu AG, Datcu D, Dor R et al. A multimodal human-computer interaction framework for research into crisis management. In van der Walle B, Burghardt P, Nieuwenhuis K, editors, Proceedings of the fourth international conference on information systems for crisis management, ISCRAM 2007. Brussels: Academic and Scientific Publishers NV. 2007. p. 149-158.