A Software Framework for Multimodal Human-Computer Interaction Systems

Jie Shen, Maja Pantic

    Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

    8 Citations (Scopus)
    122 Downloads (Pure)


    This paper describes a software framework we designed and implemented for the development and research in the area of multimodal human-computer interface. The proposed framework is based on publish / subscribe architecture, which allows developers and researchers to conveniently configure, test and expand their system in a modular and incremental manner. In order to achieve reliable and efficient data transport between modules while still providing a high degree of system flexibility, the framework uses a shared-memory based data transport protocol for message delivery together with a TCP based system management protocol to maintain the integrity of system structure at runtime. The framework is delivered as a communication middleware, providing a basic system manager and well-documented, easy-to-use and open source C++ SDKs supporting both module development and server extension. The experimental comparison between the proposed framework and other similar tools available to the community indicates that our framework greatly outperforms the others in terms of average message latency, maximum data throughput and CPU consumption level, especially in heavy workload scenarios. To demonstrate the performance of our framework in real world applications, we have built a demo system which is used to detect faces and facial feature points in real-time captured video. The result shows our framework is capable of delivering some tens of megabytes of data per second effectively and efficiently even under tight resource constraint.
    Original languageUndefined
    Title of host publicationProceedings 2009 IEEE International Conference on Systems, Man and Cybernetics (SMC'09)
    Place of PublicationLos Alamitos
    PublisherIEEE Computer Society
    Number of pages8
    ISBN (Print)978-1-4244-2794-9
    Publication statusPublished - 2009
    EventIEEE International Conference on Systems, Man and Cybernetics, SMC 2009 - Hyatt Regency Riverwalk, San Antonio, United States
    Duration: 11 Oct 200914 Oct 2009

    Publication series

    PublisherIEEE Computer Society Press


    ConferenceIEEE International Conference on Systems, Man and Cybernetics, SMC 2009
    Abbreviated titleSMC 2009
    Country/TerritoryUnited States
    CitySan Antonio


    • METIS-264306
    • IR-69479
    • Publish / Subscribe Architecture
    • EWI-17135
    • Software Framework
    • HMI-HF: Human Factors
    • Multimodal Human-Computer Interface

    Cite this