Preliminary results of a systematic review: Quality assessment of conversational agents (chatbots) for people with disabilities or special needs

Maria Laura de Filippis*, Stefano Federici, Maria Laura Mele, Simone Borsci, Marco Bracalenti, Giancarlo Gaudino, Antonello Cocco, Massimo Amendola, Emilio Simonetti

*Corresponding author for this work

    Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

    Abstract

    People with disabilities or special needs can benefit from AI-based conversational agents, which are used in competence training and well-being management. Assessment of the quality of interactions with these chatbots is key to being able to reduce dissatisfaction with them and to understand their potential long-term benefits. This will in turn help to increase adherence to their use, thereby improving the quality of life of the large population of end-users that they are able to serve. We systematically reviewed the literature on methods of assessing the perceived quality of interactions with chatbots, and identified only 15 of 192 papers on this topic that included people with disabilities or special needs in their assessments. The results also highlighted the lack of a shared theoretical framework for assessing the perceived quality of interactions with chatbots. Systematic procedures based on reliable and valid methodologies continue to be needed in this field. The current lack of reliable tools and systematic methods for assessing chatbots for people with disabilities and special needs is concerning, and may lead to unreliable systems entering the market with disruptive consequences for users. Three major conclusions can be drawn from this systematic analysis: (i) researchers should adopt consolidated and comparable methodologies to rule out risks in use; (ii) the constructs of satisfaction and acceptability are different, and should be measured separately; (iii) dedicated tools and methods for assessing the quality of interaction with chatbots should be developed and used to enable the generation of comparable evidence.

    Original languageEnglish
    Title of host publicationComputers Helping People with Special Needs - 17th International Conference, ICCHP 2020, Proceedings
    EditorsKlaus Miesenberger, Roberto Manduchi, Mario Covarrubias Rodriguez, Petr Penáz
    PublisherSpringer
    Pages250-257
    Number of pages8
    ISBN (Electronic)978-3-030-58796-3
    ISBN (Print)978-3-030-58795-6
    DOIs
    Publication statusPublished - 1 Jan 2020
    Event17th International Conference on Computers Helping People with Special Needs, ICCHP 2020 - Online conference, Lecco, Italy
    Duration: 9 Sep 202011 Sep 2020
    Conference number: 17

    Publication series

    NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
    Volume12376 LNCS
    ISSN (Print)0302-9743
    ISSN (Electronic)1611-3349

    Conference

    Conference17th International Conference on Computers Helping People with Special Needs, ICCHP 2020
    Abbreviated titleICCHP 2020
    CountryItaly
    CityLecco
    Period9/09/2011/09/20

    Keywords

    • Chatbots
    • Conversational agents
    • People with disability
    • People with special needs
    • Quality of interaction
    • Usability

    Fingerprint

    Dive into the research topics of 'Preliminary results of a systematic review: Quality assessment of conversational agents (chatbots) for people with disabilities or special needs'. Together they form a unique fingerprint.

    Cite this