Ciao AI: the Italian adaptation and validation of the Chatbot Usability Scale

Simone Borsci*, Elisa Prati, Alessio Malizia, Martin Schmettow, Alan Chamberlain, Stefano Federici

*Corresponding author for this work

Research output: Contribution to journalArticleAcademicpeer-review

74 Downloads (Pure)

Abstract

Chatbot-based tools are becoming pervasive in multiple domains from commercial websites to rehabilitation applications. Only recently, an eleven-item satisfaction inventory was developed (the ChatBot Usability Scale, BUS-11) to help designers in the assessment process of their systems. The BUS-11 has been validated in multiple contexts and languages, i.e., English, German, Dutch, and Spanish. This scale forms a solid platform enabling designers to rapidly assess chatbots both during and after the design process. The present work aims to adapt and validate the BUS-11 inventory in Italian. A total of 1360 questionnaires were collected which related to a total of 10 Italian chatbot-based systems using the BUS-11 inventory and also using the lite version of the Usability Metrics for User eXperience for convergent validity purposes. The Italian version of the BUS-11 was adapted in terms of the wording of one item, and a Multi-Group Confirmatory Factorial Analysis was performed to establish the factorial structure of the scale and compare the effects of the wording adaptation. Results indicate that the adapted Italian version of the scale matches the expected factorial structure of the original scale. The Italian BUS-11 is highly reliable (Cronbach alpha: 0.921), and it correlates to other measures of satisfaction (e.g., UMUX-Lite, τb = 0.67; p <.001) by also offering specific insights regarding the chatbots’ characteristics. The Italian BUS-11 can be confidently used by chatbot designers to assess the satisfaction of their users during formative or summative tests.

Original languageEnglish
Pages (from-to)2161-2170
Number of pages10
JournalPersonal and ubiquitous computing
Volume27
Early online date1 Jun 2023
DOIs
Publication statusPublished - Dec 2023

Keywords

  • Artificial intelligence
  • Chatbots
  • Conversational agents
  • Language
  • Satisfaction
  • Text
  • Usability
  • User experience

Fingerprint

Dive into the research topics of 'Ciao AI: the Italian adaptation and validation of the Chatbot Usability Scale'. Together they form a unique fingerprint.

Cite this