Evaluating natural language understanding services for conversational question answering systems

Daniel Braun, Adrian Hernandez Mendez, Florian Matthes, Manfred Langen

Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

82 Citations (Scopus)
1 Downloads (Pure)

Abstract

Conversational interfaces recently gained a lot of attention. One of the reasons for the current hype is the fact that chatbots (one particularly popular form of conversational interfaces) nowadays can be created without any programming knowledge, thanks to different toolkits and so-called Natural Language Understanding (NLU) services. While these NLU services are already widely used in both, industry and science, so far, they have not been analysed systematically. In this paper, we present a method to evaluate the classification performance of NLU services. Moreover, we present two new corpora, one consisting of annotated questions and one consisting of annotated questions with the corresponding answers. Based on these corpora, we conduct an evaluation of some of the most popular NLU services. Thereby we want to enable both, researchers and companies to make more educated decisions about which service they should use.

Original languageEnglish
Title of host publicationSIGDIAL 2017 - 18th Annual Meeting of the Special Interest Group on Discourse and Dialogue, Proceedings of the Conference
PublisherAssociation for Computational Linguistics (ACL)
Pages174-185
Number of pages12
ISBN (Electronic)9781945626821
DOIs
Publication statusPublished - 2017
Externally publishedYes
Event18th Annual Meeting of the Special Interest Group on Discourse and Dialogue, SIGDIAL 2017 - Saarbrucken, Germany
Duration: 15 Aug 201717 Aug 2017
Conference number: 18

Conference

Conference18th Annual Meeting of the Special Interest Group on Discourse and Dialogue, SIGDIAL 2017
Abbreviated titleSIGDIAL 2017
CountryGermany
CitySaarbrucken
Period15/08/1717/08/17

Fingerprint

Dive into the research topics of 'Evaluating natural language understanding services for conversational question answering systems'. Together they form a unique fingerprint.

Cite this