Exploiting ''Subjective'' Annotations

Dennis Reidsma, Hendrikus J.A. op den Akker

    Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

    53 Downloads (Pure)

    Abstract

    Many interesting phenomena in conversation can only be annotated as a subjective task, requiring interpretative judgements from annotators. This leads to data which is annotated with lower levels of agreement not only due to errors in the annotation, but also due to the differences in how annotators interpret conversations. This paper constitutes an attempt to find out how subjective annotations with a low level of agreement can profitably be used for machine learning purposes. We analyse the (dis)agreements between annotators for two different cases in a multimodal annotated corpus and explicitly relate the results to the way machine-learning algorithms perform on the annotated data. Finally we present two new concepts, namely `subjective entity' classifiers resp. `consensus objective' classifiers, and give recommendations for using subjective data in machine-learning applications.
    Original languageUndefined
    Title of host publicationColing 2008: Proceedings of the workshop on Human Judgements in Computational Linguistics
    EditorsR. Artstein, G. Boleda, F. Keller, S. Schulte im Walde
    PublisherColing 2008 Organizing Committee
    Pages8-16
    Number of pages9
    ISBN (Print)978-1-905593-47-7
    Publication statusPublished - 23 Aug 2008

    Publication series

    Name
    PublisherColing 2008 Organizing Committee
    NumberDTR08-9

    Keywords

    • HMI-SLT: Speech and Language Technology
    • EC Grant Agreement nr.: FP6/033812
    • EWI-13249
    • HMI-MI: MULTIMODAL INTERACTIONS
    • METIS-251138
    • IR-64928

    Cite this

    Reidsma, D., & op den Akker, H. J. A. (2008). Exploiting ''Subjective'' Annotations. In R. Artstein, G. Boleda, F. Keller, & S. Schulte im Walde (Eds.), Coling 2008: Proceedings of the workshop on Human Judgements in Computational Linguistics (pp. 8-16). Coling 2008 Organizing Committee.