Abstract

Many interesting phenomena in conversation can only be annotated as a subjective task, requiring interpretative judgements from annotators. This leads to data which is annotated with lower levels of agreement not only due to errors in the annotation, but also due to the differences in how annotators interpret conversations. This paper constitutes an attempt to find out how subjective annotations with a low level of agreement can profitably be used for machine learning purposes. We analyse the (dis)agreements between annotators for two different cases in a multimodal annotated corpus and explicitly relate the results to the way machine-learning algorithms perform on the annotated data. Finally we present two new concepts, namely `subjective entity' classifiers resp. `consensus objective' classifiers, and give recommendations for using subjective data in machine-learning applications.
Original languageUndefined
Title of host publicationColing 2008: Proceedings of the workshop on Human Judgements in Computational Linguistics
EditorsR. Artstein, G. Boleda, F. Keller, S. Schulte im Walde
PublisherColing 2008 Organizing Committee
Pages8-16
Number of pages9
ISBN (Print)978-1-905593-47-7
StatePublished - 23 Aug 2008

Publication series

Name
PublisherColing 2008 Organizing Committee
NumberDTR08-9

Fingerprint

data
conversation
machine
learning
consensus
error
algorithm
objective
purpose
concept

Keywords

  • HMI-SLT: Speech and Language Technology
  • EC Grant Agreement nr.: FP6/033812
  • EWI-13249
  • HMI-MI: MULTIMODAL INTERACTIONS
  • METIS-251138
  • IR-64928

Cite this

Reidsma, D., & op den Akker, H. J. A. (2008). Exploiting ''Subjective'' Annotations. In R. Artstein, G. Boleda, F. Keller, & S. Schulte im Walde (Eds.), Coling 2008: Proceedings of the workshop on Human Judgements in Computational Linguistics (pp. 8-16). Coling 2008 Organizing Committee.

Reidsma, Dennis; op den Akker, Hendrikus J.A. / Exploiting ''Subjective'' Annotations.

Coling 2008: Proceedings of the workshop on Human Judgements in Computational Linguistics. ed. / R. Artstein; G. Boleda; F. Keller; S. Schulte im Walde. Coling 2008 Organizing Committee, 2008. p. 8-16.

Research output: Scientific - peer-reviewConference contribution

@inbook{2b6b82eb2e1a481c8d7d17d84dcfc1c2,
title = "Exploiting ''Subjective'' Annotations",
abstract = "Many interesting phenomena in conversation can only be annotated as a subjective task, requiring interpretative judgements from annotators. This leads to data which is annotated with lower levels of agreement not only due to errors in the annotation, but also due to the differences in how annotators interpret conversations. This paper constitutes an attempt to find out how subjective annotations with a low level of agreement can profitably be used for machine learning purposes. We analyse the (dis)agreements between annotators for two different cases in a multimodal annotated corpus and explicitly relate the results to the way machine-learning algorithms perform on the annotated data. Finally we present two new concepts, namely `subjective entity' classifiers resp. `consensus objective' classifiers, and give recommendations for using subjective data in machine-learning applications.",
keywords = "HMI-SLT: Speech and Language Technology, EC Grant Agreement nr.: FP6/033812, EWI-13249, HMI-MI: MULTIMODAL INTERACTIONS, METIS-251138, IR-64928",
author = "Dennis Reidsma and {op den Akker}, {Hendrikus J.A.}",
year = "2008",
month = "8",
isbn = "978-1-905593-47-7",
publisher = "Coling 2008 Organizing Committee",
number = "DTR08-9",
pages = "8--16",
editor = "R. Artstein and G. Boleda and F. Keller and {Schulte im Walde}, S.",
booktitle = "Coling 2008: Proceedings of the workshop on Human Judgements in Computational Linguistics",

}

Reidsma, D & op den Akker, HJA 2008, Exploiting ''Subjective'' Annotations. in R Artstein, G Boleda, F Keller & S Schulte im Walde (eds), Coling 2008: Proceedings of the workshop on Human Judgements in Computational Linguistics. Coling 2008 Organizing Committee, pp. 8-16.

Exploiting ''Subjective'' Annotations. / Reidsma, Dennis; op den Akker, Hendrikus J.A.

Coling 2008: Proceedings of the workshop on Human Judgements in Computational Linguistics. ed. / R. Artstein; G. Boleda; F. Keller; S. Schulte im Walde. Coling 2008 Organizing Committee, 2008. p. 8-16.

Research output: Scientific - peer-reviewConference contribution

TY - CHAP

T1 - Exploiting ''Subjective'' Annotations

AU - Reidsma,Dennis

AU - op den Akker,Hendrikus J.A.

PY - 2008/8/23

Y1 - 2008/8/23

N2 - Many interesting phenomena in conversation can only be annotated as a subjective task, requiring interpretative judgements from annotators. This leads to data which is annotated with lower levels of agreement not only due to errors in the annotation, but also due to the differences in how annotators interpret conversations. This paper constitutes an attempt to find out how subjective annotations with a low level of agreement can profitably be used for machine learning purposes. We analyse the (dis)agreements between annotators for two different cases in a multimodal annotated corpus and explicitly relate the results to the way machine-learning algorithms perform on the annotated data. Finally we present two new concepts, namely `subjective entity' classifiers resp. `consensus objective' classifiers, and give recommendations for using subjective data in machine-learning applications.

AB - Many interesting phenomena in conversation can only be annotated as a subjective task, requiring interpretative judgements from annotators. This leads to data which is annotated with lower levels of agreement not only due to errors in the annotation, but also due to the differences in how annotators interpret conversations. This paper constitutes an attempt to find out how subjective annotations with a low level of agreement can profitably be used for machine learning purposes. We analyse the (dis)agreements between annotators for two different cases in a multimodal annotated corpus and explicitly relate the results to the way machine-learning algorithms perform on the annotated data. Finally we present two new concepts, namely `subjective entity' classifiers resp. `consensus objective' classifiers, and give recommendations for using subjective data in machine-learning applications.

KW - HMI-SLT: Speech and Language Technology

KW - EC Grant Agreement nr.: FP6/033812

KW - EWI-13249

KW - HMI-MI: MULTIMODAL INTERACTIONS

KW - METIS-251138

KW - IR-64928

M3 - Conference contribution

SN - 978-1-905593-47-7

SP - 8

EP - 16

BT - Coling 2008: Proceedings of the workshop on Human Judgements in Computational Linguistics

PB - Coling 2008 Organizing Committee

ER -

Reidsma D, op den Akker HJA. Exploiting ''Subjective'' Annotations. In Artstein R, Boleda G, Keller F, Schulte im Walde S, editors, Coling 2008: Proceedings of the workshop on Human Judgements in Computational Linguistics. Coling 2008 Organizing Committee. 2008. p. 8-16.