The Influence of Basic Tokenization on Biomedical Document Retrieval

Rudolf Berend Trieschnigg; Wessel Kraaij; Franciska M.G. de Jong

doi:10.1145/1277741.1277917

The Influence of Basic Tokenization on Biomedical Document Retrieval

Rudolf Berend Trieschnigg, Wessel Kraaij, Franciska M.G. de Jong

Databases (Former)

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

17 Citations (Scopus)

23 Downloads (Pure)

Abstract

Tokenization is a fundamental preprocessing step in Information Retrieval systems in which text is turned into index terms. This paper quantifies and compares the influence of various simple tokenization techniques on document retrieval effectiveness in two domains: biomedicine and news. As expected, biomedical retrieval is more sensitive to small changes in the tokenization method. The tokenization strategy can make the difference between a mediocre and well performing IR system, especially in the biomedical domain.

Original language	Undefined
Title of host publication	Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval
Place of Publication	New York, NY, USA
Publisher	ACM Press
Pages	803-804
Number of pages	2
ISBN (Print)	978-1-59593-597-7
DOIs	https://doi.org/10.1145/1277741.1277917
Publication status	Published - 2007
Event	30th Annual International ACM/SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2007 - Amsterdam, Netherlands Duration: 23 Jul 2007 → 27 Jul 2007 Conference number: 30

Publication series

Name
Publisher	ACM Press
Number	LNCS4549

Conference

Conference	30th Annual International ACM/SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2007
Abbreviated title	SIGIR
Country/Territory	Netherlands
City	Amsterdam
Period	23/07/07 → 27/07/07

Keywords

HMI-IE: Information Engineering
HMI-MR: MULTIMEDIA RETRIEVAL
METIS-241899
IR-61906
EWI-11033
HMI-SLT: Speech and Language Technology

Access to Document

10.1145/1277741.1277917

Cite this

@inproceedings{843dc37f7b5c4483941036862eec47cb,

title = "The Influence of Basic Tokenization on Biomedical Document Retrieval",

abstract = "Tokenization is a fundamental preprocessing step in Information Retrieval systems in which text is turned into index terms. This paper quantifies and compares the influence of various simple tokenization techniques on document retrieval effectiveness in two domains: biomedicine and news. As expected, biomedical retrieval is more sensitive to small changes in the tokenization method. The tokenization strategy can make the difference between a mediocre and well performing IR system, especially in the biomedical domain.",

keywords = "HMI-IE: Information Engineering, HMI-MR: MULTIMEDIA RETRIEVAL, METIS-241899, IR-61906, EWI-11033, HMI-SLT: Speech and Language Technology",

author = "Trieschnigg, {Rudolf Berend} and Wessel Kraaij and {de Jong}, {Franciska M.G.}",

note = "10.1145/1277741.1277917 ; 30th Annual International ACM/SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2007 ; Conference date: 23-07-2007 Through 27-07-2007",

year = "2007",

doi = "10.1145/1277741.1277917",

language = "Undefined",

isbn = "978-1-59593-597-7",

publisher = "ACM Press",

number = "LNCS4549",

pages = "803--804",

booktitle = "Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval",

}

Trieschnigg, RB, Kraaij, W & de Jong, FMG 2007, The Influence of Basic Tokenization on Biomedical Document Retrieval. in Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval., 10.1145/1277741.1277917, ACM Press, New York, NY, USA, pp. 803-804, 30th Annual International ACM/SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2007, Amsterdam, Netherlands, 23/07/07. https://doi.org/10.1145/1277741.1277917

The Influence of Basic Tokenization on Biomedical Document Retrieval. / Trieschnigg, Rudolf Berend; Kraaij, Wessel; de Jong, Franciska M.G.
Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval. New York, NY, USA: ACM Press, 2007. p. 803-804 10.1145/1277741.1277917.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

TY - GEN

T1 - The Influence of Basic Tokenization on Biomedical Document Retrieval

AU - Trieschnigg, Rudolf Berend

AU - Kraaij, Wessel

AU - de Jong, Franciska M.G.

N1 - Conference code: 30

PY - 2007

Y1 - 2007

N2 - Tokenization is a fundamental preprocessing step in Information Retrieval systems in which text is turned into index terms. This paper quantifies and compares the influence of various simple tokenization techniques on document retrieval effectiveness in two domains: biomedicine and news. As expected, biomedical retrieval is more sensitive to small changes in the tokenization method. The tokenization strategy can make the difference between a mediocre and well performing IR system, especially in the biomedical domain.

AB - Tokenization is a fundamental preprocessing step in Information Retrieval systems in which text is turned into index terms. This paper quantifies and compares the influence of various simple tokenization techniques on document retrieval effectiveness in two domains: biomedicine and news. As expected, biomedical retrieval is more sensitive to small changes in the tokenization method. The tokenization strategy can make the difference between a mediocre and well performing IR system, especially in the biomedical domain.

KW - HMI-IE: Information Engineering

KW - HMI-MR: MULTIMEDIA RETRIEVAL

KW - METIS-241899

KW - IR-61906

KW - EWI-11033

KW - HMI-SLT: Speech and Language Technology

U2 - 10.1145/1277741.1277917

DO - 10.1145/1277741.1277917

M3 - Conference contribution

SN - 978-1-59593-597-7

SP - 803

EP - 804

BT - Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval

PB - ACM Press

CY - New York, NY, USA

T2 - 30th Annual International ACM/SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2007

Y2 - 23 July 2007 through 27 July 2007

ER -