A Linguistically Motivated Probabilistic Model of Information Retrieval

Djoerd Hiemstra

doi:10.1007/3-540-49653-X_34

A Linguistically Motivated Probabilistic Model of Information Retrieval

Djoerd Hiemstra

Databases (Former)

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

83 Citations (Scopus)

234 Downloads (Pure)

Abstract

This paper presents a new probabilistic model of information retrieval. The most important modeling assumption made is that documents and queries are defined by an ordered sequence of single terms. This assumption is not made in well known existing models of information retrieval, but is essential in the field of statistical natural language processing. Advances already made in statistical natural language processing will be used in this paper to formulate a probabilistic justification for using tf x idf term weighting. The paper shows that the new probabilistic interpretation of tf x idf term weighting might lead to better understanding of statistical ranking mechanisms, for example by explaining how they relate to coordination level ranking. A pilot experiment on the Cranfield test collection indicates that the presented model outperforms the vector space model with classical tf x idf and cosine length normalisation.

Original language	English
Title of host publication	Research and Advanced Technology for Digital Libraries
Subtitle of host publication	Second European Conference, ECDL’98 Heraklion, Crete, Greece September 21–23, 1998 Proceedings
Editors	Christos Nikolaou, Constantine Stephanidis
Place of Publication	Berlin, Heidelberg
Publisher	Springer
Pages	569-584
Number of pages	16
ISBN (Electronic)	978-3-540-49653-3
ISBN (Print)	978-3-540-65101-7
DOIs	https://doi.org/10.1007/3-540-49653-X_34
Publication status	Published - 1998
Event	2nd European Conference on Research and Advanced Technology for Digital Libraries, ECDL 1998 - Heraklion, Greece Duration: 21 Sept 1998 → 23 Sept 1998 Conference number: 2

Publication series

Name	Lecture Notes in Computer Science
Publisher	Springer
Volume	1513
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	2nd European Conference on Research and Advanced Technology for Digital Libraries, ECDL 1998
Abbreviated title	ECDL
Country/Territory	Greece
City	Heraklion
Period	21/09/98 → 23/09/98

Keywords

HMI-MR: MULTIMEDIA RETRIEVAL
Statistical Natural Language Processing
Statistical Information Retrieval
Information Retrieval Theory
CR-H.3.3

Access to Document

10.1007/3-540-49653-X_34

Hiemstra98linguisticallyAccepted author version, 131 KB

Cite this

Hiemstra, D. (1998). A Linguistically Motivated Probabilistic Model of Information Retrieval. In C. Nikolaou, & C. Stephanidis (Eds.), Research and Advanced Technology for Digital Libraries: Second European Conference, ECDL’98 Heraklion, Crete, Greece September 21–23, 1998 Proceedings (pp. 569-584). (Lecture Notes in Computer Science; Vol. 1513). Springer. https://doi.org/10.1007/3-540-49653-X_34

Hiemstra, Djoerd. / A Linguistically Motivated Probabilistic Model of Information Retrieval. Research and Advanced Technology for Digital Libraries: Second European Conference, ECDL’98 Heraklion, Crete, Greece September 21–23, 1998 Proceedings. editor / Christos Nikolaou ; Constantine Stephanidis. Berlin, Heidelberg : Springer, 1998. pp. 569-584 (Lecture Notes in Computer Science).

@inproceedings{ef6efe4d520f490f971f9895a37c6aa2,

title = "A Linguistically Motivated Probabilistic Model of Information Retrieval",

abstract = "This paper presents a new probabilistic model of information retrieval. The most important modeling assumption made is that documents and queries are defined by an ordered sequence of single terms. This assumption is not made in well known existing models of information retrieval, but is essential in the field of statistical natural language processing. Advances already made in statistical natural language processing will be used in this paper to formulate a probabilistic justification for using tf x idf term weighting. The paper shows that the new probabilistic interpretation of tf x idf term weighting might lead to better understanding of statistical ranking mechanisms, for example by explaining how they relate to coordination level ranking. A pilot experiment on the Cranfield test collection indicates that the presented model outperforms the vector space model with classical tf x idf and cosine length normalisation.",

keywords = "HMI-MR: MULTIMEDIA RETRIEVAL, Statistical Natural Language Processing, Statistical Information Retrieval, Information Retrieval Theory, CR-H.3.3",

author = "Djoerd Hiemstra",

year = "1998",

doi = "10.1007/3-540-49653-X_34",

language = "English",

isbn = "978-3-540-65101-7",

series = "Lecture Notes in Computer Science",

publisher = "Springer",

pages = "569--584",

editor = "Christos Nikolaou and Constantine Stephanidis",

booktitle = "Research and Advanced Technology for Digital Libraries",

address = "Germany",

note = "2nd European Conference on Research and Advanced Technology for Digital Libraries, ECDL 1998, ECDL ; Conference date: 21-09-1998 Through 23-09-1998",

}

Hiemstra, D 1998, A Linguistically Motivated Probabilistic Model of Information Retrieval. in C Nikolaou & C Stephanidis (eds), Research and Advanced Technology for Digital Libraries: Second European Conference, ECDL’98 Heraklion, Crete, Greece September 21–23, 1998 Proceedings. Lecture Notes in Computer Science, vol. 1513, Springer, Berlin, Heidelberg, pp. 569-584, 2nd European Conference on Research and Advanced Technology for Digital Libraries, ECDL 1998, Heraklion, Greece, 21/09/98. https://doi.org/10.1007/3-540-49653-X_34

A Linguistically Motivated Probabilistic Model of Information Retrieval. / Hiemstra, Djoerd.
Research and Advanced Technology for Digital Libraries: Second European Conference, ECDL’98 Heraklion, Crete, Greece September 21–23, 1998 Proceedings. ed. / Christos Nikolaou; Constantine Stephanidis. Berlin, Heidelberg: Springer, 1998. p. 569-584 (Lecture Notes in Computer Science; Vol. 1513).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

TY - GEN

T1 - A Linguistically Motivated Probabilistic Model of Information Retrieval

AU - Hiemstra, Djoerd

N1 - Conference code: 2

PY - 1998

Y1 - 1998

N2 - This paper presents a new probabilistic model of information retrieval. The most important modeling assumption made is that documents and queries are defined by an ordered sequence of single terms. This assumption is not made in well known existing models of information retrieval, but is essential in the field of statistical natural language processing. Advances already made in statistical natural language processing will be used in this paper to formulate a probabilistic justification for using tf x idf term weighting. The paper shows that the new probabilistic interpretation of tf x idf term weighting might lead to better understanding of statistical ranking mechanisms, for example by explaining how they relate to coordination level ranking. A pilot experiment on the Cranfield test collection indicates that the presented model outperforms the vector space model with classical tf x idf and cosine length normalisation.

AB - This paper presents a new probabilistic model of information retrieval. The most important modeling assumption made is that documents and queries are defined by an ordered sequence of single terms. This assumption is not made in well known existing models of information retrieval, but is essential in the field of statistical natural language processing. Advances already made in statistical natural language processing will be used in this paper to formulate a probabilistic justification for using tf x idf term weighting. The paper shows that the new probabilistic interpretation of tf x idf term weighting might lead to better understanding of statistical ranking mechanisms, for example by explaining how they relate to coordination level ranking. A pilot experiment on the Cranfield test collection indicates that the presented model outperforms the vector space model with classical tf x idf and cosine length normalisation.

KW - HMI-MR: MULTIMEDIA RETRIEVAL

KW - Statistical Natural Language Processing

KW - Statistical Information Retrieval

KW - Information Retrieval Theory

KW - CR-H.3.3

U2 - 10.1007/3-540-49653-X_34

DO - 10.1007/3-540-49653-X_34

M3 - Conference contribution

SN - 978-3-540-65101-7

T3 - Lecture Notes in Computer Science

SP - 569

EP - 584

BT - Research and Advanced Technology for Digital Libraries

A2 - Nikolaou, Christos

A2 - Stephanidis, Constantine

PB - Springer

CY - Berlin, Heidelberg

T2 - 2nd European Conference on Research and Advanced Technology for Digital Libraries, ECDL 1998

Y2 - 21 September 1998 through 23 September 1998

ER -

Hiemstra D. A Linguistically Motivated Probabilistic Model of Information Retrieval. In Nikolaou C, Stephanidis C, editors, Research and Advanced Technology for Digital Libraries: Second European Conference, ECDL’98 Heraklion, Crete, Greece September 21–23, 1998 Proceedings. Berlin, Heidelberg: Springer. 1998. p. 569-584. (Lecture Notes in Computer Science). doi: 10.1007/3-540-49653-X_34

A Linguistically Motivated Probabilistic Model of Information Retrieval

Abstract

Publication series

Conference

Keywords

Access to Document

Fingerprint

Cite this