The Combination and Evaluation of Query Performance Prediction Methods

Claudia Hauff; Leif Azzopardi; Djoerd Hiemstra

doi:10.1007/978-3-642-00958-7_28

The Combination and Evaluation of Query Performance Prediction Methods

Claudia Hauff, Leif Azzopardi, Djoerd Hiemstra

Databases (Former)

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

3 Citations (Scopus)

17 Downloads (Pure)

Abstract

In this paper, we examine a number of newly applied methods for combining pre-retrieval query performance predictors in order to obtain a better prediction of the query's performance. However, in order to adequately and appropriately compare such techniques, we critically examine the current evaluation methodology and show how using linear correlation coefficients (i) do not provide an intuitive measure indicative of a method's quality, (ii) can provide a misleading indication of performance, and (iii) overstate the performance of combined methods. To address this, we extend the current evaluation methodology to include cross validation, report a more intuitive and descriptive statistic, and apply statistical testing to determine significant differences. During the course of a comprehensive empirical study over several TREC collections, we evaluate nineteen pre-retrieval predictors and three combination methods.

Original language	English
Title of host publication	Advances in Information Retrieval
Subtitle of host publication	31th European Conference on IR Research, ECIR 2009, Toulouse, France, April 6-9, 2009. Proceedings
Editors	Mohand Boughanem, Catherine Berrut, Josiane Mothe, Chantal Soule-Dupuy
Place of Publication	Berlin, Heidelberg
Publisher	Springer
Pages	301-312
Number of pages	12
ISBN (Electronic)	978-3-642-00958-7
ISBN (Print)	978-3-642-00957-0
DOIs	https://doi.org/10.1007/978-3-642-00958-7_28
Publication status	Published - 2009
Event	31th European Conference on Information Retrieval, ECIR 2009: (IR Research) - Toulouse, France Duration: 6 Apr 2009 → 9 Apr 2009 Conference number: 31

Publication series

Name	Lecture Notes In Computer Science
Publisher	Springer
Volume	5478
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	31th European Conference on Information Retrieval, ECIR 2009
Abbreviated title	ECIR
Country/Territory	France
City	Toulouse
Period	6/04/09 → 9/04/09

Keywords

METIS-263977
CR-H.3.3
EWI-15904
IR-67852
Root Mean Square Error
Average Precision
Retrieval Performance
Query Term
Query Performance

Access to Document

10.1007/978-3-642-00958-7_28

Cite this

Hauff, C., Azzopardi, L., & Hiemstra, D. (2009). The Combination and Evaluation of Query Performance Prediction Methods. In M. Boughanem, C. Berrut, J. Mothe, & C. Soule-Dupuy (Eds.), Advances in Information Retrieval: 31th European Conference on IR Research, ECIR 2009, Toulouse, France, April 6-9, 2009. Proceedings (pp. 301-312). (Lecture Notes In Computer Science; Vol. 5478). Springer. https://doi.org/10.1007/978-3-642-00958-7_28

Hauff, Claudia ; Azzopardi, Leif ; Hiemstra, Djoerd. / The Combination and Evaluation of Query Performance Prediction Methods. Advances in Information Retrieval: 31th European Conference on IR Research, ECIR 2009, Toulouse, France, April 6-9, 2009. Proceedings. editor / Mohand Boughanem ; Catherine Berrut ; Josiane Mothe ; Chantal Soule-Dupuy. Berlin, Heidelberg : Springer, 2009. pp. 301-312 (Lecture Notes In Computer Science).

@inproceedings{a76387eaec7d4c32ab276025957f7989,

title = "The Combination and Evaluation of Query Performance Prediction Methods",

abstract = "In this paper, we examine a number of newly applied methods for combining pre-retrieval query performance predictors in order to obtain a better prediction of the query's performance. However, in order to adequately and appropriately compare such techniques, we critically examine the current evaluation methodology and show how using linear correlation coefficients (i) do not provide an intuitive measure indicative of a method's quality, (ii) can provide a misleading indication of performance, and (iii) overstate the performance of combined methods. To address this, we extend the current evaluation methodology to include cross validation, report a more intuitive and descriptive statistic, and apply statistical testing to determine significant differences. During the course of a comprehensive empirical study over several TREC collections, we evaluate nineteen pre-retrieval predictors and three combination methods.",

keywords = "METIS-263977, CR-H.3.3, EWI-15904, IR-67852, Root Mean Square Error, Average Precision, Retrieval Performance, Query Term, Query Performance",

author = "Claudia Hauff and Leif Azzopardi and Djoerd Hiemstra",

year = "2009",

doi = "10.1007/978-3-642-00958-7_28",

language = "English",

isbn = "978-3-642-00957-0",

series = "Lecture Notes In Computer Science",

publisher = "Springer",

pages = "301--312",

editor = "Mohand Boughanem and Catherine Berrut and Josiane Mothe and Chantal Soule-Dupuy",

booktitle = "Advances in Information Retrieval",

address = "Germany",

note = "31th European Conference on Information Retrieval, ECIR 2009 : (IR Research), ECIR ; Conference date: 06-04-2009 Through 09-04-2009",

}

Hauff, C, Azzopardi, L & Hiemstra, D 2009, The Combination and Evaluation of Query Performance Prediction Methods. in M Boughanem, C Berrut, J Mothe & C Soule-Dupuy (eds), Advances in Information Retrieval: 31th European Conference on IR Research, ECIR 2009, Toulouse, France, April 6-9, 2009. Proceedings. Lecture Notes In Computer Science, vol. 5478, Springer, Berlin, Heidelberg, pp. 301-312, 31th European Conference on Information Retrieval, ECIR 2009, Toulouse, France, 6/04/09. https://doi.org/10.1007/978-3-642-00958-7_28

The Combination and Evaluation of Query Performance Prediction Methods. / Hauff, Claudia; Azzopardi, Leif; Hiemstra, Djoerd.
Advances in Information Retrieval: 31th European Conference on IR Research, ECIR 2009, Toulouse, France, April 6-9, 2009. Proceedings. ed. / Mohand Boughanem; Catherine Berrut; Josiane Mothe; Chantal Soule-Dupuy. Berlin, Heidelberg: Springer, 2009. p. 301-312 (Lecture Notes In Computer Science; Vol. 5478).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

TY - GEN

T1 - The Combination and Evaluation of Query Performance Prediction Methods

AU - Hauff, Claudia

AU - Azzopardi, Leif

AU - Hiemstra, Djoerd

N1 - Conference code: 31

PY - 2009

Y1 - 2009

N2 - In this paper, we examine a number of newly applied methods for combining pre-retrieval query performance predictors in order to obtain a better prediction of the query's performance. However, in order to adequately and appropriately compare such techniques, we critically examine the current evaluation methodology and show how using linear correlation coefficients (i) do not provide an intuitive measure indicative of a method's quality, (ii) can provide a misleading indication of performance, and (iii) overstate the performance of combined methods. To address this, we extend the current evaluation methodology to include cross validation, report a more intuitive and descriptive statistic, and apply statistical testing to determine significant differences. During the course of a comprehensive empirical study over several TREC collections, we evaluate nineteen pre-retrieval predictors and three combination methods.

AB - In this paper, we examine a number of newly applied methods for combining pre-retrieval query performance predictors in order to obtain a better prediction of the query's performance. However, in order to adequately and appropriately compare such techniques, we critically examine the current evaluation methodology and show how using linear correlation coefficients (i) do not provide an intuitive measure indicative of a method's quality, (ii) can provide a misleading indication of performance, and (iii) overstate the performance of combined methods. To address this, we extend the current evaluation methodology to include cross validation, report a more intuitive and descriptive statistic, and apply statistical testing to determine significant differences. During the course of a comprehensive empirical study over several TREC collections, we evaluate nineteen pre-retrieval predictors and three combination methods.

KW - METIS-263977

KW - CR-H.3.3

KW - EWI-15904

KW - IR-67852

KW - Root Mean Square Error

KW - Average Precision

KW - Retrieval Performance

KW - Query Term

KW - Query Performance

U2 - 10.1007/978-3-642-00958-7_28

DO - 10.1007/978-3-642-00958-7_28

M3 - Conference contribution

SN - 978-3-642-00957-0

T3 - Lecture Notes In Computer Science

SP - 301

EP - 312

BT - Advances in Information Retrieval

A2 - Boughanem, Mohand

A2 - Berrut, Catherine

A2 - Mothe, Josiane

A2 - Soule-Dupuy, Chantal

PB - Springer

CY - Berlin, Heidelberg

T2 - 31th European Conference on Information Retrieval, ECIR 2009

Y2 - 6 April 2009 through 9 April 2009

ER -

Hauff C, Azzopardi L, Hiemstra D. The Combination and Evaluation of Query Performance Prediction Methods. In Boughanem M, Berrut C, Mothe J, Soule-Dupuy C, editors, Advances in Information Retrieval: 31th European Conference on IR Research, ECIR 2009, Toulouse, France, April 6-9, 2009. Proceedings. Berlin, Heidelberg: Springer. 2009. p. 301-312. (Lecture Notes In Computer Science). doi: 10.1007/978-3-642-00958-7_28

The Combination and Evaluation of Query Performance Prediction Methods

Abstract

Publication series

Conference

Keywords

Access to Document

Fingerprint

Cite this