The Combination and Evaluation of Query Performance Prediction Methods

Claudia Hauff, Leif Azzopardi, Djoerd Hiemstra

Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

3 Citations (Scopus)

Abstract

In this paper, we examine a number of newly applied methods for combining pre-retrieval query performance predictors in order to obtain a better prediction of the query's performance. However, in order to adequately and appropriately compare such techniques, we critically examine the current evaluation methodology and show how using linear correlation coefficients (i) do not provide an intuitive measure indicative of a method's quality, (ii) can provide a misleading indication of performance, and (iii) overstate the performance of combined methods. To address this, we extend the current evaluation methodology to include cross validation, report a more intuitive and descriptive statistic, and apply statistical testing to determine significant differences. During the course of a comprehensive empirical study over several TREC collections, we evaluate nineteen pre-retrieval predictors and three combination methods.
Original languageEnglish
Title of host publicationAdvances in Information Retrieval
Subtitle of host publication31th European Conference on IR Research, ECIR 2009, Toulouse, France, April 6-9, 2009. Proceedings
EditorsMohand Boughanem, Catherine Berrut, Josiane Mothe, Chantal Soule-Dupuy
Place of PublicationBerlin, Heidelberg
PublisherSpringer
Pages301-312
Number of pages12
ISBN (Electronic)978-3-642-00958-7
ISBN (Print)978-3-642-00957-0
DOIs
Publication statusPublished - 2009
Event31th European Conference on Information Retrieval, ECIR 2009: (IR Research) - Toulouse, France
Duration: 6 Apr 20099 Apr 2009
Conference number: 31

Publication series

NameLecture Notes In Computer Science
PublisherSpringer
Volume5478
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Conference

Conference31th European Conference on Information Retrieval, ECIR 2009
Abbreviated titleECIR
CountryFrance
CityToulouse
Period6/04/099/04/09

Keywords

  • METIS-263977
  • CR-H.3.3
  • EWI-15904
  • IR-67852
  • Root Mean Square Error
  • Average Precision
  • Retrieval Performance
  • Query Term
  • Query Performance

Fingerprint Dive into the research topics of 'The Combination and Evaluation of Query Performance Prediction Methods'. Together they form a unique fingerprint.

Cite this