Relying on topic subsets for system ranking estimation

C. Hauff; Djoerd Hiemstra; Franciska M.G. de Jong; Leif Azzopardi

doi:10.1145/1645953.1646249

Relying on topic subsets for system ranking estimation

C. Hauff, Djoerd Hiemstra, Franciska M.G. de Jong, Leif Azzopardi

Databases (Former)

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

10 Citations (Scopus)

Abstract

Ranking a number of retrieval systems according to their retrieval effectiveness without relying on costly relevance judgments was first explored by Soboroff et al [6]. Over the years, a number of alternative approaches have been proposed. We perform a comprehensive analysis of system ranking estimation approaches on a wide variety of TREC test collections and topics sets. Our analysis reveals that the performance of such approaches is highly dependent upon the topic or topic subset, used for estimation. We hypothesize that the performance of system ranking estimation approaches can be improved by selecting the "right" subset of topics and show that using topic subsets improves the performance by 32% on average, with a maximum improvement of up to 70% in some cases.

Original language	Undefined
Title of host publication	Proceeding of the 18th ACM conference on Information and knowledge management
Place of Publication	New York
Publisher	Association for Computing Machinery
Pages	1859-1862
Number of pages	4
ISBN (Print)	978-1-60558-512-3
DOIs	https://doi.org/10.1145/1645953.1646249
Publication status	Published - 2009
Event	18th ACM Conference on Information and Knowledge Management, CIKM 2009: Proceeding of the 18th ACM conference on Information and knowledge management - Hong Kong, Hong Kong Duration: 2 Nov 2009 → 6 Nov 2009 Conference number: 18

Publication series

Name
Publisher	ACM

Conference

Conference	18th ACM Conference on Information and Knowledge Management, CIKM 2009
Country/Territory	Hong Kong
City	Hong Kong
Period	2/11/09 → 6/11/09

Keywords

METIS-265255
IR-69482
EWI-17152
Information Retrieval
Query performance prediction

Access to Document

10.1145/1645953.1646249

Cite this

@inproceedings{f231d991ecc04922aaf2965779f40a81,

title = "Relying on topic subsets for system ranking estimation",

abstract = "Ranking a number of retrieval systems according to their retrieval effectiveness without relying on costly relevance judgments was first explored by Soboroff et al [6]. Over the years, a number of alternative approaches have been proposed. We perform a comprehensive analysis of system ranking estimation approaches on a wide variety of TREC test collections and topics sets. Our analysis reveals that the performance of such approaches is highly dependent upon the topic or topic subset, used for estimation. We hypothesize that the performance of system ranking estimation approaches can be improved by selecting the {"}right{"} subset of topics and show that using topic subsets improves the performance by 32% on average, with a maximum improvement of up to 70% in some cases.",

keywords = "METIS-265255, IR-69482, EWI-17152, Information Retrieval, Query performance prediction",

author = "C. Hauff and Djoerd Hiemstra and {de Jong}, {Franciska M.G.} and Leif Azzopardi",

note = "10.1145/1645953.1646249 ; null ; Conference date: 02-11-2009 Through 06-11-2009",

year = "2009",

doi = "10.1145/1645953.1646249",

language = "Undefined",

isbn = "978-1-60558-512-3",

publisher = "Association for Computing Machinery",

pages = "1859--1862",

booktitle = "Proceeding of the 18th ACM conference on Information and knowledge management",

}

Hauff, C, Hiemstra, D, de Jong, FMG & Azzopardi, L 2009, Relying on topic subsets for system ranking estimation. in Proceeding of the 18th ACM conference on Information and knowledge management., 10.1145/1645953.1646249, Association for Computing Machinery, New York, pp. 1859-1862, 18th ACM Conference on Information and Knowledge Management, CIKM 2009, Hong Kong, Hong Kong, 2/11/09. https://doi.org/10.1145/1645953.1646249

Relying on topic subsets for system ranking estimation. / Hauff, C.; Hiemstra, Djoerd; de Jong, Franciska M.G. et al.

Proceeding of the 18th ACM conference on Information and knowledge management. New York : Association for Computing Machinery, 2009. p. 1859-1862 10.1145/1645953.1646249.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

TY - GEN

T1 - Relying on topic subsets for system ranking estimation

AU - Hauff, C.

AU - Hiemstra, Djoerd

AU - de Jong, Franciska M.G.

AU - Azzopardi, Leif

N1 - Conference code: 18

PY - 2009

Y1 - 2009

N2 - Ranking a number of retrieval systems according to their retrieval effectiveness without relying on costly relevance judgments was first explored by Soboroff et al [6]. Over the years, a number of alternative approaches have been proposed. We perform a comprehensive analysis of system ranking estimation approaches on a wide variety of TREC test collections and topics sets. Our analysis reveals that the performance of such approaches is highly dependent upon the topic or topic subset, used for estimation. We hypothesize that the performance of system ranking estimation approaches can be improved by selecting the "right" subset of topics and show that using topic subsets improves the performance by 32% on average, with a maximum improvement of up to 70% in some cases.

AB - Ranking a number of retrieval systems according to their retrieval effectiveness without relying on costly relevance judgments was first explored by Soboroff et al [6]. Over the years, a number of alternative approaches have been proposed. We perform a comprehensive analysis of system ranking estimation approaches on a wide variety of TREC test collections and topics sets. Our analysis reveals that the performance of such approaches is highly dependent upon the topic or topic subset, used for estimation. We hypothesize that the performance of system ranking estimation approaches can be improved by selecting the "right" subset of topics and show that using topic subsets improves the performance by 32% on average, with a maximum improvement of up to 70% in some cases.

KW - METIS-265255

KW - IR-69482

KW - EWI-17152

KW - Information Retrieval

KW - Query performance prediction

U2 - 10.1145/1645953.1646249

DO - 10.1145/1645953.1646249

M3 - Conference contribution

SN - 978-1-60558-512-3

SP - 1859

EP - 1862

BT - Proceeding of the 18th ACM conference on Information and knowledge management

PB - Association for Computing Machinery

CY - New York

Y2 - 2 November 2009 through 6 November 2009

ER -