A cross-benchmark comparison of 87 learning to rank methods

Niek Tax*, Sander Bockting, Djoerd Hiemstra

*Corresponding author for this work

Research output: Contribution to journalArticleAcademicpeer-review

65 Citations (Scopus)
1331 Downloads (Pure)

Abstract

Learning to rank is an increasingly important scientific field that comprises the use of machine learning for the ranking task. New learning to rank methods are generally evaluated on benchmark test collections. However, comparison of learning to rank methods based on evaluation results is hindered by the absence of a standard set of evaluation benchmark collections. In this paper we propose a way to compare learning to rank methods based on a sparse set of evaluation results on a set of benchmark datasets. Our comparison methodology consists of two components: (1) Normalized Winning Number, which gives insight in the ranking accuracy of the learning to rank method, and (2) Ideal Winning Number, which gives insight in the degree of certainty concerning its ranking accuracy. Evaluation results of 87 learning to rank methods on 20 well-known benchmark datasets are collected through a structured literature search. ListNet, SmoothRank, FenchelRank, FSMRank, LRUF and LARF are Pareto optimal learning to rank methods in the Normalized Winning Number and Ideal Winning Number dimensions, listed in increasing order of Normalized Winning Number and decreasing order of Ideal Winning Number.
Original languageEnglish
Pages (from-to)757-772
Number of pages16
JournalInformation processing & management
Volume51
Issue number6
Early online date22 Aug 2015
DOIs
Publication statusPublished - Nov 2015

Keywords

  • 2024 OA procedure
  • Learning to rank
  • Information retrieval
  • Evaluation metric

Fingerprint

Dive into the research topics of 'A cross-benchmark comparison of 87 learning to rank methods'. Together they form a unique fingerprint.

Cite this