PEIR: Modeling Performance in Neural Information Retrieval

Pooya Khandel; Andrew Yates; Ana-Lucia Varbanescu; Maarten de Rijke; Andy Pimentel

doi:10.1007/978-3-031-88711-6_18

PEIR: Modeling Performance in Neural Information Retrieval

Pooya Khandel^*
, Andrew Yates
, Ana-Lucia Varbanescu
, Maarten de Rijke
, Andy Pimentel

^*Corresponding author for this work

Computer Architecture Design and Test for Embedded Systems

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

4 Downloads (Pure)

Abstract

The efficiency of neural information retrieval methods is primarily evaluated by measuring query latency. In practice, measuring latency is highly tied to hardware configurations and requires extensive computational resources. Given the rapid introduction of retrieval models, achieving an overall comparison of their efficiency is challenging. In this paper, we introduce PEIR, a framework for hardware-independent efficiency measurements in Learned Sparse Retrieval (LSR). By employing performance modeling approaches from high-performance computing, we derive performance models for query evaluation approaches such as BlockMax-MaxScore (BMM) and propose to measure memory and/or floating-point operations while performing retrieval on input queries. We demonstrate that by using PEIR, similar conclusions on comparing the latency of retrieval models are obtained.

Original language	English
Title of host publication	Advances in Information Retrieval
Subtitle of host publication	47th European Conference on Information Retrieval, ECIR 2025, Proceedings
Editors	Claudia Hauff, Craig Macdonald, Dietmar Jannach, Gabriella Kazai, Franco Maria Nardini, Fabio Pinelli, Fabrizio Silvestri, Nicola Tonellotto
Publisher	Springer
Pages	279-294
Number of pages	16
ISBN (Electronic)	978-3-031-88711-6
ISBN (Print)	978-3-031-88710-9
DOIs	https://doi.org/10.1007/978-3-031-88711-6_18
Publication status	Published - 4 Apr 2025
Event	47th European Conference on Information Retrieval, ECIR 2025 - Lucca, Italy Duration: 6 Apr 2025 → 10 Apr 2025 Conference number: 47

Publication series

Name	Lecture Notes in Computer Science
Volume	15573 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	47th European Conference on Information Retrieval, ECIR 2025
Abbreviated title	ECIR 2025
Country/Territory	Italy
City	Lucca
Period	6/04/25 → 10/04/25

Keywords

2025 OA procedure
Latency
Learned Sparse Retrieval
Performance Modelling
Efficiency

Access to Document

10.1007/978-3-031-88711-6_18

Conference paperFinal published version, 733 KBLicence: Taverne

Cite this

Khandel, P., Yates, A., Varbanescu, A.-L., de Rijke, M., & Pimentel, A. (2025). PEIR: Modeling Performance in Neural Information Retrieval. In C. Hauff, C. Macdonald, D. Jannach, G. Kazai, F. M. Nardini, F. Pinelli, F. Silvestri, & N. Tonellotto (Eds.), Advances in Information Retrieval: 47th European Conference on Information Retrieval, ECIR 2025, Proceedings (pp. 279-294). (Lecture Notes in Computer Science; Vol. 15573 LNCS). Springer. https://doi.org/10.1007/978-3-031-88711-6_18

Khandel, Pooya ; Yates, Andrew ; Varbanescu, Ana-Lucia et al. / PEIR : Modeling Performance in Neural Information Retrieval. Advances in Information Retrieval: 47th European Conference on Information Retrieval, ECIR 2025, Proceedings. editor / Claudia Hauff ; Craig Macdonald ; Dietmar Jannach ; Gabriella Kazai ; Franco Maria Nardini ; Fabio Pinelli ; Fabrizio Silvestri ; Nicola Tonellotto. Springer, 2025. pp. 279-294 (Lecture Notes in Computer Science).

@inproceedings{7e9eca9d90df4206a9b3c0f76ad853fe,

title = "PEIR: Modeling Performance in Neural Information Retrieval",

abstract = "The efficiency of neural information retrieval methods is primarily evaluated by measuring query latency. In practice, measuring latency is highly tied to hardware configurations and requires extensive computational resources. Given the rapid introduction of retrieval models, achieving an overall comparison of their efficiency is challenging. In this paper, we introduce PEIR, a framework for hardware-independent efficiency measurements in Learned Sparse Retrieval (LSR). By employing performance modeling approaches from high-performance computing, we derive performance models for query evaluation approaches such as BlockMax-MaxScore (BMM) and propose to measure memory and/or floating-point operations while performing retrieval on input queries. We demonstrate that by using PEIR, similar conclusions on comparing the latency of retrieval models are obtained.",

keywords = "2025 OA procedure, Latency, Learned Sparse Retrieval, Performance Modelling, Efficiency",

author = "Pooya Khandel and Andrew Yates and Ana-Lucia Varbanescu and \{de Rijke\}, Maarten and Andy Pimentel",

note = "Publisher Copyright: {\textcopyright} The Author(s), under exclusive license to Springer Nature Switzerland AG 2025.; 47th European Conference on Information Retrieval, ECIR 2025, ECIR 2025 ; Conference date: 06-04-2025 Through 10-04-2025",

year = "2025",

month = apr,

day = "4",

doi = "10.1007/978-3-031-88711-6\_18",

language = "English",

isbn = "978-3-031-88710-9",

series = "Lecture Notes in Computer Science",

publisher = "Springer",

pages = "279--294",

editor = "Claudia Hauff and Craig Macdonald and Dietmar Jannach and Gabriella Kazai and Nardini, \{Franco Maria\} and Fabio Pinelli and Fabrizio Silvestri and Nicola Tonellotto",

booktitle = "Advances in Information Retrieval",

address = "Germany",

}

Khandel, P, Yates, A, Varbanescu, A-L, de Rijke, M & Pimentel, A 2025, PEIR: Modeling Performance in Neural Information Retrieval. in C Hauff, C Macdonald, D Jannach, G Kazai, FM Nardini, F Pinelli, F Silvestri & N Tonellotto (eds), Advances in Information Retrieval: 47th European Conference on Information Retrieval, ECIR 2025, Proceedings. Lecture Notes in Computer Science, vol. 15573 LNCS, Springer, pp. 279-294, 47th European Conference on Information Retrieval, ECIR 2025, Lucca, Italy, 6/04/25. https://doi.org/10.1007/978-3-031-88711-6_18

PEIR: Modeling Performance in Neural Information Retrieval. / Khandel, Pooya; Yates, Andrew; Varbanescu, Ana-Lucia et al.
Advances in Information Retrieval: 47th European Conference on Information Retrieval, ECIR 2025, Proceedings. ed. / Claudia Hauff; Craig Macdonald; Dietmar Jannach; Gabriella Kazai; Franco Maria Nardini; Fabio Pinelli; Fabrizio Silvestri; Nicola Tonellotto. Springer, 2025. p. 279-294 (Lecture Notes in Computer Science; Vol. 15573 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

TY - GEN

T1 - PEIR

T2 - 47th European Conference on Information Retrieval, ECIR 2025

AU - Khandel, Pooya

AU - Yates, Andrew

AU - Varbanescu, Ana-Lucia

AU - de Rijke, Maarten

AU - Pimentel, Andy

N1 - Conference code: 47

PY - 2025/4/4

Y1 - 2025/4/4

N2 - The efficiency of neural information retrieval methods is primarily evaluated by measuring query latency. In practice, measuring latency is highly tied to hardware configurations and requires extensive computational resources. Given the rapid introduction of retrieval models, achieving an overall comparison of their efficiency is challenging. In this paper, we introduce PEIR, a framework for hardware-independent efficiency measurements in Learned Sparse Retrieval (LSR). By employing performance modeling approaches from high-performance computing, we derive performance models for query evaluation approaches such as BlockMax-MaxScore (BMM) and propose to measure memory and/or floating-point operations while performing retrieval on input queries. We demonstrate that by using PEIR, similar conclusions on comparing the latency of retrieval models are obtained.

AB - The efficiency of neural information retrieval methods is primarily evaluated by measuring query latency. In practice, measuring latency is highly tied to hardware configurations and requires extensive computational resources. Given the rapid introduction of retrieval models, achieving an overall comparison of their efficiency is challenging. In this paper, we introduce PEIR, a framework for hardware-independent efficiency measurements in Learned Sparse Retrieval (LSR). By employing performance modeling approaches from high-performance computing, we derive performance models for query evaluation approaches such as BlockMax-MaxScore (BMM) and propose to measure memory and/or floating-point operations while performing retrieval on input queries. We demonstrate that by using PEIR, similar conclusions on comparing the latency of retrieval models are obtained.

KW - 2025 OA procedure

KW - Latency

KW - Learned Sparse Retrieval

KW - Performance Modelling

KW - Efficiency

UR - https://www.scopus.com/pages/publications/105006510507

U2 - 10.1007/978-3-031-88711-6_18

DO - 10.1007/978-3-031-88711-6_18

M3 - Conference contribution

AN - SCOPUS:105006510507

SN - 978-3-031-88710-9

T3 - Lecture Notes in Computer Science

SP - 279

EP - 294

BT - Advances in Information Retrieval

A2 - Hauff, Claudia

A2 - Macdonald, Craig

A2 - Jannach, Dietmar

A2 - Kazai, Gabriella

A2 - Nardini, Franco Maria

A2 - Pinelli, Fabio

A2 - Silvestri, Fabrizio

A2 - Tonellotto, Nicola

PB - Springer

Y2 - 6 April 2025 through 10 April 2025

ER -

Khandel P, Yates A, Varbanescu AL, de Rijke M, Pimentel A. PEIR: Modeling Performance in Neural Information Retrieval. In Hauff C, Macdonald C, Jannach D, Kazai G, Nardini FM, Pinelli F, Silvestri F, Tonellotto N, editors, Advances in Information Retrieval: 47th European Conference on Information Retrieval, ECIR 2025, Proceedings. Springer. 2025. p. 279-294. (Lecture Notes in Computer Science). doi: 10.1007/978-3-031-88711-6_18