MapReduce for information retrieval evaluation: "Let's quickly test this on 12 TB of data"

Djoerd Hiemstra; C. Hauff

doi:10.1007/978-3-642-15998-5_8

MapReduce for information retrieval evaluation: "Let's quickly test this on 12 TB of data"

Djoerd Hiemstra
, C. Hauff

Databases (Former)

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

12 Citations (Scopus)

443 Downloads (Pure)

Abstract

We propose to use MapReduce to quickly test new retrieval approaches on a cluster of machines by sequentially scanning all documents. We present a small case study in which we use a cluster of 15 low cost machines to search a web crawl of 0.5 billion pages showing that sequential scanning is a viable approach to running large-scale information retrieval experiments with little effort. The code is available to other researchers at: http://mirex.sourceforge.net.

Original language	Undefined
Title of host publication	International Conference of the Cross-Language Evaluation Forum, CLEF: Multilingual and Multimodal Information Access Evaluation
Editors	Maristella Agosti, Nicola Ferro, Carol Peters, Maarten de Rijke, Alan Smeaton
Place of Publication	Berlin
Publisher	Springer
Pages	64-69
Number of pages	6
ISBN (Print)	978-3-642-15997-8
DOIs	https://doi.org/10.1007/978-3-642-15998-5_8
Publication status	Published - 2010
Event	CLEF (Cross-Language Evaluation Forum) Conference on Multilingual and Multimodal Information Access Evaluation 2010 - Padua, Italy Duration: 20 Sept 2010 → 23 Sept 2010 http://clef2010.clef-initiative.eu/

Publication series

Name	Lecture Notes in Computer Science
Publisher	Springer Verlag
Volume	6360

Conference

Conference	CLEF (Cross-Language Evaluation Forum) Conference on Multilingual and Multimodal Information Access Evaluation 2010
Abbreviated title	CLEF
Country/Territory	Italy
City	Padua
Period	20/09/10 → 23/09/10
Internet address	http://clef2010.clef-initiative.eu/

Keywords

IR-73226
EWI-18469
METIS-271032

Access to Document

10.1007/978-3-642-15998-5_8

Cite this

Hiemstra, D., & Hauff, C. (2010). MapReduce for information retrieval evaluation: "Let's quickly test this on 12 TB of data". In M. Agosti, N. Ferro, C. Peters, M. de Rijke, & A. Smeaton (Eds.), International Conference of the Cross-Language Evaluation Forum, CLEF: Multilingual and Multimodal Information Access Evaluation (pp. 64-69). (Lecture Notes in Computer Science; Vol. 6360). Springer. https://doi.org/10.1007/978-3-642-15998-5_8

Hiemstra, Djoerd ; Hauff, C. / MapReduce for information retrieval evaluation: "Let's quickly test this on 12 TB of data". International Conference of the Cross-Language Evaluation Forum, CLEF: Multilingual and Multimodal Information Access Evaluation. editor / Maristella Agosti ; Nicola Ferro ; Carol Peters ; Maarten de Rijke ; Alan Smeaton. Berlin : Springer, 2010. pp. 64-69 (Lecture Notes in Computer Science).

@inproceedings{3177e4975a8b403087874dcb9555a573,

title = "MapReduce for information retrieval evaluation: {"}Let's quickly test this on 12 TB of data{"}",

abstract = "We propose to use MapReduce to quickly test new retrieval approaches on a cluster of machines by sequentially scanning all documents. We present a small case study in which we use a cluster of 15 low cost machines to search a web crawl of 0.5 billion pages showing that sequential scanning is a viable approach to running large-scale information retrieval experiments with little effort. The code is available to other researchers at: http://mirex.sourceforge.net.",

keywords = "IR-73226, EWI-18469, METIS-271032",

author = "Djoerd Hiemstra and C. Hauff",

note = "eemcs-eprint-18469 ; CLEF (Cross-Language Evaluation Forum) Conference on Multilingual and Multimodal Information Access Evaluation 2010 ; Conference date: 20-09-2010 Through 23-09-2010",

year = "2010",

doi = "10.1007/978-3-642-15998-5\_8",

language = "Undefined",

isbn = "978-3-642-15997-8",

series = "Lecture Notes in Computer Science",

publisher = "Springer",

pages = "64--69",

editor = "Maristella Agosti and Nicola Ferro and Carol Peters and \{de Rijke\}, Maarten and Alan Smeaton",

booktitle = "International Conference of the Cross-Language Evaluation Forum, CLEF: Multilingual and Multimodal Information Access Evaluation",

address = "Germany",

url = "http://clef2010.clef-initiative.eu/",

}

Hiemstra, D & Hauff, C 2010, MapReduce for information retrieval evaluation: "Let's quickly test this on 12 TB of data". in M Agosti, N Ferro, C Peters, M de Rijke & A Smeaton (eds), International Conference of the Cross-Language Evaluation Forum, CLEF: Multilingual and Multimodal Information Access Evaluation. Lecture Notes in Computer Science, vol. 6360, Springer, Berlin, pp. 64-69, CLEF (Cross-Language Evaluation Forum) Conference on Multilingual and Multimodal Information Access Evaluation 2010, Padua, Italy, 20/09/10. https://doi.org/10.1007/978-3-642-15998-5_8

MapReduce for information retrieval evaluation: "Let's quickly test this on 12 TB of data". / Hiemstra, Djoerd; Hauff, C.
International Conference of the Cross-Language Evaluation Forum, CLEF: Multilingual and Multimodal Information Access Evaluation. ed. / Maristella Agosti; Nicola Ferro; Carol Peters; Maarten de Rijke; Alan Smeaton. Berlin: Springer, 2010. p. 64-69 (Lecture Notes in Computer Science; Vol. 6360).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

TY - GEN

T1 - MapReduce for information retrieval evaluation: "Let's quickly test this on 12 TB of data"

AU - Hiemstra, Djoerd

AU - Hauff, C.

N1 - eemcs-eprint-18469

PY - 2010

Y1 - 2010

N2 - We propose to use MapReduce to quickly test new retrieval approaches on a cluster of machines by sequentially scanning all documents. We present a small case study in which we use a cluster of 15 low cost machines to search a web crawl of 0.5 billion pages showing that sequential scanning is a viable approach to running large-scale information retrieval experiments with little effort. The code is available to other researchers at: http://mirex.sourceforge.net.

AB - We propose to use MapReduce to quickly test new retrieval approaches on a cluster of machines by sequentially scanning all documents. We present a small case study in which we use a cluster of 15 low cost machines to search a web crawl of 0.5 billion pages showing that sequential scanning is a viable approach to running large-scale information retrieval experiments with little effort. The code is available to other researchers at: http://mirex.sourceforge.net.

KW - IR-73226

KW - EWI-18469

KW - METIS-271032

U2 - 10.1007/978-3-642-15998-5_8

DO - 10.1007/978-3-642-15998-5_8

M3 - Conference contribution

SN - 978-3-642-15997-8

T3 - Lecture Notes in Computer Science

SP - 64

EP - 69

BT - International Conference of the Cross-Language Evaluation Forum, CLEF: Multilingual and Multimodal Information Access Evaluation

A2 - Agosti, Maristella

A2 - Ferro, Nicola

A2 - Peters, Carol

A2 - de Rijke, Maarten

A2 - Smeaton, Alan

PB - Springer

CY - Berlin

T2 - CLEF (Cross-Language Evaluation Forum) Conference on Multilingual and Multimodal Information Access Evaluation 2010

Y2 - 20 September 2010 through 23 September 2010

ER -

Hiemstra D, Hauff C. MapReduce for information retrieval evaluation: "Let's quickly test this on 12 TB of data". In Agosti M, Ferro N, Peters C, de Rijke M, Smeaton A, editors, International Conference of the Cross-Language Evaluation Forum, CLEF: Multilingual and Multimodal Information Access Evaluation. Berlin: Springer. 2010. p. 64-69. (Lecture Notes in Computer Science). doi: 10.1007/978-3-642-15998-5_8