Exploiting User Disagreement for Web Search Evaluation: an Experimental Approach

Thomas Demeester, Robin Aly, Djoerd Hiemstra, Dong-Phuong Nguyen, Rudolf Berend Trieschnigg, Chris Develder

Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

10 Citations (Scopus)
55 Downloads (Pure)


To express a more nuanced notion of relevance as compared to binary judgments, graded relevance levels can be used for the evaluation of search results. Especially in Web search, users strongly prefer top results over less relevant results, and yet they often disagree on which are the top results for a given information need. Whereas previous works have generally considered disagreement as a negative effect, this paper proposes a method to exploit this user disagreement by integrating it into the evaluation procedure. First, we present experiments that investigate the user disagreement. We argue that, with a high disagreement, lower relevance levels might need to be promoted more than in the case where there is global consensus on the top results. This is formalized by introducing the User Disagreement Model, resulting in a weighting of the relevance levels with a probabilistic interpretation. A validity analysis is given, and we explain how to integrate the model with well-established evaluation metrics. Finally, we discuss a specific application of the model, in the estimation of suitable weights for the combined relevance of Web search snippets and pages.
Original languageUndefined
Title of host publicationProceedings of the 7th ACM International Conference on Web Search and Data Mining, WSDM 2014
Place of PublicationNew York
PublisherAssociation for Computing Machinery
Number of pages10
ISBN (Print)978-1-4503-2351-2
Publication statusPublished - Feb 2014
Event7th ACM International Conference on Web Search and Data Mining, WSDM 2014 - New York, United States
Duration: 24 Feb 201428 Feb 2014
Conference number: 7

Publication series



Conference7th ACM International Conference on Web Search and Data Mining, WSDM 2014
Abbreviated titleWSDM
Country/TerritoryUnited States
CityNew York


  • EWI-25653
  • IR-94305
  • METIS-309865

Cite this