Luhn Revisited: Significant Words Language Models

M. Dehghani, H. Azarbonyad, J. Kamps, Djoerd Hiemstra, M. Marx

Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

23 Citations (Scopus)
153 Downloads (Pure)


Users tend to articulate their complex information needs in only a few keywords, making underspecified statements of request the main bottleneck for retrieval effectiveness. Taking advantage of feedback information is one of the best ways to enrich the query representation, but can also lead to loss of query focus and harm performance - in particular when the initial query retrieves only little relevant information - when overfitting to accidental features of the particular observed feedback documents. Inspired by the early work of Hans Peter Luhn, we propose significant words language models of feedback documents that capture all, and only, the significant shared terms from feedback documents. We adjust the weights of common terms that are already well explained by the document collection as well as the weight of rare terms that are only explained by specific feedback documents, which eventually results in having only the significant terms left in the feedback model. Establishing a set of 'Significant Words' Our main contributions are the following. First, we present significant words language models as the effective models capturing the essential terms and their probabilities. Second, we apply the resulting models to the relevance feedback task, and see a better performance over the state-of-the-art methods. Third, we see that the estimation method is remarkably robust making the models insensitive to noisy non-relevant terms in feedback documents. Our general observation is that the significant words language models more accurately capture relevance by excluding general terms and feedback document specific terms.
Original languageUndefined
Title of host publicationProceedings of the 25th ACM International on Conference on Information and Knowledge Management (CIKM 2016)
Place of PublicationNew York
PublisherAssociation for Computing Machinery
Number of pages10
ISBN (Print)978-1-4503-4073-1
Publication statusPublished - Oct 2016
Event25th ACM International on Conference on Information and Knowledge Management, CIKM 2016 - Indianapolis, United States
Duration: 24 Oct 201628 Oct 2016
Conference number: 25


Conference25th ACM International on Conference on Information and Knowledge Management, CIKM 2016
Abbreviated titleCIKM
Country/TerritoryUnited States


  • EWI-27808

Cite this