A comparison of user and system query performance predictions

C. Hauff, Diane Kelly, Leif Azzopardi

    Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

    19 Citations (Scopus)

    Abstract

    Query performance prediction methods are usually applied to estimate the retrieval effectiveness of queries, where the evaluation is largely system sided. However, little work has been conducted to understand query performance prediction from the user's perspective. The question we consider is, whether the predictions of query performance that systems make are in line with the predictions that users make. To this aim, we compare the performance ratings users assign to queries with the performance scores estimated by a range of pre-retrieval and post-retrieval query performance predictors. Two studies are presented that explore the relationship between user ratings and system predictions on two levels: (i) the topic level, and, (ii) the query suggestions level. It is shown that when predicting the performance of query suggestions, user ratings were mostly uncorrelated with system predictions. At the topic level though, where a single query is judged for each information need, we observed moderate correlations between user ratings and a subset of system predictions. As query performance prediction methods are often based on intuitions of how users might rate queries, these findings suggest that such methods are not representative of how users actually rate query suggestions and topics. This motivates further research into understanding the rating process engaged by users, and developing models of query performance prediction in order to bridge the divide between systems and users.
    Original languageUndefined
    Title of host publicationProceedings of the 19th ACM International Conference on Information and Knowledge Management (CIKM '10)
    Place of PublicationNew York
    PublisherAssociation for Computing Machinery (ACM)
    Pages979-988
    Number of pages10
    ISBN (Print)978-1-4503-0099-5
    DOIs
    Publication statusPublished - Oct 2010
    Event19th ACM International Conference on Information and Knowledge Management, CIKM 2010 - Toronto, Canada
    Duration: 26 Oct 201030 Oct 2010
    Conference number: 19

    Publication series

    Name
    PublisherACM

    Conference

    Conference19th ACM International Conference on Information and Knowledge Management, CIKM 2010
    Abbreviated titleCIKM
    CountryCanada
    CityToronto
    Period26/10/1030/10/10

    Keywords

    • METIS-271107
    • IR-74334
    • Query performance prediction
    • EWI-18716
    • user ratings
    • CR-H.3.3
    • Information Retrieval

    Cite this

    Hauff, C., Kelly, D., & Azzopardi, L. (2010). A comparison of user and system query performance predictions. In Proceedings of the 19th ACM International Conference on Information and Knowledge Management (CIKM '10) (pp. 979-988). New York: Association for Computing Machinery (ACM). https://doi.org/10.1145/1871437.1871562