A comparison of user and system query performance predictions

C. Hauff, Diane Kelly, Leif Azzopardi

    Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

    17 Citations (Scopus)

    Abstract

    Query performance prediction methods are usually applied to estimate the retrieval effectiveness of queries, where the evaluation is largely system sided. However, little work has been conducted to understand query performance prediction from the user's perspective. The question we consider is, whether the predictions of query performance that systems make are in line with the predictions that users make. To this aim, we compare the performance ratings users assign to queries with the performance scores estimated by a range of pre-retrieval and post-retrieval query performance predictors. Two studies are presented that explore the relationship between user ratings and system predictions on two levels: (i) the topic level, and, (ii) the query suggestions level. It is shown that when predicting the performance of query suggestions, user ratings were mostly uncorrelated with system predictions. At the topic level though, where a single query is judged for each information need, we observed moderate correlations between user ratings and a subset of system predictions. As query performance prediction methods are often based on intuitions of how users might rate queries, these findings suggest that such methods are not representative of how users actually rate query suggestions and topics. This motivates further research into understanding the rating process engaged by users, and developing models of query performance prediction in order to bridge the divide between systems and users.
    Original languageUndefined
    Title of host publicationProceedings of the 19th ACM International Conference on Information and Knowledge Management (CIKM '10)
    Place of PublicationNew York
    PublisherAssociation for Computing Machinery (ACM)
    Pages979-988
    Number of pages10
    ISBN (Print)978-1-4503-0099-5
    DOIs
    Publication statusPublished - Oct 2010
    Event19th ACM International Conference on Information and Knowledge Management, CIKM 2010 - Toronto, Canada
    Duration: 26 Oct 201030 Oct 2010
    Conference number: 19

    Publication series

    Name
    PublisherACM

    Conference

    Conference19th ACM International Conference on Information and Knowledge Management, CIKM 2010
    Abbreviated titleCIKM
    CountryCanada
    CityToronto
    Period26/10/1030/10/10

    Keywords

    • METIS-271107
    • IR-74334
    • Query performance prediction
    • EWI-18716
    • user ratings
    • CR-H.3.3
    • Information Retrieval

    Cite this

    Hauff, C., Kelly, D., & Azzopardi, L. (2010). A comparison of user and system query performance predictions. In Proceedings of the 19th ACM International Conference on Information and Knowledge Management (CIKM '10) (pp. 979-988). New York: Association for Computing Machinery (ACM). https://doi.org/10.1145/1871437.1871562
    Hauff, C. ; Kelly, Diane ; Azzopardi, Leif. / A comparison of user and system query performance predictions. Proceedings of the 19th ACM International Conference on Information and Knowledge Management (CIKM '10). New York : Association for Computing Machinery (ACM), 2010. pp. 979-988
    @inproceedings{2ee823edfc9d41409327e7b1481bfc89,
    title = "A comparison of user and system query performance predictions",
    abstract = "Query performance prediction methods are usually applied to estimate the retrieval effectiveness of queries, where the evaluation is largely system sided. However, little work has been conducted to understand query performance prediction from the user's perspective. The question we consider is, whether the predictions of query performance that systems make are in line with the predictions that users make. To this aim, we compare the performance ratings users assign to queries with the performance scores estimated by a range of pre-retrieval and post-retrieval query performance predictors. Two studies are presented that explore the relationship between user ratings and system predictions on two levels: (i) the topic level, and, (ii) the query suggestions level. It is shown that when predicting the performance of query suggestions, user ratings were mostly uncorrelated with system predictions. At the topic level though, where a single query is judged for each information need, we observed moderate correlations between user ratings and a subset of system predictions. As query performance prediction methods are often based on intuitions of how users might rate queries, these findings suggest that such methods are not representative of how users actually rate query suggestions and topics. This motivates further research into understanding the rating process engaged by users, and developing models of query performance prediction in order to bridge the divide between systems and users.",
    keywords = "METIS-271107, IR-74334, Query performance prediction, EWI-18716, user ratings, CR-H.3.3, Information Retrieval",
    author = "C. Hauff and Diane Kelly and Leif Azzopardi",
    note = "10.1145/1871437.1871562",
    year = "2010",
    month = "10",
    doi = "10.1145/1871437.1871562",
    language = "Undefined",
    isbn = "978-1-4503-0099-5",
    publisher = "Association for Computing Machinery (ACM)",
    pages = "979--988",
    booktitle = "Proceedings of the 19th ACM International Conference on Information and Knowledge Management (CIKM '10)",
    address = "United States",

    }

    Hauff, C, Kelly, D & Azzopardi, L 2010, A comparison of user and system query performance predictions. in Proceedings of the 19th ACM International Conference on Information and Knowledge Management (CIKM '10). Association for Computing Machinery (ACM), New York, pp. 979-988, 19th ACM International Conference on Information and Knowledge Management, CIKM 2010, Toronto, Canada, 26/10/10. https://doi.org/10.1145/1871437.1871562

    A comparison of user and system query performance predictions. / Hauff, C.; Kelly, Diane; Azzopardi, Leif.

    Proceedings of the 19th ACM International Conference on Information and Knowledge Management (CIKM '10). New York : Association for Computing Machinery (ACM), 2010. p. 979-988.

    Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

    TY - GEN

    T1 - A comparison of user and system query performance predictions

    AU - Hauff, C.

    AU - Kelly, Diane

    AU - Azzopardi, Leif

    N1 - 10.1145/1871437.1871562

    PY - 2010/10

    Y1 - 2010/10

    N2 - Query performance prediction methods are usually applied to estimate the retrieval effectiveness of queries, where the evaluation is largely system sided. However, little work has been conducted to understand query performance prediction from the user's perspective. The question we consider is, whether the predictions of query performance that systems make are in line with the predictions that users make. To this aim, we compare the performance ratings users assign to queries with the performance scores estimated by a range of pre-retrieval and post-retrieval query performance predictors. Two studies are presented that explore the relationship between user ratings and system predictions on two levels: (i) the topic level, and, (ii) the query suggestions level. It is shown that when predicting the performance of query suggestions, user ratings were mostly uncorrelated with system predictions. At the topic level though, where a single query is judged for each information need, we observed moderate correlations between user ratings and a subset of system predictions. As query performance prediction methods are often based on intuitions of how users might rate queries, these findings suggest that such methods are not representative of how users actually rate query suggestions and topics. This motivates further research into understanding the rating process engaged by users, and developing models of query performance prediction in order to bridge the divide between systems and users.

    AB - Query performance prediction methods are usually applied to estimate the retrieval effectiveness of queries, where the evaluation is largely system sided. However, little work has been conducted to understand query performance prediction from the user's perspective. The question we consider is, whether the predictions of query performance that systems make are in line with the predictions that users make. To this aim, we compare the performance ratings users assign to queries with the performance scores estimated by a range of pre-retrieval and post-retrieval query performance predictors. Two studies are presented that explore the relationship between user ratings and system predictions on two levels: (i) the topic level, and, (ii) the query suggestions level. It is shown that when predicting the performance of query suggestions, user ratings were mostly uncorrelated with system predictions. At the topic level though, where a single query is judged for each information need, we observed moderate correlations between user ratings and a subset of system predictions. As query performance prediction methods are often based on intuitions of how users might rate queries, these findings suggest that such methods are not representative of how users actually rate query suggestions and topics. This motivates further research into understanding the rating process engaged by users, and developing models of query performance prediction in order to bridge the divide between systems and users.

    KW - METIS-271107

    KW - IR-74334

    KW - Query performance prediction

    KW - EWI-18716

    KW - user ratings

    KW - CR-H.3.3

    KW - Information Retrieval

    U2 - 10.1145/1871437.1871562

    DO - 10.1145/1871437.1871562

    M3 - Conference contribution

    SN - 978-1-4503-0099-5

    SP - 979

    EP - 988

    BT - Proceedings of the 19th ACM International Conference on Information and Knowledge Management (CIKM '10)

    PB - Association for Computing Machinery (ACM)

    CY - New York

    ER -

    Hauff C, Kelly D, Azzopardi L. A comparison of user and system query performance predictions. In Proceedings of the 19th ACM International Conference on Information and Knowledge Management (CIKM '10). New York: Association for Computing Machinery (ACM). 2010. p. 979-988 https://doi.org/10.1145/1871437.1871562