Speech Transcript Evaluation for Information Retrieval

Laurens Bastiaan van der Werff, Wessel Kraaij, Franciska M.G. de Jong

    Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

    25 Downloads (Pure)

    Abstract

    Speech recognition transcripts are being used in various fields of research and practical applications, putting various demands on their accuracy. Traditionally ASR research has used intrinsic evaluation measures such as word error rate to determine transcript quality. In non-dictation-type applications such as speech retrieval, it is better to use extrinsic (or task specific) measures. Indexation and the associated processing may eliminate certain errors, whereas the search query may reveal others. In this work, we argue that the standard extrinsic speech retrieval measure average precision is unpractical for ASR evaluation. As an alternative we propose the use of ranked correlation measures on the output of the speech retrieval task, with the goal of predicting relative mean average precision. The measures we used showed a reasonably high correlation with average precision, but require much less human effort to calculate and can be more easily deployed in a variety of real-life settings.
    Original languageUndefined
    Title of host publication12th Annual Conference of the International Speech Communication Association, Interspeech 2011
    Place of PublicationAvignon, France
    PublisherInternational Speech Communication Association (ISCA)
    Pages1525-1528
    Number of pages4
    ISBN (Print)1990-9772
    Publication statusPublished - Aug 2011
    Event12th Annual Conference of the International Speech Communication Association, INTERSPEECH 2011 - Florence, Italy
    Duration: 28 Aug 201131 Aug 2011
    Conference number: 12

    Publication series

    Name
    PublisherInternational Speech Communication Association
    ISSN (Print)1990-9772

    Conference

    Conference12th Annual Conference of the International Speech Communication Association, INTERSPEECH 2011
    Abbreviated titleINTERSPEECH
    CountryItaly
    CityFlorence
    Period28/08/1131/08/11

    Keywords

    • IR-78194
    • METIS-278845
    • Evaluation
    • EWI-20617
    • Speech retrieval
    • Information Retrieval
    • rank correlation
    • Speech Recognition

    Cite this

    van der Werff, L. B., Kraaij, W., & de Jong, F. M. G. (2011). Speech Transcript Evaluation for Information Retrieval. In 12th Annual Conference of the International Speech Communication Association, Interspeech 2011 (pp. 1525-1528). Avignon, France: International Speech Communication Association (ISCA).
    van der Werff, Laurens Bastiaan ; Kraaij, Wessel ; de Jong, Franciska M.G. / Speech Transcript Evaluation for Information Retrieval. 12th Annual Conference of the International Speech Communication Association, Interspeech 2011. Avignon, France : International Speech Communication Association (ISCA), 2011. pp. 1525-1528
    @inproceedings{a714b6e0c2ff4dbfa02bd9e41c93be9c,
    title = "Speech Transcript Evaluation for Information Retrieval",
    abstract = "Speech recognition transcripts are being used in various fields of research and practical applications, putting various demands on their accuracy. Traditionally ASR research has used intrinsic evaluation measures such as word error rate to determine transcript quality. In non-dictation-type applications such as speech retrieval, it is better to use extrinsic (or task specific) measures. Indexation and the associated processing may eliminate certain errors, whereas the search query may reveal others. In this work, we argue that the standard extrinsic speech retrieval measure average precision is unpractical for ASR evaluation. As an alternative we propose the use of ranked correlation measures on the output of the speech retrieval task, with the goal of predicting relative mean average precision. The measures we used showed a reasonably high correlation with average precision, but require much less human effort to calculate and can be more easily deployed in a variety of real-life settings.",
    keywords = "IR-78194, METIS-278845, Evaluation, EWI-20617, Speech retrieval, Information Retrieval, rank correlation, Speech Recognition",
    author = "{van der Werff}, {Laurens Bastiaan} and Wessel Kraaij and {de Jong}, {Franciska M.G.}",
    year = "2011",
    month = "8",
    language = "Undefined",
    isbn = "1990-9772",
    publisher = "International Speech Communication Association (ISCA)",
    pages = "1525--1528",
    booktitle = "12th Annual Conference of the International Speech Communication Association, Interspeech 2011",

    }

    van der Werff, LB, Kraaij, W & de Jong, FMG 2011, Speech Transcript Evaluation for Information Retrieval. in 12th Annual Conference of the International Speech Communication Association, Interspeech 2011. International Speech Communication Association (ISCA), Avignon, France, pp. 1525-1528, 12th Annual Conference of the International Speech Communication Association, INTERSPEECH 2011, Florence, Italy, 28/08/11.

    Speech Transcript Evaluation for Information Retrieval. / van der Werff, Laurens Bastiaan; Kraaij, Wessel; de Jong, Franciska M.G.

    12th Annual Conference of the International Speech Communication Association, Interspeech 2011. Avignon, France : International Speech Communication Association (ISCA), 2011. p. 1525-1528.

    Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

    TY - GEN

    T1 - Speech Transcript Evaluation for Information Retrieval

    AU - van der Werff, Laurens Bastiaan

    AU - Kraaij, Wessel

    AU - de Jong, Franciska M.G.

    PY - 2011/8

    Y1 - 2011/8

    N2 - Speech recognition transcripts are being used in various fields of research and practical applications, putting various demands on their accuracy. Traditionally ASR research has used intrinsic evaluation measures such as word error rate to determine transcript quality. In non-dictation-type applications such as speech retrieval, it is better to use extrinsic (or task specific) measures. Indexation and the associated processing may eliminate certain errors, whereas the search query may reveal others. In this work, we argue that the standard extrinsic speech retrieval measure average precision is unpractical for ASR evaluation. As an alternative we propose the use of ranked correlation measures on the output of the speech retrieval task, with the goal of predicting relative mean average precision. The measures we used showed a reasonably high correlation with average precision, but require much less human effort to calculate and can be more easily deployed in a variety of real-life settings.

    AB - Speech recognition transcripts are being used in various fields of research and practical applications, putting various demands on their accuracy. Traditionally ASR research has used intrinsic evaluation measures such as word error rate to determine transcript quality. In non-dictation-type applications such as speech retrieval, it is better to use extrinsic (or task specific) measures. Indexation and the associated processing may eliminate certain errors, whereas the search query may reveal others. In this work, we argue that the standard extrinsic speech retrieval measure average precision is unpractical for ASR evaluation. As an alternative we propose the use of ranked correlation measures on the output of the speech retrieval task, with the goal of predicting relative mean average precision. The measures we used showed a reasonably high correlation with average precision, but require much less human effort to calculate and can be more easily deployed in a variety of real-life settings.

    KW - IR-78194

    KW - METIS-278845

    KW - Evaluation

    KW - EWI-20617

    KW - Speech retrieval

    KW - Information Retrieval

    KW - rank correlation

    KW - Speech Recognition

    M3 - Conference contribution

    SN - 1990-9772

    SP - 1525

    EP - 1528

    BT - 12th Annual Conference of the International Speech Communication Association, Interspeech 2011

    PB - International Speech Communication Association (ISCA)

    CY - Avignon, France

    ER -

    van der Werff LB, Kraaij W, de Jong FMG. Speech Transcript Evaluation for Information Retrieval. In 12th Annual Conference of the International Speech Communication Association, Interspeech 2011. Avignon, France: International Speech Communication Association (ISCA). 2011. p. 1525-1528