Speech Transcript Evaluation for Information Retrieval

Laurens Bastiaan van der Werff, Wessel Kraaij, Franciska M.G. de Jong

    Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

    36 Downloads (Pure)


    Speech recognition transcripts are being used in various fields of research and practical applications, putting various demands on their accuracy. Traditionally ASR research has used intrinsic evaluation measures such as word error rate to determine transcript quality. In non-dictation-type applications such as speech retrieval, it is better to use extrinsic (or task specific) measures. Indexation and the associated processing may eliminate certain errors, whereas the search query may reveal others. In this work, we argue that the standard extrinsic speech retrieval measure average precision is unpractical for ASR evaluation. As an alternative we propose the use of ranked correlation measures on the output of the speech retrieval task, with the goal of predicting relative mean average precision. The measures we used showed a reasonably high correlation with average precision, but require much less human effort to calculate and can be more easily deployed in a variety of real-life settings.
    Original languageUndefined
    Title of host publication12th Annual Conference of the International Speech Communication Association, Interspeech 2011
    Place of PublicationAvignon, France
    PublisherInternational Speech Communication Association (ISCA)
    Number of pages4
    ISBN (Print)1990-9772
    Publication statusPublished - Aug 2011
    Event12th Annual Conference of the International Speech Communication Association, INTERSPEECH 2011 - Florence, Italy
    Duration: 28 Aug 201131 Aug 2011
    Conference number: 12

    Publication series

    PublisherInternational Speech Communication Association
    ISSN (Print)1990-9772


    Conference12th Annual Conference of the International Speech Communication Association, INTERSPEECH 2011
    Abbreviated titleINTERSPEECH


    • IR-78194
    • METIS-278845
    • Evaluation
    • EWI-20617
    • Speech retrieval
    • Information Retrieval
    • rank correlation
    • Speech Recognition

    Cite this