Evaluating ASR Output for Information Retrieval

Laurens Bastiaan van der Werff; W.F.L. Heeren

Evaluating ASR Output for Information Retrieval

Laurens Bastiaan van der Werff, W.F.L. Heeren

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

Abstract

Within the context of international benchmarks and collection specific projects, much work on spoken document retrieval has been done in recent years. In 2000 the issue of automatic speech recognition for spoken document retrieval was declared 'solved' for the broadcast news domain. Many collections, however, are not in this domain and automatic speech recognition for these collections may contain specific new challenges. This requires a method to evaluate automatic speech recognition optimization schemes for these application areas. Traditional measures such as word error rate and story word error rate are not ideal for this. In this paper, three new metrics are proposed. Their behaviour is investigated on a cultural heritage collection and performance is compared to traditional measurements on TREC broadcast news data.

Original language	Undefined
Title of host publication	Proceedings of the ACM SIGIR Workshop on Searching Spontaneous Conversational Speech
Editors	Franciska M.G. de Jong, D.W. Oard, Roeland J.F. Ordelman, S. Raaijmakers
Place of Publication	Enschede
Publisher	Centre for Telematics and Information Technology (CTIT)
Pages	7-14
Number of pages	8
ISBN (Print)	978-90-365-2542-8
Publication status	Published - Jul 2007
Event	ACM/SIGIR Workshop on Searching Spontaneous Conversational Speech, SSCS 2007 - Amsterdam, Netherlands Duration: 27 Jul 2007 → 27 Jul 2007

Publication series

Name
Publisher	Centre for Telematics and Information Technology, University of Twente
Number	LNCS4549

Workshop

Workshop	ACM/SIGIR Workshop on Searching Spontaneous Conversational Speech, SSCS 2007
Abbreviated title	SSCS
Country/Territory	Netherlands
City	Amsterdam
Period	27/07/07 → 27/07/07

Keywords

EWI-10826
IR-61873
Evaluation
METIS-241816
Lattices
Spoken Document Retrieval
Automatic Speech Recognition

Access to Document

http://eprints.eemcs.utwente.nl/secure2/10826/01/ASR4SDR_SSCS2007_final.pdf

Cite this

van der Werff, L. B., & Heeren, W. F. L. (2007). Evaluating ASR Output for Information Retrieval. In F. M. G. de Jong, D. W. Oard, R. J. F. Ordelman, & S. Raaijmakers (Eds.), Proceedings of the ACM SIGIR Workshop on Searching Spontaneous Conversational Speech (pp. 7-14). Centre for Telematics and Information Technology (CTIT). http://eprints.eemcs.utwente.nl/secure2/10826/01/ASR4SDR_SSCS2007_final.pdf

@inproceedings{deed466d75a842969e9d843af5c2f10c,

title = "Evaluating ASR Output for Information Retrieval",

abstract = "Within the context of international benchmarks and collection specific projects, much work on spoken document retrieval has been done in recent years. In 2000 the issue of automatic speech recognition for spoken document retrieval was declared 'solved' for the broadcast news domain. Many collections, however, are not in this domain and automatic speech recognition for these collections may contain specific new challenges. This requires a method to evaluate automatic speech recognition optimization schemes for these application areas. Traditional measures such as word error rate and story word error rate are not ideal for this. In this paper, three new metrics are proposed. Their behaviour is investigated on a cultural heritage collection and performance is compared to traditional measurements on TREC broadcast news data.",

keywords = "EWI-10826, IR-61873, Evaluation, METIS-241816, Lattices, Spoken Document Retrieval, Automatic Speech Recognition",

author = "{van der Werff}, {Laurens Bastiaan} and W.F.L. Heeren",

year = "2007",

month = jul,

language = "Undefined",

isbn = "978-90-365-2542-8",

publisher = "Centre for Telematics and Information Technology (CTIT)",

number = "LNCS4549",

pages = "7--14",

editor = "{de Jong}, {Franciska M.G.} and D.W. Oard and Ordelman, {Roeland J.F.} and S. Raaijmakers",

booktitle = "Proceedings of the ACM SIGIR Workshop on Searching Spontaneous Conversational Speech",

address = "Netherlands",

note = "null ; Conference date: 27-07-2007 Through 27-07-2007",

}

van der Werff, LB & Heeren, WFL 2007, Evaluating ASR Output for Information Retrieval. in FMG de Jong, DW Oard, RJF Ordelman & S Raaijmakers (eds), Proceedings of the ACM SIGIR Workshop on Searching Spontaneous Conversational Speech. Centre for Telematics and Information Technology (CTIT), Enschede, pp. 7-14, ACM/SIGIR Workshop on Searching Spontaneous Conversational Speech, SSCS 2007, Amsterdam, Netherlands, 27/07/07. <http://eprints.eemcs.utwente.nl/secure2/10826/01/ASR4SDR_SSCS2007_final.pdf>

Evaluating ASR Output for Information Retrieval. / van der Werff, Laurens Bastiaan; Heeren, W.F.L.

Proceedings of the ACM SIGIR Workshop on Searching Spontaneous Conversational Speech. ed. / Franciska M.G. de Jong; D.W. Oard; Roeland J.F. Ordelman; S. Raaijmakers. Enschede : Centre for Telematics and Information Technology (CTIT), 2007. p. 7-14.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

TY - GEN

T1 - Evaluating ASR Output for Information Retrieval

AU - van der Werff, Laurens Bastiaan

AU - Heeren, W.F.L.

PY - 2007/7

Y1 - 2007/7

N2 - Within the context of international benchmarks and collection specific projects, much work on spoken document retrieval has been done in recent years. In 2000 the issue of automatic speech recognition for spoken document retrieval was declared 'solved' for the broadcast news domain. Many collections, however, are not in this domain and automatic speech recognition for these collections may contain specific new challenges. This requires a method to evaluate automatic speech recognition optimization schemes for these application areas. Traditional measures such as word error rate and story word error rate are not ideal for this. In this paper, three new metrics are proposed. Their behaviour is investigated on a cultural heritage collection and performance is compared to traditional measurements on TREC broadcast news data.

AB - Within the context of international benchmarks and collection specific projects, much work on spoken document retrieval has been done in recent years. In 2000 the issue of automatic speech recognition for spoken document retrieval was declared 'solved' for the broadcast news domain. Many collections, however, are not in this domain and automatic speech recognition for these collections may contain specific new challenges. This requires a method to evaluate automatic speech recognition optimization schemes for these application areas. Traditional measures such as word error rate and story word error rate are not ideal for this. In this paper, three new metrics are proposed. Their behaviour is investigated on a cultural heritage collection and performance is compared to traditional measurements on TREC broadcast news data.

KW - EWI-10826

KW - IR-61873

KW - Evaluation

KW - METIS-241816

KW - Lattices

KW - Spoken Document Retrieval

KW - Automatic Speech Recognition

M3 - Conference contribution

SN - 978-90-365-2542-8

SP - 7

EP - 14

BT - Proceedings of the ACM SIGIR Workshop on Searching Spontaneous Conversational Speech

A2 - de Jong, Franciska M.G.

A2 - Oard, D.W.

A2 - Ordelman, Roeland J.F.

A2 - Raaijmakers, S.

PB - Centre for Telematics and Information Technology (CTIT)

CY - Enschede

Y2 - 27 July 2007 through 27 July 2007

ER -