Concept detectors: how good is good enough?

Robin Aly, Djoerd Hiemstra

Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

7 Citations (Scopus)

Abstract

Today, semantic concept based video retrieval systems often show insufficient performance for real-life applications. Clearly, a big share of the reason is the lacking performance of the detectors of these concepts. While concept detectors are on their endeavor to improve, following important questions need to be addressed: ``How good do detectors need to be to produce usable search systems?'' and ``How does the detector performance influence different concept combination methods?''. We use Monte Carlo Simulations to provide answers to the above questions. The main contribution of this paper is a probabilistic model of detectors which outputs confidence scores to indicate the likelihood of a concept to occur. This score is also converted into a posterior probability and a binary classification. We investigate the influence of changes to the model's parameters on the performance of multiple concept combination methods. Current web search engines produce a mean average precision (MAP) of around $0.20$. Our simulation reveals that the best performing video search method achieve this performance using detectors with $0.60$ MAP and is therefore usable in real-life. Furthermore, perfect detection allows the best performing combination method to produce $0.39$ search MAP in an artificial environment with Oracle settings. We also find that MAP is not necessarily a good evaluation measure for concept detectors since it is not always correlated with search performance.
Original languageUndefined
Title of host publicationProceeding of the 17th ACM International Conference on Multimedia
Place of PublicationNew York
PublisherAssociation for Computing Machinery (ACM)
Pages233-242
Number of pages10
ISBN (Print)978-1-60558-608-3
DOIs
Publication statusPublished - 2009
Event17th ACM International Conference on Multimedia, MM 2009 - Beijing, China
Duration: 19 Oct 200922 Oct 2009
Conference number: 17

Publication series

Name
PublisherACM

Conference

Conference17th ACM International Conference on Multimedia, MM 2009
Abbreviated titleMM
CountryChina
CityBeijing
Period19/10/0922/10/09

Keywords

  • METIS-264419
  • CR-H.3.3
  • EWI-15905
  • IR-68254

Cite this