Simulating the Future of Concept-Based Video Retrieval under Improved Detector Performance

Robin Aly, Djoerd Hiemstra, Franciska M.G. de Jong, Peter M.G. Apers

Research output: Contribution to journalArticleAcademicpeer-review

10 Citations (Scopus)
134 Downloads (Pure)


In this paper we address the following important questions for concept-based video retrieval: (1) What is the impact of detector performance on the performance of concept-based retrieval engines, and (2) will these engines be applicable to real-life search tasks if detector performance improves in the future? We use Monte Carlo simulations to answer these questions. To generate the simulation input, we propose to use a probabilistic model of two Gaussians for the confidence scores that concept detectors emit. Modifying the model's parameters affects the detector performance and the search performance. We study the relation between these two performances on two video collections. For detectors with similar discriminative power and a concept vocabulary of around 100 concepts, the simulation reveals that in order to achieve a search performance of 0.20 mean average precision (MAP) -- which is considered sufficient performance for real-life applications -- one needs detectors with at least 0.60 MAP. We also find that, given our simulation model and low detector performance, MAP is not always a good evaluation measure for concept detectors since it is not strongly correlated with the search performance.
Original languageUndefined
Pages (from-to)203-231
Number of pages28
JournalMultimedia tools and applications
Issue number1
Publication statusPublished - 2012


  • Performance Prediction
  • EWI-20201
  • METIS-279155
  • Simulation
  • Concept Detection
  • Concept-based retrieval
  • IR-78120

Cite this