Simulating the Future of Concept-Based Video Retrieval under Improved Detector Performance

Research output: Contribution to journalArticleAcademicpeer-review

9 Citations (Scopus)
45 Downloads (Pure)

Abstract

In this paper we address the following important questions for concept-based video retrieval: (1) What is the impact of detector performance on the performance of concept-based retrieval engines, and (2) will these engines be applicable to real-life search tasks if detector performance improves in the future? We use Monte Carlo simulations to answer these questions. To generate the simulation input, we propose to use a probabilistic model of two Gaussians for the confidence scores that concept detectors emit. Modifying the model's parameters affects the detector performance and the search performance. We study the relation between these two performances on two video collections. For detectors with similar discriminative power and a concept vocabulary of around 100 concepts, the simulation reveals that in order to achieve a search performance of 0.20 mean average precision (MAP) -- which is considered sufficient performance for real-life applications -- one needs detectors with at least 0.60 MAP. We also find that, given our simulation model and low detector performance, MAP is not always a good evaluation measure for concept detectors since it is not strongly correlated with the search performance.
Original languageUndefined
Pages (from-to)203-231
Number of pages28
JournalMultimedia tools and applications
Volume1
Issue number1
DOIs
Publication statusPublished - 2012

Keywords

  • Performance Prediction
  • EWI-20201
  • METIS-279155
  • Simulation
  • Concept Detection
  • Concept-based retrieval
  • IR-78120

Cite this

@article{5e840b6d7a784608b80c786de73f6595,
title = "Simulating the Future of Concept-Based Video Retrieval under Improved Detector Performance",
abstract = "In this paper we address the following important questions for concept-based video retrieval: (1) What is the impact of detector performance on the performance of concept-based retrieval engines, and (2) will these engines be applicable to real-life search tasks if detector performance improves in the future? We use Monte Carlo simulations to answer these questions. To generate the simulation input, we propose to use a probabilistic model of two Gaussians for the confidence scores that concept detectors emit. Modifying the model's parameters affects the detector performance and the search performance. We study the relation between these two performances on two video collections. For detectors with similar discriminative power and a concept vocabulary of around 100 concepts, the simulation reveals that in order to achieve a search performance of 0.20 mean average precision (MAP) -- which is considered sufficient performance for real-life applications -- one needs detectors with at least 0.60 MAP. We also find that, given our simulation model and low detector performance, MAP is not always a good evaluation measure for concept detectors since it is not strongly correlated with the search performance.",
keywords = "Performance Prediction, EWI-20201, METIS-279155, Simulation, Concept Detection, Concept-based retrieval, IR-78120",
author = "Robin Aly and Djoerd Hiemstra and {de Jong}, {Franciska M.G.} and Apers, {Peter M.G.}",
note = "Open access article",
year = "2012",
doi = "10.1007/s11042-011-0818-x",
language = "Undefined",
volume = "1",
pages = "203--231",
journal = "Multimedia tools and applications",
issn = "1380-7501",
publisher = "Springer",
number = "1",

}

Simulating the Future of Concept-Based Video Retrieval under Improved Detector Performance. / Aly, Robin; Hiemstra, Djoerd; de Jong, Franciska M.G.; Apers, Peter M.G.

In: Multimedia tools and applications, Vol. 1, No. 1, 2012, p. 203-231.

Research output: Contribution to journalArticleAcademicpeer-review

TY - JOUR

T1 - Simulating the Future of Concept-Based Video Retrieval under Improved Detector Performance

AU - Aly, Robin

AU - Hiemstra, Djoerd

AU - de Jong, Franciska M.G.

AU - Apers, Peter M.G.

N1 - Open access article

PY - 2012

Y1 - 2012

N2 - In this paper we address the following important questions for concept-based video retrieval: (1) What is the impact of detector performance on the performance of concept-based retrieval engines, and (2) will these engines be applicable to real-life search tasks if detector performance improves in the future? We use Monte Carlo simulations to answer these questions. To generate the simulation input, we propose to use a probabilistic model of two Gaussians for the confidence scores that concept detectors emit. Modifying the model's parameters affects the detector performance and the search performance. We study the relation between these two performances on two video collections. For detectors with similar discriminative power and a concept vocabulary of around 100 concepts, the simulation reveals that in order to achieve a search performance of 0.20 mean average precision (MAP) -- which is considered sufficient performance for real-life applications -- one needs detectors with at least 0.60 MAP. We also find that, given our simulation model and low detector performance, MAP is not always a good evaluation measure for concept detectors since it is not strongly correlated with the search performance.

AB - In this paper we address the following important questions for concept-based video retrieval: (1) What is the impact of detector performance on the performance of concept-based retrieval engines, and (2) will these engines be applicable to real-life search tasks if detector performance improves in the future? We use Monte Carlo simulations to answer these questions. To generate the simulation input, we propose to use a probabilistic model of two Gaussians for the confidence scores that concept detectors emit. Modifying the model's parameters affects the detector performance and the search performance. We study the relation between these two performances on two video collections. For detectors with similar discriminative power and a concept vocabulary of around 100 concepts, the simulation reveals that in order to achieve a search performance of 0.20 mean average precision (MAP) -- which is considered sufficient performance for real-life applications -- one needs detectors with at least 0.60 MAP. We also find that, given our simulation model and low detector performance, MAP is not always a good evaluation measure for concept detectors since it is not strongly correlated with the search performance.

KW - Performance Prediction

KW - EWI-20201

KW - METIS-279155

KW - Simulation

KW - Concept Detection

KW - Concept-based retrieval

KW - IR-78120

U2 - 10.1007/s11042-011-0818-x

DO - 10.1007/s11042-011-0818-x

M3 - Article

VL - 1

SP - 203

EP - 231

JO - Multimedia tools and applications

JF - Multimedia tools and applications

SN - 1380-7501

IS - 1

ER -