A Probabilistic Multimedia Retrieval Model and its Evaluation

A.H. Sayed (Editor), T.H.W. Westerveld, A.P. de Vries, A.J. de Vries, A. van Ballegooij, Franciska M.G. de Jong, Djoerd Hiemstra

Research output: Contribution to journalArticleAcademicpeer-review

37 Citations (Scopus)
67 Downloads (Pure)

Abstract

We present a probabilistic model for the retrieval of multimodal documents. The model is based on Bayesian decision theory and combines models for text-based search with models for visual search. The textual model is based on the language modelling approach to text retrieval, and the visual information is modelled as a mixture of Gaussian densities. Both models have proved successful on various standard retrieval tasks. We evaluate the multimodal model on the search task of TREC′s video track. We found that the disclosure of video material based on visual information only is still too difficult. Even with purely visual information needs, text-based retrieval still outperforms visual approaches. The probabilistic model is useful for text, visual, and multimedia retrieval. Unfortunately, simplifying assumptions that reduce its computational complexity degrade retrieval effectiveness. Regarding the question whether the model can effectively combine information from different modalities, we conclude that whenever both modalities yield reasonable scores, a combined run outperforms the individual runs.
Original languageUndefined
Article number10.1155/S111086570321101X
Pages (from-to)186-198
Number of pages13
JournalEURASIP journal on applied signal processing
Volume2
Issue number2
DOIs
Publication statusPublished - 2003

Keywords

  • DB-MMR: MULTIMEDIA RETRIEVAL
  • METIS-216472
  • IR-66373
  • EWI-6962

Cite this

Sayed, A. H. (Ed.), Westerveld, T. H. W., de Vries, A. P., de Vries, A. J., van Ballegooij, A., de Jong, F. M. G., & Hiemstra, D. (2003). A Probabilistic Multimedia Retrieval Model and its Evaluation. EURASIP journal on applied signal processing, 2(2), 186-198. [10.1155/S111086570321101X]. https://doi.org/10.1155/S111086570321101X
Sayed, A.H. (Editor) ; Westerveld, T.H.W. ; de Vries, A.P. ; de Vries, A.J. ; van Ballegooij, A. ; de Jong, Franciska M.G. ; Hiemstra, Djoerd. / A Probabilistic Multimedia Retrieval Model and its Evaluation. In: EURASIP journal on applied signal processing. 2003 ; Vol. 2, No. 2. pp. 186-198.
@article{08b83000ab4d460786d1a1a007340d5b,
title = "A Probabilistic Multimedia Retrieval Model and its Evaluation",
abstract = "We present a probabilistic model for the retrieval of multimodal documents. The model is based on Bayesian decision theory and combines models for text-based search with models for visual search. The textual model is based on the language modelling approach to text retrieval, and the visual information is modelled as a mixture of Gaussian densities. Both models have proved successful on various standard retrieval tasks. We evaluate the multimodal model on the search task of TREC′s video track. We found that the disclosure of video material based on visual information only is still too difficult. Even with purely visual information needs, text-based retrieval still outperforms visual approaches. The probabilistic model is useful for text, visual, and multimedia retrieval. Unfortunately, simplifying assumptions that reduce its computational complexity degrade retrieval effectiveness. Regarding the question whether the model can effectively combine information from different modalities, we conclude that whenever both modalities yield reasonable scores, a combined run outperforms the individual runs.",
keywords = "DB-MMR: MULTIMEDIA RETRIEVAL, METIS-216472, IR-66373, EWI-6962",
author = "A.H. Sayed and T.H.W. Westerveld and {de Vries}, A.P. and {de Vries}, A.J. and {van Ballegooij}, A. and {de Jong}, {Franciska M.G.} and Djoerd Hiemstra",
note = "Imported from EWI/DB PMS [db-utwente:arti:0000000017]",
year = "2003",
doi = "10.1155/S111086570321101X",
language = "Undefined",
volume = "2",
pages = "186--198",
journal = "EURASIP journal on applied signal processing",
issn = "1110-8657",
publisher = "Hindawi Publishing Corporation",
number = "2",

}

Sayed, AH (ed.), Westerveld, THW, de Vries, AP, de Vries, AJ, van Ballegooij, A, de Jong, FMG & Hiemstra, D 2003, 'A Probabilistic Multimedia Retrieval Model and its Evaluation', EURASIP journal on applied signal processing, vol. 2, no. 2, 10.1155/S111086570321101X, pp. 186-198. https://doi.org/10.1155/S111086570321101X

A Probabilistic Multimedia Retrieval Model and its Evaluation. / Sayed, A.H. (Editor); Westerveld, T.H.W.; de Vries, A.P.; de Vries, A.J.; van Ballegooij, A.; de Jong, Franciska M.G.; Hiemstra, Djoerd.

In: EURASIP journal on applied signal processing, Vol. 2, No. 2, 10.1155/S111086570321101X, 2003, p. 186-198.

Research output: Contribution to journalArticleAcademicpeer-review

TY - JOUR

T1 - A Probabilistic Multimedia Retrieval Model and its Evaluation

AU - Westerveld, T.H.W.

AU - de Vries, A.P.

AU - de Vries, A.J.

AU - van Ballegooij, A.

AU - de Jong, Franciska M.G.

AU - Hiemstra, Djoerd

A2 - Sayed, A.H.

N1 - Imported from EWI/DB PMS [db-utwente:arti:0000000017]

PY - 2003

Y1 - 2003

N2 - We present a probabilistic model for the retrieval of multimodal documents. The model is based on Bayesian decision theory and combines models for text-based search with models for visual search. The textual model is based on the language modelling approach to text retrieval, and the visual information is modelled as a mixture of Gaussian densities. Both models have proved successful on various standard retrieval tasks. We evaluate the multimodal model on the search task of TREC′s video track. We found that the disclosure of video material based on visual information only is still too difficult. Even with purely visual information needs, text-based retrieval still outperforms visual approaches. The probabilistic model is useful for text, visual, and multimedia retrieval. Unfortunately, simplifying assumptions that reduce its computational complexity degrade retrieval effectiveness. Regarding the question whether the model can effectively combine information from different modalities, we conclude that whenever both modalities yield reasonable scores, a combined run outperforms the individual runs.

AB - We present a probabilistic model for the retrieval of multimodal documents. The model is based on Bayesian decision theory and combines models for text-based search with models for visual search. The textual model is based on the language modelling approach to text retrieval, and the visual information is modelled as a mixture of Gaussian densities. Both models have proved successful on various standard retrieval tasks. We evaluate the multimodal model on the search task of TREC′s video track. We found that the disclosure of video material based on visual information only is still too difficult. Even with purely visual information needs, text-based retrieval still outperforms visual approaches. The probabilistic model is useful for text, visual, and multimedia retrieval. Unfortunately, simplifying assumptions that reduce its computational complexity degrade retrieval effectiveness. Regarding the question whether the model can effectively combine information from different modalities, we conclude that whenever both modalities yield reasonable scores, a combined run outperforms the individual runs.

KW - DB-MMR: MULTIMEDIA RETRIEVAL

KW - METIS-216472

KW - IR-66373

KW - EWI-6962

U2 - 10.1155/S111086570321101X

DO - 10.1155/S111086570321101X

M3 - Article

VL - 2

SP - 186

EP - 198

JO - EURASIP journal on applied signal processing

JF - EURASIP journal on applied signal processing

SN - 1110-8657

IS - 2

M1 - 10.1155/S111086570321101X

ER -

Sayed AH, (ed.), Westerveld THW, de Vries AP, de Vries AJ, van Ballegooij A, de Jong FMG et al. A Probabilistic Multimedia Retrieval Model and its Evaluation. EURASIP journal on applied signal processing. 2003;2(2):186-198. 10.1155/S111086570321101X. https://doi.org/10.1155/S111086570321101X