Large-scale zero-shot learning in the wild: classifying zoological illustrations

Lise Stork*, Andreas Weber, Jaap van den Herik, Aske Plaat, Fons Verbeek, Katherine Wolstencroft

*Corresponding author for this work

Research output: Contribution to journalArticleAcademicpeer-review

Abstract

In this paper we analyse the classification of zoological illustrations. Historically, zoological illustrations were the modus operandi for the documentation of new species, and now serve as crucial sources for long-term ecological and biodiversity research. By employing computational methods for classification, the data can be made amenable to research. Automated species identification is challenging due to the long-tailed nature of the data, and the millions of possible classes in the species taxonomy. Success commonly depends on large training sets with many examples per class, but images from only a subset of classes are digitally available, and many images are unlabelled, since labelling requires domain expertise. We explore zero-shot learning to address the problem, where features are learned from classes with medium to large samples, which are then transferred to recognise classes with few or no training samples. We specifically explore how distributed, multi-modal background knowledge from data providers, such as the Global Biodiversity Information Facility (GBIF), iNaturalist, and the Biodiversity Heritage Library (BHL), can be used to share knowledge between classes for zero-shot learning. We train a prototypical network for zero-shot classification, and introduce fused prototypes (FP) and hierarchical prototype loss (HPL) to optimise the model. Finally, we analyse the performance of the model for use in real-world applications. The experimental results are encouraging, indicating potential for use of such models in an expert support system, but also express the difficulty of our task, showing a necessity for research into computer vision methods that are able to learn from small samples.
Original languageEnglish
Article number101222
JournalEcological informatics
Volume62
Early online date30 Jan 2021
DOIs
Publication statusE-pub ahead of print/First online - 30 Jan 2021

Keywords

  • machine learning
  • computer vision
  • zero-shot learning
  • digital cultural heritage
  • digital natural heritage
  • digital heritage
  • digital biodiversity heritage
  • biodiversity
  • History of Science
  • Emerging technology

Fingerprint Dive into the research topics of 'Large-scale zero-shot learning in the wild: classifying zoological illustrations'. Together they form a unique fingerprint.

Cite this