What do neural networks learn in image classification? A frequency shortcut perspective

Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

Abstract

Frequency analysis is useful for understanding the mechanisms of representation learning in neural networks (NNs). Most research in this area focuses on the learning dynamics of NNs for regression tasks, while little for classification. This study empirically investigates the latter and expands the understanding of frequency shortcuts. First, we perform experiments on synthetic datasets, designed to have a bias in different frequency bands. Our results demonstrate that NNs tend to find simple solutions for classification, and what they learn first during training depends on the most distinctive frequency characteristics, which can be either low- or high-frequencies. Second, we confirm this phenomenon on natural images. We propose a metric to measure class-wise frequency characteristics and a method to identify frequency shortcuts. The results show that frequency shortcuts can be texture-based or shape-based, depending on what best simplifies the objective. Third, we validate the transferability of frequency shortcuts on out-of-distribution (OOD) test sets. Our results suggest that frequency shortcuts can be transferred across datasets and cannot be fully avoided by larger model capacity and data augmentation. We recommend that future research should focus on effective training schemes mitigating frequency shortcut learning. Codes and data are available at https://github.com/nis-research/nn-frequency-shortcuts.
Original languageEnglish
Title of host publication2023 IEEE/CVF International Conference on Computer Vision (ICCV)
Place of PublicationPiscataway, NJ
PublisherIEEE
Pages1433-1442
Number of pages10
ISBN (Electronic)979-8-3503-0718-4
ISBN (Print)979-8-3503-0719-1
DOIs
Publication statusPublished - 6 Oct 2023
Event2023 IEEE/CVF International Conference on Computer Vision, ICCV 2023 - Paris, France
Duration: 1 Oct 20236 Oct 2023

Publication series

NameProceedings IEEE/CVF International Conference on Computer Vision (ICCV)
PublisherIEEE
Volume2023
ISSN (Print)1550-5499
ISSN (Electronic)2380-7504

Conference

Conference2023 IEEE/CVF International Conference on Computer Vision, ICCV 2023
Abbreviated titleICCV
Country/TerritoryFrance
CityParis
Period1/10/236/10/23

Keywords

  • Training
  • Representation learning
  • Systematics
  • Artificial neural networks
  • Data augmentation
  • Data models
  • Frequency measurement
  • 2024 OA procedure

Fingerprint

Dive into the research topics of 'What do neural networks learn in image classification? A frequency shortcut perspective'. Together they form a unique fingerprint.

Cite this