Multi-Resolution Feature Fusion for Image Classification of Building Damages with Convolutional Neural Networks

D. Duarte (Corresponding Author), F. Nex, N. Kerle, G. Vosselman

Research output: Contribution to journalArticleAcademicpeer-review

8 Citations (Scopus)
8 Downloads (Pure)

Abstract

Remote sensing images have long been preferred to perform building damage assessments. The recently proposed methods to extract damaged regions from remote sensing imagery rely on convolutional neural networks (CNN). The common approach is to train a CNN independently considering each of the different resolution levels (satellite, aerial, and terrestrial) in a binary classification approach. In this regard, an ever-growing amount of multi-resolution imagery are being collected, but the current approaches use one single resolution as their input. The use of up/down-sampled images for training has been reported as beneficial for the image classification accuracy both in the computer vision and remote sensing domains. However, it is still unclear if such multi-resolution information can also be captured from images with different spatial resolutions such as imagery of the satellite and airborne (from both manned and unmanned platforms) resolutions. In this paper, three multi-resolution CNN feature fusion approaches are proposed and tested against two baseline (mono-resolution) methods to perform the image classification of building damages. Overall, the results show better accuracy and localization capabilities when fusing multi-resolution feature maps, specifically when these feature maps are merged and consider feature information from the intermediate layers of each of the resolution level networks. Nonetheless, these multi-resolution feature fusion approaches behaved differently considering each level of resolution. In the satellite and aerial (unmanned) cases, the improvements in the accuracy reached 2% while the accuracy improvements for the airborne (manned) case was marginal. The results were further confirmed by testing the approach for geographical transferability, in which the improvements between the baseline and multi-resolution experiments were overall maintained
Original languageEnglish
Article number1636
Pages (from-to)1-26
Number of pages26
JournalRemote sensing
Volume10
Issue number10
DOIs
Publication statusPublished - 14 Oct 2018

Fingerprint

image classification
damage
imagery
remote sensing
computer vision
train
spatial resolution

Keywords

  • ITC-ISI-JOURNAL-ARTICLE
  • ITC-GOLD

Cite this

@article{d58f90e99391481eb0876db4003243b6,
title = "Multi-Resolution Feature Fusion for Image Classification of Building Damages with Convolutional Neural Networks",
abstract = "Remote sensing images have long been preferred to perform building damage assessments. The recently proposed methods to extract damaged regions from remote sensing imagery rely on convolutional neural networks (CNN). The common approach is to train a CNN independently considering each of the different resolution levels (satellite, aerial, and terrestrial) in a binary classification approach. In this regard, an ever-growing amount of multi-resolution imagery are being collected, but the current approaches use one single resolution as their input. The use of up/down-sampled images for training has been reported as beneficial for the image classification accuracy both in the computer vision and remote sensing domains. However, it is still unclear if such multi-resolution information can also be captured from images with different spatial resolutions such as imagery of the satellite and airborne (from both manned and unmanned platforms) resolutions. In this paper, three multi-resolution CNN feature fusion approaches are proposed and tested against two baseline (mono-resolution) methods to perform the image classification of building damages. Overall, the results show better accuracy and localization capabilities when fusing multi-resolution feature maps, specifically when these feature maps are merged and consider feature information from the intermediate layers of each of the resolution level networks. Nonetheless, these multi-resolution feature fusion approaches behaved differently considering each level of resolution. In the satellite and aerial (unmanned) cases, the improvements in the accuracy reached 2{\%} while the accuracy improvements for the airborne (manned) case was marginal. The results were further confirmed by testing the approach for geographical transferability, in which the improvements between the baseline and multi-resolution experiments were overall maintained",
keywords = "ITC-ISI-JOURNAL-ARTICLE, ITC-GOLD",
author = "D. Duarte and F. Nex and N. Kerle and G. Vosselman",
year = "2018",
month = "10",
day = "14",
doi = "10.3390/rs10101636",
language = "English",
volume = "10",
pages = "1--26",
journal = "Remote sensing",
issn = "2072-4292",
publisher = "Multidisciplinary Digital Publishing Institute",
number = "10",

}

Multi-Resolution Feature Fusion for Image Classification of Building Damages with Convolutional Neural Networks. / Duarte, D. (Corresponding Author); Nex, F.; Kerle, N.; Vosselman, G.

In: Remote sensing, Vol. 10, No. 10, 1636, 14.10.2018, p. 1-26.

Research output: Contribution to journalArticleAcademicpeer-review

TY - JOUR

T1 - Multi-Resolution Feature Fusion for Image Classification of Building Damages with Convolutional Neural Networks

AU - Duarte, D.

AU - Nex, F.

AU - Kerle, N.

AU - Vosselman, G.

PY - 2018/10/14

Y1 - 2018/10/14

N2 - Remote sensing images have long been preferred to perform building damage assessments. The recently proposed methods to extract damaged regions from remote sensing imagery rely on convolutional neural networks (CNN). The common approach is to train a CNN independently considering each of the different resolution levels (satellite, aerial, and terrestrial) in a binary classification approach. In this regard, an ever-growing amount of multi-resolution imagery are being collected, but the current approaches use one single resolution as their input. The use of up/down-sampled images for training has been reported as beneficial for the image classification accuracy both in the computer vision and remote sensing domains. However, it is still unclear if such multi-resolution information can also be captured from images with different spatial resolutions such as imagery of the satellite and airborne (from both manned and unmanned platforms) resolutions. In this paper, three multi-resolution CNN feature fusion approaches are proposed and tested against two baseline (mono-resolution) methods to perform the image classification of building damages. Overall, the results show better accuracy and localization capabilities when fusing multi-resolution feature maps, specifically when these feature maps are merged and consider feature information from the intermediate layers of each of the resolution level networks. Nonetheless, these multi-resolution feature fusion approaches behaved differently considering each level of resolution. In the satellite and aerial (unmanned) cases, the improvements in the accuracy reached 2% while the accuracy improvements for the airborne (manned) case was marginal. The results were further confirmed by testing the approach for geographical transferability, in which the improvements between the baseline and multi-resolution experiments were overall maintained

AB - Remote sensing images have long been preferred to perform building damage assessments. The recently proposed methods to extract damaged regions from remote sensing imagery rely on convolutional neural networks (CNN). The common approach is to train a CNN independently considering each of the different resolution levels (satellite, aerial, and terrestrial) in a binary classification approach. In this regard, an ever-growing amount of multi-resolution imagery are being collected, but the current approaches use one single resolution as their input. The use of up/down-sampled images for training has been reported as beneficial for the image classification accuracy both in the computer vision and remote sensing domains. However, it is still unclear if such multi-resolution information can also be captured from images with different spatial resolutions such as imagery of the satellite and airborne (from both manned and unmanned platforms) resolutions. In this paper, three multi-resolution CNN feature fusion approaches are proposed and tested against two baseline (mono-resolution) methods to perform the image classification of building damages. Overall, the results show better accuracy and localization capabilities when fusing multi-resolution feature maps, specifically when these feature maps are merged and consider feature information from the intermediate layers of each of the resolution level networks. Nonetheless, these multi-resolution feature fusion approaches behaved differently considering each level of resolution. In the satellite and aerial (unmanned) cases, the improvements in the accuracy reached 2% while the accuracy improvements for the airborne (manned) case was marginal. The results were further confirmed by testing the approach for geographical transferability, in which the improvements between the baseline and multi-resolution experiments were overall maintained

KW - ITC-ISI-JOURNAL-ARTICLE

KW - ITC-GOLD

UR - https://ezproxy2.utwente.nl/login?url=https://webapps.itc.utwente.nl/library/2018/isi/duarte_mul.pdf

U2 - 10.3390/rs10101636

DO - 10.3390/rs10101636

M3 - Article

VL - 10

SP - 1

EP - 26

JO - Remote sensing

JF - Remote sensing

SN - 2072-4292

IS - 10

M1 - 1636

ER -