Building usage classification using a transformer-based multimodal deep learning method

Building usage classification is of great significance for urban planning and city digital twinning applications. So far, however, the problem of mixed building use has not been addressed, and detailed categories cannot be assigned to individual buildings. This paper employs a state-of-the-art Transformer-based multimodal deep learning method to extract and fuse image features from satellite images with textual features of point-of-interest (POI) data. The derived features along with the relationship between the two types of data are utilized for the classification task. A custom dataset prepared for the city of Wuhan, China, with eight land-use categories has been classified yielding a microf1-score of 80.7%. Results show that the proposed method can effectively improve the classification results, achieving 5.6% higher accuracy as compared to the results based upon a single data source.