Multimodal satellite-field data integration for scalable crop type mapping with limited labels

Crop-type mapping is crucial for improving crop yield predictions, informing agricultural policy decisions, and addressing the impacts of climate change. Despite achieving state-of-the-art performance, current transformer-based architectures that use Satellite image time series (SITS) require costly labeled data and struggle to generalize to new regions. To address this limitation, this article presents a method that leverages multimodal information from field images and satellite data to address the scarcity of reference data in crop-type mapping. The proposed approach consists of three main steps: 1) multimodal data preparation; 2) multimodal pseudo-labeling; and 3) SITS-based crop-type mapping. The first step processes both data sources for facilitating their integration, while the novelty lies in the second step, where additional pseudo-labels are generated by classifying crop parcels with both SITS and field image availability, leveraging multimodal information to produce reliable label predictions. Finally, the third step produces a crop-type map for the entire study area using the SITS and the enriched training set. Experimental results at European level, using the 2018 land use and land cover survey (LUCAS) field images paired with Sentinel-2 SITS, demonstrate the effectiveness of integrating satellite top-view data with ground-level field images. The pseudo-labeled data achieves performance comparable with models trained on the complete labeled dataset, while substantially reducing the need for extensive labeled data collection.