The existing methods of building change detection (CD) using remote sensing (RS) images are still deficient in handling scale variation and class imbalance problems, indicating a decrease in the robustness of small-object detection and pseudo-change information. Thus, a novel building CD framework called the multiscale context aggregation network (MSCANet) is proposed. The high-resolution network is integrated into the feature extracting stage to maintain high-resolution representations throughout the whole process. Then, multiscale context information is aggregated using a scale-aware feature pyramid module (FPM). Recognition performance can be improved from discriminant feature representation learning by using a channel–spatial attention module. Furthermore, a class-balanced loss is proposed to reduce the impact of class imbalance in long-tail datasets. Experimental results from using the LEVIR-CD and SZTAKI AirChange benchmark datasets prove the superiority of the MSCANet over the other baseline methods, with improved maximum F1 scores of 5.28 and 8.47, respectively.