Rapid urbanization and increasing population in cities with a large portion of them settled in deprived neighborhoods, mostly defined as slum areas, have escalated inequality and vulnerability to natural disasters. As a result, monitoring such areas is essential to provide information and support decision-makers and urban planners, especially in case of disaster recovery. Here, we developed an approach to monitor the urban deprived areas over a four-year period after super Typhoon Haiyan, which struck Tacloban city, in the Philippines, in 2013, using high-resolution satellite images and machine learning methods. A Support Vector Machine classification method supported by a local binary patterns feature extraction model was initially performed to detect slum areas in the pre-disaster, just after/event, and post-disaster images. Afterward, a dense conditional random fields model was employed to produce the final slum areas maps. The developed method detected slum areas with accuracies over 83%. We produced the damage and recovery maps based on change analysis over the detected slum areas. The results revealed that most of the slum areas were reconstructed 4 years after Typhoon Haiyan, and thus, the city returned to the pre-existing vulnerability level.