In many cities of the Global South, informal and deprived neighborhoods, also commonly called slums, continue to proliferate, but their locations and dwellers' socio-economic status are often invisible in official statistics and maps. Very high resolution (VHR) satellite images coupled with deep learning allow us to efficiently map these areas and study their socio-economic and spatio-temporal variability to support interventions. This paper investigates a deep transfer learning approach based on convolutional neural networks (CNN) to identify the socio-economic variability of poor neighborhoods in Bangalore, India. Our deep network, pre-trained on a slum classification data set, is tuned towards the prediction of a continuous-valued socio-economic index capturing multiple levels of deprivation. Experimental results show that the CNN-based regression model can explain the socio-economic variability with an R2 of 0.75. The use of additional publicly available geographic information layers allow us to spatially extend the analysis beyond the surveyed deprived area data samples to uncover city-wide patterns of socio-economic inequalities.