Cartography has been, in its pre-modern and modern production of maps, influential in determining how space and territory is experienced and defined. But advancements in telecommunications and geovisualization software, along with geoinformation systems and geoinformation science (GIS), have transformed cartographic practice from a tool of dominantly state apparatus to a scientific, commercial, and humanitarian enterprise. This is exemplified in the use of remote sensing (RS) techniques to acquire, process, and visualize images of the Earth. In the last decade, RS techniques have increasingly incorporated Artificial Intelligence (e.g., Convolutional Neural Networks) to improve the speed and accuracy of feature extraction and classification in remotely sensed images. This paper will investigate the use of CNNs in the classification of deprived urban areas referred to as “slums” and “informal settlements” in the Global South. Using a postphenomenological methodology, this paper shall analyze the role of classification and use of geoinformation in shaping how deprived urban areas are algorithmically classified. This analysis will reveal that besides the technical opportunities and challenges, attention needs to be given to three ethical areas of concern: how deprived area mapping using AI impacts the agency of communities, how there is a potential lack in the democratization of these RS technologies, and how the privacy and data protection of communities being mapped is endangered.