Spatiotemporal data partitioning for distributed random forest algorithm: Air quality prediction using imbalanced big spatiotemporal data on spark distributed framework

Marjan Asgari*, Wanhong Yang, M. Farnaghi

*Corresponding author for this work

Research output: Contribution to journalArticleAcademicpeer-review

11 Citations (Scopus)
87 Downloads (Pure)

Abstract

Spatiotemporal air quality datasets are typically collected hourly in monitoring stations deployed non-uniformly across a metropolitan city. These datasets are not only big, which poses challenges on the storage and processing capacity of centralized computing systems but also imbalanced and spatially heterogeneous, which may result in biased air quality prediction. To address these challenges, we designed and developed a parallel air quality prediction system equipped with a spatiotemporal data partitioning method, a distributed machine learning algorithm, Hadoop's distributed data storage platform and its resource scheduler/manager, and Spark's efficient and in-memory execution environment, which is suitable for running iterative algorithms, e.g., machine learning. Our proposed spatiotemporal partitioning method accounted for imbalance and spatial heterogeneity features of big air quality data in predictive models, which comply with the load-balancing requirement of distributed computing systems. Distributed Random Forest algorithm in the H2O library of the Spark framework was selected as the distributed machine learning algorithm to develop the air quality predictive model. This algorithm is an ensemble forest with algorithm-level adjustments to perform as efficiently as possible for big imbalanced datasets. An application of the parallel quality prediction system for Tehran, Iran showed that the parallel prediction system had considerable speedup gain and improved both the overall accuracy and class precision of air quality prediction when working with imbalanced big spatiotemporal air quality datasets. A future research direction is to add data streaming and visualization functions to the system to provide rapid and reliable air quality prediction for supporting environmental health management.

Original languageEnglish
Article number102776
JournalEnvironmental Technology and Innovation
Volume27
Early online date27 Jun 2022
DOIs
Publication statusPublished - Aug 2022

Keywords

  • Air quality prediction
  • Big spatiotemporal data
  • Distributed random forest algorithm
  • Distributed systems
  • Imbalanced data
  • ITC-ISI-JOURNAL-ARTICLE
  • ITC-GOLD

Fingerprint

Dive into the research topics of 'Spatiotemporal data partitioning for distributed random forest algorithm: Air quality prediction using imbalanced big spatiotemporal data on spark distributed framework'. Together they form a unique fingerprint.

Cite this