End-to-end process orchestration of Earth Observation data workflows with apache airflow on high performance computing

Liang Tian; Rocco Sedona; Amirpasha Mozaffari; Enxhi Kreshpa; C. Paris; Morris Riedel; Martin G. Schultz; Gabriele Cavallaro

doi:10.1109/IGARSS52108.2023.10283416

End-to-end process orchestration of Earth Observation data workflows with apache airflow on high performance computing

Liang Tian
, Rocco Sedona
, Amirpasha Mozaffari
, Enxhi Kreshpa
, C. Paris
, Morris Riedel
, Martin G. Schultz
, Gabriele Cavallaro

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

7 Citations (Scopus)

311 Downloads (Pure)

Abstract

Earth Observation (EO) data processing faces challenges due to large volumes, multiple sources, and diverse formats. To address this issue, this paper presents a scalable and parallelizable workflow using Apache Airflow, capable of integrating Machine Learning (ML) and Deep Learning (DL) models with Modular Supercomputing Architecture (MSA) systems. To test the workflow, we considered the production of large-scale Land-Cover (LC) maps as a case study. The workflow manager, Airflow, offers scalability, extensibility, and programmable task definition in Python. It allows us to execute different steps of the workflow in different High-Performance Computing (HPC) systems. The workflow is demonstrated on the Dynamical Exascale Entry Platform (DEEP) and Jülich Research on Exascale Cluster Architectures (JURECA) hosted at the Jülich Supercomputing Centre (JSC), a platform that incorporates heterogeneous JSC systems.

Original language	English
Title of host publication	IGARSS 2023
Subtitle of host publication	2023 IEEE International Geoscience and Remote Sensing Symposium
Publisher	IEEE
Pages	711-714
Number of pages	4
ISBN (Print)	979-8-3503-3174-5
DOIs	https://doi.org/10.1109/IGARSS52108.2023.10283416
Publication status	Published - 20 Oct 2023
Event	43rd IEEE International Geoscience and Remote Sensing Symposium, IGARSS 2023 - Pasadena Convention Center, Pasadena, United States Duration: 16 Jul 2023 → 21 Jul 2023 Conference number: 43 https://2023.ieeeigarss.org/index.php

Conference

Conference	43rd IEEE International Geoscience and Remote Sensing Symposium, IGARSS 2023
Abbreviated title	IGARSS 2023
Country/Territory	United States
City	Pasadena
Period	16/07/23 → 21/07/23
Internet address	https://2023.ieeeigarss.org/index.php

Keywords

Earth
High performance computing
Computational modeling
Scalability
Computer architecture
Transformers
Extensibility
2023 OA procedure

Access to Document

10.1109/IGARSS52108.2023.10283416

End-to-End_Process_Orchestration_of_Earth_Observation_Data_Workflows_with_Apache_Airflow_on_High_Performance_ComputingFinal published version, 978 KBLicence: Taverne

Cite this

Tian, L., Sedona, R., Mozaffari, A., Kreshpa, E., Paris, C., Riedel, M., Schultz, M. G., & Cavallaro, G. (2023). End-to-end process orchestration of Earth Observation data workflows with apache airflow on high performance computing. In IGARSS 2023: 2023 IEEE International Geoscience and Remote Sensing Symposium (pp. 711-714). Article 10283416 IEEE. https://doi.org/10.1109/IGARSS52108.2023.10283416

@inproceedings{646689df9dcf49bfa639e77ef133d2d7,

title = "End-to-end process orchestration of Earth Observation data workflows with apache airflow on high performance computing",

abstract = "Earth Observation (EO) data processing faces challenges due to large volumes, multiple sources, and diverse formats. To address this issue, this paper presents a scalable and parallelizable workflow using Apache Airflow, capable of integrating Machine Learning (ML) and Deep Learning (DL) models with Modular Supercomputing Architecture (MSA) systems. To test the workflow, we considered the production of large-scale Land-Cover (LC) maps as a case study. The workflow manager, Airflow, offers scalability, extensibility, and programmable task definition in Python. It allows us to execute different steps of the workflow in different High-Performance Computing (HPC) systems. The workflow is demonstrated on the Dynamical Exascale Entry Platform (DEEP) and J{\"u}lich Research on Exascale Cluster Architectures (JURECA) hosted at the J{\"u}lich Supercomputing Centre (JSC), a platform that incorporates heterogeneous JSC systems.",

keywords = "Earth, High performance computing, Computational modeling, Scalability, Computer architecture, Transformers, Extensibility, 2023 OA procedure",

author = "Liang Tian and Rocco Sedona and Amirpasha Mozaffari and Enxhi Kreshpa and C. Paris and Morris Riedel and Schultz, \{Martin G.\} and Gabriele Cavallaro",

year = "2023",

month = oct,

day = "20",

doi = "10.1109/IGARSS52108.2023.10283416",

language = "English",

isbn = "979-8-3503-3174-5",

pages = "711--714",

booktitle = "IGARSS 2023",

publisher = "IEEE",

address = "United States",

note = "43rd IEEE International Geoscience and Remote Sensing Symposium, IGARSS 2023, IGARSS 2023 ; Conference date: 16-07-2023 Through 21-07-2023",

url = "https://2023.ieeeigarss.org/index.php",

}

Tian, L, Sedona, R, Mozaffari, A, Kreshpa, E, Paris, C, Riedel, M, Schultz, MG & Cavallaro, G 2023, End-to-end process orchestration of Earth Observation data workflows with apache airflow on high performance computing. in IGARSS 2023: 2023 IEEE International Geoscience and Remote Sensing Symposium., 10283416, IEEE, pp. 711-714, 43rd IEEE International Geoscience and Remote Sensing Symposium, IGARSS 2023, Pasadena, California, United States, 16/07/23. https://doi.org/10.1109/IGARSS52108.2023.10283416

End-to-end process orchestration of Earth Observation data workflows with apache airflow on high performance computing. / Tian, Liang; Sedona, Rocco; Mozaffari, Amirpasha et al.
IGARSS 2023: 2023 IEEE International Geoscience and Remote Sensing Symposium. IEEE, 2023. p. 711-714 10283416.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

TY - GEN

T1 - End-to-end process orchestration of Earth Observation data workflows with apache airflow on high performance computing

AU - Tian, Liang

AU - Sedona, Rocco

AU - Mozaffari, Amirpasha

AU - Kreshpa, Enxhi

AU - Paris, C.

AU - Riedel, Morris

AU - Schultz, Martin G.

AU - Cavallaro, Gabriele

N1 - Conference code: 43

PY - 2023/10/20

Y1 - 2023/10/20

N2 - Earth Observation (EO) data processing faces challenges due to large volumes, multiple sources, and diverse formats. To address this issue, this paper presents a scalable and parallelizable workflow using Apache Airflow, capable of integrating Machine Learning (ML) and Deep Learning (DL) models with Modular Supercomputing Architecture (MSA) systems. To test the workflow, we considered the production of large-scale Land-Cover (LC) maps as a case study. The workflow manager, Airflow, offers scalability, extensibility, and programmable task definition in Python. It allows us to execute different steps of the workflow in different High-Performance Computing (HPC) systems. The workflow is demonstrated on the Dynamical Exascale Entry Platform (DEEP) and Jülich Research on Exascale Cluster Architectures (JURECA) hosted at the Jülich Supercomputing Centre (JSC), a platform that incorporates heterogeneous JSC systems.

AB - Earth Observation (EO) data processing faces challenges due to large volumes, multiple sources, and diverse formats. To address this issue, this paper presents a scalable and parallelizable workflow using Apache Airflow, capable of integrating Machine Learning (ML) and Deep Learning (DL) models with Modular Supercomputing Architecture (MSA) systems. To test the workflow, we considered the production of large-scale Land-Cover (LC) maps as a case study. The workflow manager, Airflow, offers scalability, extensibility, and programmable task definition in Python. It allows us to execute different steps of the workflow in different High-Performance Computing (HPC) systems. The workflow is demonstrated on the Dynamical Exascale Entry Platform (DEEP) and Jülich Research on Exascale Cluster Architectures (JURECA) hosted at the Jülich Supercomputing Centre (JSC), a platform that incorporates heterogeneous JSC systems.

KW - Earth

KW - High performance computing

KW - Computational modeling

KW - Scalability

KW - Computer architecture

KW - Transformers

KW - Extensibility

KW - 2023 OA procedure

U2 - 10.1109/IGARSS52108.2023.10283416

DO - 10.1109/IGARSS52108.2023.10283416

M3 - Conference contribution

SN - 979-8-3503-3174-5

SP - 711

EP - 714

BT - IGARSS 2023

PB - IEEE

T2 - 43rd IEEE International Geoscience and Remote Sensing Symposium, IGARSS 2023

Y2 - 16 July 2023 through 21 July 2023

ER -

End-to-end process orchestration of Earth Observation data workflows with apache airflow on high performance computing

Abstract

Conference

Keywords

Access to Document

Fingerprint

Cite this