Benchmaking data access and processing performance of OpenEO backends: a reproducible approach

openEO is a community-driven initiative supported by prominent research institutions, as well as cloud service providers and international agencies, that is designed to simplify the access and processing of Earth Observation (EO) data. It offers a standardized interface that allows users to interact with multiple EO data providers, cloud services, and processing engines, facilitating the efficient retrieval, analysis, and visualization of EO data. This facilitates scalable and flexible processing workflows for a broad spectrum of applications. To achieve this, openEO defines a HTTP-based API that enables communication between cloud backends with large EO datasets and frontend applications in an interoperable manner by specifying how to manage users, discover available EO data and processes on backends, execute processes and user-defined functions on backends, and download results. Currently there are different backends available that are based on various big EO data processing frameworks, such as Geotrellis, GRASS GIS/actinia, Google Earth Engine, OpenDataCube, Xarray, and Dask. There are also operational platforms that provide openEO services (i.e. openEO providers) such as Copernicus Data Space Ecosystem, EODC, and VITO. Recently, the openEO Platform also has been launched, providing federated access to various openEO services.

Although openEO provides a standard mechanism to access the provided EO data and processing services, available data and processing capabilities differ between the service providers. Data availability is determined by the size of the data archives of the providers, whereas processing capabilities are largely influenced by the specific backend implementations deployed by the operators and their IT infrastructure. Although openEO provides a web portal (openEO Hub), which lists the reported capabilities of the openEO providers, limited information is available on the actual data availability and data access and processing performance of the providers. Benchmarking is crucial for assessing the performance of various openEO backends, because it provides objective and standardized measurements of efficiency, performance, and reliability. By comparing different backends under controlled conditions, it provides valuable information that enables users to select the most suitable backend for their specific needs, based on factors such as processing time and scalability. It can also help to identify strengths and weaknesses, driving improvements and optimizations by backend developers that benefit the entire user community. Eventually, it can support the development of more effective systems, and helps maintain high standards in EO data access and processing workflows.

To benchmark the performance of different openEO backends and service providers, we have created an openEO benchmarking framework. Built using a unit testing approach, the framework allows for replicable testing of data availability, data access, and data processing capabilities across various backends. The results are presented in standard formats, providing both detailed reports and summaries. This poster presentation provides a comprehensive overview of the design principles and operational structure of the developed openEO benchmarking framework. It covers the framework's core features, functional capabilities, along with a comprehensive overview of the tests that are designed to assess data availability on different backends and their data access and processing performance. The results of the tests performed are summarized to provide an insight into the current state of the operational openEO providers and their services.