Abstract
Thanks to the success of the Copernicus program and the general awareness towards satellite Earth Observation (EO) data, a growing number of cloud-based EO services are now offered on the European and global market for working on and with the available EO data. From the user perspective this is currently creating confusion due to the large number of available services and the lack of comparability between the offers and an inherent risk of vendor lock-in, when selecting offers based on propriety and/or closed source solutions.
Alongside other issues like the growing size of data to handle and computational requirements have led to the development of the openEO API (see https://openeo.org ) since 2017, which already greatly reduces the risk of vendor lock-in, when a sufficient number of back-ends is available. This project was successfully concluded at the end of 2020 and has provided the first version of openEO API, which has been implemented in a growing number of cloud back-ends and three different client libraries, supporting R, Python, and JavaScript users.
Under the umbrella of ESA in form of openEO Platform this work is being continued, and the concepts are further evolved, by introducing new aspects of federating different cloud back-ends that go much further than just offering the same interfaces from different back-ends. openEO Platform has the goal to provide openEO as a service to EO data users, where they can easily access all kinds of data and processing, share results, and potentially offer their own value-added services on top of the platform (see https://openeo.cloud ).
Newly added features include a single sign-on solution, data and process harmonization, integration of commercially offered datasets, shared accounting and billing procedures between the integrated back-ends, marketplace offerings of user generated applications and workflows. Driven by a number of challenging use cases, new processing capabilities are also introduced and defined in openEO, including generation of analysis ready data (on-demand and on-the-fly) following CARD4L recommendations, machine learning, regression modelling, sampling and improved time series modelling.
The federation on top of which openEO Platform is built includes existing and new features, allowing for a much more seamless user experience than previously possible. A comprehensive library of standardized, well-documented processes has been defined and a set of core processes to be supported by each federation member is currently in development. Alignments in the implementation and availability of those pre-defined processes are key for true interoperability in the federation. The same goes for the numerous data collections that are offered in openEO platform from all the currently participating back-ends such as Terrascope, the Earth Observation Data Centre (EODC) and Sentinel Hub via the Euro Data Cube. All data providers adopted metadata defined by the SpatioTemporal Asset Catalog (STAC) and moreover naming conventions of the elements defining this metadata such as collection names and band names are harmonized in the federation. Shared user identity management allows for a single point of entry for users, implemented through EGI-Check-in, which also allows for further integration with the European Open Science Cloud (EOSC).
All newly implemented core components of the federation enable now the development of distributed processing of workflows. Federated back-ends providing required data and processes will then be able to collectively work on larger jobs or complement each other in case of missing data or processing capabilities.