Abstract
The continuous evolution of cloud computing, together with the growth of cloud-based tools, gives us new opportunities to improve data management, application delivery, and automated processing. However, there is still a broad range of challenges in integrating these systems to achieve seamless interoperability for certain applications.
Kubernetes, a platform for orchestrating container-based applications, has become the de facto standard in the cloud industry. At the same time, event-driven computing and data lake architectures are fundamental in providing access to large datasets to be processed.
Building on these advancements, our work combines these technologies through two tools: OSCAR, an open-source platform enabling serverless event-driven computing for efficient and scalable applications, and DCNiOS (Data Connector through NiFi for OSCAR), which facilitates efficient data management and transfer through data processing flows.
In this work, we present the versatility of OSCAR and DCNiOS through use case integrations from the interTwin project:
Specializing in flood hazard and impact modelling, Deltares employs OSCAR to offload computationally demanding tasks. By leveraging OSCAR’s ability to chain multiple processes defined through Common Workflow Language, their workflows are executed automatically and efficiently on remote OSCAR clusters. In turn, the integration of OSCAR with interLink allows to further offload the execution on HPC clusters, as exemplified with the VEGA supercomputer.
Focused on drought forecasting, EURAC uses OSCAR for computation. Integration with OpenEO further enhances their workflows by enabling automated calls to OSCAR, streamlining their modelling processes.
OSCAR supports Rucio, an advanced data lake designed for scientific data management developed by CERN, as a data source and destination, with plans to enable event-driven computing based on the Rucio notification system for optimised workflows execution.
These integrations showcase OSCAR’s flexibility in addressing computational challenges, enabling efficient, automated workflows that leverage cloud-based resources.
This work was supported by the project “An interdisciplinary Digital Twin Engine for science’’ (interTwin) that has received funding from the European Union’s Horizon Europe Programme under Grant 101058386. GM would like to thank Grant PID2020-113126RB-I00 funded by MCIU/AEI/10.13039/501100011033.