DECICE Framework Expands with Unified Workflow Support
The DECICE framework has taken a significant step forward in workflow flexibility by introducing support for Snakemake and Argo Workflows in addition to native Kubernetes job definitions.
Previously, users could submit their computational jobs by providing a Kubernetes job definition YAML file along with any required input files. With our latest enhancement, users can now upload Snakemake files or Argo Workflow definitions, enabling seamless integration of workflows created in these popular workflow management systems.
This advancement brings workflow unification to DECICE. No matter which workflow format users choose, our internal parsers automatically convert the submitted files into the DECICE Workflow representation. This ensures that:
- Steps, dependencies, and execution order are preserved across formats
- Workflows can be executed efficiently and consistently on a cluster
- Users have more freedom to design and manage workflows in the tool of their choice without sacrificing compatibility
By supporting multiple workflow formats, DECICE enables researchers and engineers to run complex data processing pipelines and scientific workflows seamlessly across the entire compute continuum — from edge devices, through cloud platforms, and all the way to the largest high-performance computing (HPC) systems. This is achieved without the need for any manual file conversion or reconfiguration.
With this flexibility, users can create workflows in their preferred tools and deploy them anywhere within a federated, heterogeneous infrastructure. This capability not only saves valuable time but also fosters cross-platform interoperability, making it easier for research groups and organizations to collaborate and share workflows, regardless of how or where they were originally created.
Author(s): Mirac Aydin, Gesellschaft für Wissenschaftliche Datenverarbeitung MBH Göttingen
Key words: #DECICE #WorkflowIntegration #Snakemake #ArgoWorkflows #CrossPlatformInteroperability
