Workflow Orchestration

For the workflow orchestration we provide two frameworks: Flyte and StreamFlow. Please visit the Machine Learning Workflow Orchestration Seminar for more information.

Flyte

Flyte is a highly scalable cloud-native workflow orchestration platform on top of containers and Kubernetes.

_images/flyte-architecture.svg

StreamFlow

StreamFlow executes Common Workflow Language (CWL) using a deployment model that includes containers, Slurm (HPC), and Kubernetes.

_images/streamflow-model.png

Source: StreamFlow