Skip to main content

Develop mode architecture

The parts of FlowDeploy develop mode, and how they fit together.

Components​

There are four primary FlowDeploy Develop components:

  1. A workflow manager running on a macOS or Linux machine
  2. A shared file system
  3. The FlowDeploy API
  4. Execution environments

Workflow managers​

A workflow manager is a pipelining tool geared towards running bioinformatics pipelines. The three most commonly used pipeline languages are Nextflow, Snakemake, and WDL.

The workflow manager sets:

  1. The order of execution of tasks (usually a DAG)
  2. Code to be executed per task
  3. Input and output file locations
  4. Docker containers for each task

Workflow managers can connect directly to workload schedulers like SLURM or AWS Batch, but many facilities recommend against it. See the NERSC pages on Nextflow and Snakemake.

Check your logs!

Check what your local workflow manager is logging. Sometimes, they include environment variables and keys.

Shared file system​

See the shared file system guide for more.

FlowDeploy API​

Workflow managers communicate with the FlowDeploy API over HTTPS using the Task Execution Service (TES) standard from the Global Alliance for Genomics and Health (GA4GH) non-profit. Nextflow, Snakemake, and WDL (via Cromwell) have a built-in TES integration.

Execution environments​

By default, FlowDeploy executes in our managed AWS environment. You can set specific steps to run on your machine, or disable the integration completely to run your pipelines locally.

FlowDeploy can execute in other cloud environments. Contact us to learn more.