Skip to main content

Shared file systems

More about the shared, distributed file systems we like and use.

Low-to-medium scale

We use an S3-based POSIX-compliant distributed file system that's mounted via FUSE. It's great for low-to-medium scale pipelines, because it:

  • has a low initial cost.
  • scales like an S3 bucket.
  • bioinformaticians can mount it to their computer and use it like a hard drive.

Transfer speeds for large files is similar to S3 – ~250 MB/s. Caching is configured to speed up usage without requiring an S3 call for every file operation.

High scale

For higher scale pipelines, Lustre is preferred (FSx for Lustre on AWS). It's higher cost and harder to manage, but it incurs less latency for file I/O, doesn't use S3 bandwidth, and has a much higher maximum bandwidth.

How it's used

The shared file system is mounted on the machine running the workflow manager, as well as all cloud-based worker nodes. This means the workflow manager running the S3-based file system can be located anywhere, while a shared Lustre file system requires the workflow manager to run in the same private network.


Initial Sync Time

The first time a machine uses the S3-backed file system, it may take a minute to synchronize with the shared file system.

Apple Silicon M1 and M2 Macs

Mounting any FUSE-based shared file system requires a system change in startup mode. This applies to all S3-based file systems.

S3 File Handling

When using S3 files directly with a workflow manager in FlowDeploy Develop mode, they are downloaded through the machine running the workflow manager.