Expanse Documentation

Expanse is an HPC workflow orchestration platform. Define your computational pipeline once, run it anywhere: on your laptop, a Slurm cluster, or across multiple HPC systems.

What You Get

• CLI tool: initialise projects, submit jobs, stream logs, and manage clusters from your terminal
• Automatic data transfer: Expanse takes care of moving data between nodes, whether they run on the same cluster or across different systems, with or without Globus endpoints. No manual file management required.
• Run nodes or full pipelines: run a single node for quick tasks, or chain nodes into multi-stage workflows with automatic dependency resolution
• Multi-scheduler support: supports Slurm, Ray, Kubernetes, and bare SSH out of the box
• Node registry: publish, discover, and reuse computational nodes across teams and projects
• Intelligent caching: Expanse remembers previous builds and results, so unchanged steps are never re-executed
• Any language: run Python scripts, compiled Fortran/C binaries, or commercial solvers. Data flows between languages seamlessly.

How It Works

1. Define nodes: each node is a computational step with typed inputs, outputs, and resource requirements
2. Run individually or as workflows: run a single node with expanse run my-node, or wire nodes into a workflow YAML for multi-stage pipelines
3. Run anywhere: target any configured cluster per node. Expanse handles data transfer, module loading, and execution automatically.

Quick Links