hpc-compose

Compose-style multi-service workflows, compiled into one inspectable Slurm job.

One allocation · one script · Slurm-native runtime.

hpc-compose gives research and HPC teams a small YAML authoring model for services, startup order, readiness checks, runtime backends, logs, artifacts, and follow-up commands.

services:
  app:
    image: python:3.12-slim
    command: python train.py

$ hpc-compose plan --show-script -f compose.yaml
spec is valid
service order: app
#SBATCH --job-name=my-app

Use hpc-compose when you want Docker Compose-style authoring on Slurm without adding Kubernetes, a long-running control plane, or custom cluster-side services.

Start with the Support Matrix before planning a real runtime workflow. Linux is the maintained runtime target; macOS is intended for authoring, validation, rendering, and inspection.

Safe First Path

These commands are safe from a laptop, workstation, or login node because new writes a local starter spec and plan is purely static. It does not call sbatch, import images, or write a script file:

hpc-compose new --template minimal-batch --name my-app --output compose.yaml
hpc-compose plan -f compose.yaml
hpc-compose plan --show-script -f compose.yaml

plan validates the spec and resolves service order; plan --show-script adds the rendered batch script. Expected output includes:

spec is valid
service order: app
Rendered script:
#SBATCH --job-name=my-app

For real cluster runs, configure a cache path visible from both the Slurm submission host and compute nodes, either in x-slurm.cache_dir, hpc-compose setup --cache-dir, or [defaults.cache] / [profiles.<name>.cache] settings. From a source checkout, you can also inspect the checked-in examples with hpc-compose plan -f examples/minimal-batch.yaml.

Run hpc-compose up -f compose.yaml only on a supported Linux Slurm submission host with the runtime backend your spec selects. If it fails, start with hpc-compose debug -f compose.yaml --preflight.

If you have a source checkout and want to exercise real sbatch without a cluster login, use the Local Slurm Dev Cluster as a host-backend smoke test.

Download the asciinema-style quickstart demo cast if you want the same flow as a terminal recording.

Terms To Know

See the Glossary for the full set of terms.

What It Is For

• model serving plus helper services inside one Slurm allocation
• data and ETL pipelines with startup ordering or stage-completion dependencies
• training jobs with checkpoint export, artifact tracking, and resume-aware reruns
• explicit multi-node launch patterns that still fit inside one allocation

What It Is Not

hpc-compose is not a full Docker Compose runtime and is not a general cluster orchestrator.

Unsupported Compose features include:

• build:
• ports
• networks / network_mode
• Compose restart as a Docker key
• deploy
• dynamic node bin packing

For exact boundaries, read Execution Model, Slurm Capability Scope, and Spec Reference.

Read Next

1. Why hpc-compose for the problem it solves.
2. Quickstart for the shortest safe path.
3. Examples to choose a starting spec.
4. Runtime Backends before changing runtime.backend.
5. Runbook when adapting a real workload on a cluster.
6. Troubleshooting when the first cluster run fails.

Reference

• Installation
• Task Guide
• CLI Reference
• Spec Reference
• Roadmap and Non-Goals

Support Matrix

This page separates what hpc-compose can build, what CI currently exercises, and what is officially supported for real workflows.

Support levels

Officially supported

CI-tested

Current CI validates full runtime-facing behavior on Ubuntu and authoring/distribution behavior on macOS. Other published builds should be treated as lower-confidence until corresponding CI coverage exists.

Release-built

Windows status

Windows archives are published so users can inspect the CLI surface or experiment with non-runtime commands, but Windows is currently release-built only:

• Slurm plus HPC runtime workflows are not an officially supported Windows target.
• Issues that are specific to Windows runtime execution may be closed as out of scope until the support policy changes.

Cluster assumptions for full support

For full runtime support on Linux, the target environment should provide:

• sbatch, srun, and related Slurm client tools on the submission host
• one supported runtime path:
  • Pyxis container support in srun plus Enroot on the submission host,
  • Apptainer on the submission host and compute nodes,
  • Singularity on the submission host and compute nodes,
  • or module/vendor software available on the host runtime path
• shared storage for the resolved cache directory

Use Runtime Backends, Runbook, and Execution Model before adapting a real workload to a cluster.

Related Docs

• Installation
• Quickstart
• Why hpc-compose
• Runtime Backends
• Runbook

Installation

For normal use, install from a published GitHub Release. Build from source when you are developing the project or need to inspect a local checkout before using it on a cluster.

Install From A Published Release

Latest release (zero edits)

The quickest path needs no version substitution. When HPC_COMPOSE_VERSION is unset, the installer resolves the latest published GitHub Release tag automatically and downloads the matching asset:

curl -fsSL https://raw.githubusercontent.com/NicolasSchuler/hpc-compose/main/install.sh | sh

This runs the moving install.sh from main, but it always installs from a published releases/download/<tag>/... asset (never unreleased main).

Pinned release (reproducible / recommended for clusters)

For reproducible installs across a shared cluster, pin the exact release tag from the GitHub Releases page so every machine lands on the same build:

RELEASE_TAG=vX.Y.Z
curl -fsSL "https://raw.githubusercontent.com/NicolasSchuler/hpc-compose/${RELEASE_TAG}/install.sh" \
  | env HPC_COMPOSE_VERSION="${RELEASE_TAG}" sh

The installer downloads the matching archive for the current Linux or macOS machine, verifies the published .sha256 sidecar, installs hpc-compose into ~/.local/bin by default, and installs shipped Unix manpages when present.

After installation, make sure the install directory is on your shell PATH and verify the binary:

export PATH="$HOME/.local/bin:$PATH"
command -v hpc-compose
hpc-compose --version

Useful overrides:

RELEASE_TAG=vX.Y.Z

curl -fsSL "https://raw.githubusercontent.com/NicolasSchuler/hpc-compose/${RELEASE_TAG}/install.sh" \
  | env HPC_COMPOSE_INSTALL_DIR=/usr/local/bin HPC_COMPOSE_VERSION="$RELEASE_TAG" sh

Installer availability does not imply full runtime support. Check the Support Matrix before assuming a platform can run submission, prepare, or watch workflows end to end.

About The main Installer Script

Fetching install.sh from main without HPC_COMPOSE_VERSION does not install unreleased main:

curl -fsSL https://raw.githubusercontent.com/NicolasSchuler/hpc-compose/main/install.sh | sh

That command runs the moving script from main, but the script resolves the latest published GitHub Release and downloads from releases/download/<tag>/.... Use the version-pinned command above for reproducible installs. Use a source checkout when you want unreleased code.

Manual Release Download

Prebuilt archives are published on the release page. Pick the archive that matches your platform.

Example for Linux x86_64:

RELEASE_TAG=vX.Y.Z

curl -L "https://github.com/NicolasSchuler/hpc-compose/releases/download/${RELEASE_TAG}/hpc-compose-${RELEASE_TAG}-x86_64-unknown-linux-musl.tar.gz" -o hpc-compose.tar.gz
tar -xzf hpc-compose.tar.gz
./hpc-compose --help

Linux x86_64 releases use a musl target to avoid common cluster glibc mismatches. Unix release archives also contain share/man/man1/. See CLI Reference for browsing the installed man pages.

Windows release archives are zip-only for inspection and checksum parity. The installer script and end-to-end Slurm runtime workflows target Unix-like systems; use Windows primarily through WSL or a remote Linux/macOS authoring environment.

Native Packages

Published Linux releases may include .deb and .rpm assets:

RELEASE_TAG=vX.Y.Z

sudo apt install "./hpc-compose-${RELEASE_TAG}-x86_64-unknown-linux-musl.deb"
sudo dnf install "./hpc-compose-${RELEASE_TAG}-x86_64-unknown-linux-musl.rpm"

Package availability does not change runtime support policy. Linux cluster workflows still need Slurm client tools, the selected runtime backend, and shared storage for the resolved cache directory.

Homebrew On macOS

The repository exposes a same-repo Homebrew tap:

brew install NicolasSchuler/hpc-compose/hpc-compose

The formula is refreshed by release automation when a Homebrew-published release is cut. Check brew info NicolasSchuler/hpc-compose/hpc-compose when you need to confirm the formula version before installing.

macOS support is for authoring and local non-runtime commands such as new, plan, validate, inspect, render, and completions; it is not a supported Slurm runtime target.

Verify A Release

Use GitHub-native verification as the primary trust path for published binaries.

1. Verify the release:

RELEASE_TAG=vX.Y.Z
gh release verify "$RELEASE_TAG" -R NicolasSchuler/hpc-compose

2. Verify a downloaded asset:

RELEASE_TAG=vX.Y.Z
ASSET="hpc-compose-${RELEASE_TAG}-x86_64-unknown-linux-musl.tar.gz"

gh release download "$RELEASE_TAG" -R NicolasSchuler/hpc-compose -p "$ASSET"
gh release verify-asset "$RELEASE_TAG" "./$ASSET" -R NicolasSchuler/hpc-compose

3. Verify the artifact attestation directly:

gh attestation verify "./$ASSET" \
  --repo NicolasSchuler/hpc-compose \
  --signer-workflow NicolasSchuler/hpc-compose/.github/workflows/release.yml

Published releases also ship SHA256SUMS and per-asset .sha256 files. Those checksums are primarily for installer compatibility, mirroring, and corruption checks; attestations are the stronger authenticity signal.

Internal Mirrors And Cluster-Admin Installs

For internal mirrors, preserve release filenames exactly, including:

• platform archives or native packages
• SHA256SUMS
• each per-asset .sha256 sidecar

Then point the installer at the mirrored base URL and pin the matching version:

RELEASE_TAG=vX.Y.Z
curl -fsSL "https://raw.githubusercontent.com/NicolasSchuler/hpc-compose/${RELEASE_TAG}/install.sh" \
  | env HPC_COMPOSE_BASE_URL="https://mirror.example.org/hpc-compose/${RELEASE_TAG}" \
        HPC_COMPOSE_VERSION="$RELEASE_TAG" sh

HPC_COMPOSE_VERSION is required when HPC_COMPOSE_BASE_URL is set so the installer, mirrored assets, and checksum files stay aligned.

Build From Source

Use this path for development, unreleased testing, or local inspection. Building from source requires a Rust 1.88 or newer toolchain (the crate uses edition 2024):

git clone https://github.com/NicolasSchuler/hpc-compose.git
cd hpc-compose
cargo build --release
./target/release/hpc-compose --help

Before using a local build on a cluster workflow, validate the binary and one example spec. validate and plan are static and need no cache directory:

target/release/hpc-compose validate -f examples/minimal-batch.yaml
target/release/hpc-compose plan --verbose -f examples/minimal-batch.yaml

To point the binary at a shared cache when you do run a job, set HPC_COMPOSE_CACHE_DIR, pass --cache-dir, or set x-slurm.cache_dir in the spec.

Local Docs Commands

The repo ships two documentation layers:

• mdbook for the user manual
• cargo doc for contributor-facing crate internals

Useful commands:

mdbook build docs
mdbook serve docs
cargo doc --no-deps

Regenerate checked-in manpages from a checkout with:

cargo run --locked --features manpage-bin --bin gen-manpages
cargo test --locked --test release_metadata
man -l man/man1/hpc-compose.1

Related Docs

• Support Matrix
• Quickstart
• Why hpc-compose
• Runtime Backends

Quickstart

This is the shortest safe path from an empty shell to a static plan, a first real Slurm run, and one-command failure triage.

If Slurm terms such as sbatch, srun, allocation, job step, Pyxis, or Enroot are unfamiliar, read Slurm And Container Basics before the first real cluster run.

1. Install The CLI

Installation is the single owner of install, verify, mirror, and source-build commands. Install the CLI from there, confirm hpc-compose --version works, then return here.

2. Learn The Safe Authoring Path First

The safe authoring path runs entirely on a laptop, workstation, or login node — new writes a local starter spec and plan is purely static (no sbatch, no image import):

hpc-compose new --template minimal-batch --name my-app --output compose.yaml
hpc-compose plan -f compose.yaml
hpc-compose plan --show-script -f compose.yaml

plan validates the spec and resolves service order; plan --show-script adds the rendered batch script. Run that block first on macOS, a laptop, or any machine where you want to evaluate the authoring model before touching a real cluster. The Overview page covers the same walkthrough with full expected output.

If you want a guided learning path instead of a single starter template, run the Spec Metamorphosis tutorial:

hpc-compose evolve --output compose.yaml

The normal workflow to remember is:

hpc-compose plan -f compose.yaml
hpc-compose up -f compose.yaml
hpc-compose debug -f compose.yaml --preflight

3. Choose A Starting Spec

Use the built-in starter templates when you want a fresh compose.yaml with your application name filled in:

hpc-compose new \
  --template minimal-batch \
  --name my-app \
  --output compose.yaml

Add --cache-dir '<shared-cache-dir>' when you want the generated file to include an explicit x-slurm.cache_dir. Otherwise the plan uses the active settings cache default or $HOME/.cache/hpc-compose.

From a source checkout, you can also inspect a known-good repository example:

hpc-compose plan -f examples/minimal-batch.yaml

The Examples page is the single selection guide for beginner, LLM, training, distributed, and pipeline workflows.

Use Spec Metamorphosis when you want to learn those concepts progressively in one evolving valid spec.

4. Pick And Test A Cache Directory

cache_dir is optional in the spec, but real clusters usually need a site-specific shared path because image preparation happens before the job starts and compute nodes must later see those artifacts.

Ask your cluster documentation or support team for a project scratch, work, or shared filesystem path, then test it:

export CACHE_DIR=/cluster/shared/hpc-compose-cache
mkdir -p "$CACHE_DIR"
test -w "$CACHE_DIR"

Persist it in project settings when you want the same value every time:

hpc-compose setup --profile-name dev --cache-dir "$CACHE_DIR" --default-profile dev --non-interactive

Or keep using an environment-backed explicit spec value and persist it next to your copied spec:

printf 'CACHE_DIR=%s\n' "$CACHE_DIR" > .env

Do not use /tmp, /var/tmp, /private/tmp, or /dev/shm for x-slurm.cache_dir. Validation may accept those strings, but preflight reports them as unsafe because prepare happens before runtime and compute nodes must later see the cached artifacts.

5. Before Your First Cluster Run

For Pyxis, srun --help should mention --container-image.

Everything above is safe on any machine. Everything below requires a real Slurm submission host.

The steps up to here only author specs, prepare a cache path, and read static plans. From this point the commands call sbatch, srun, and the runtime backend, so run them only on a supported Linux Slurm submission host.

6. Submit On A Real Cluster

When you move to a supported Linux submission host, the normal run is:

hpc-compose up -f compose.yaml

up runs preflight, prepares missing artifacts, renders the batch script, submits it through sbatch, then follows scheduler state and tracked logs. On the first run (or after cache eviction) the prepare step imports your container image with enroot — a multi-GB download, then extract and squashfs build — which can take several minutes; later runs reuse the cache, and an interactive terminal streams live import sub-progress. On an interactive TTY it opens the full-screen watch UI; otherwise it falls back to line-oriented output. Add --watch-queue when you want line-oriented queue polling until the Slurm job reaches RUNNING before the normal watch view opens; --queue-warn-after <DURATION> controls the one-time long-pending warning. The watch UI holds the final screen on failures by default; use --hold-on-exit never|failure|always to tune that behavior. Use hpc-compose up --detach -f compose.yaml when you want submit-and-return behavior.

Success looks like:

• the job is submitted or launched
• a tracked job id is recorded
• the watch UI or text follower shows scheduler progress
• status, ps, and logs can reconnect to the tracked run later

7. If The First Cluster Run Fails

The longer symptom guide is Troubleshooting.

8. Revisit A Tracked Run Later

hpc-compose jobs list
hpc-compose status -f compose.yaml
hpc-compose ps -f compose.yaml
hpc-compose watch -f compose.yaml
hpc-compose stats -f compose.yaml
hpc-compose logs -f compose.yaml --follow

Use jobs list first when you need to rediscover tracked runs under the current repo tree. Use ps for a stable per-service snapshot, watch to reconnect to the live UI, and logs --follow for a text-only follower.

From A Source Checkout

If you are developing from a local checkout instead of an installed binary:

cargo build --release
target/release/hpc-compose validate -f examples/minimal-batch.yaml
target/release/hpc-compose plan -f examples/minimal-batch.yaml
target/release/hpc-compose plan --show-script -f examples/minimal-batch.yaml

Read Next

• Installation
• Support Matrix
• Why hpc-compose
• Slurm And Container Basics
• Examples
• Runtime Backends
• Runbook
• Troubleshooting

Why hpc-compose

This is the canonical explainer for hpc-compose.

hpc-compose exists because two common approaches leave a gap:

• plain sbatch scripts give you control, but multi-service coordination, startup ordering, and repeatability stay ad hoc
• Docker Compose is familiar, but its networking and orchestration assumptions do not map cleanly to one Slurm allocation

hpc-compose takes the narrow path between them: a Compose-like authoring model that still produces one inspectable Slurm job.

The Pain in Current Slurm Workflows

Once a job stops being a single process, the friction climbs quickly:

• helper services need explicit startup ordering
• cluster-specific environment setup gets mixed into hand-written shell
• debugging starts from generated state you never inspected beforehand
• repeated workflows drift because the real behavior lives across scripts, notes, and local conventions

This is especially common in research ML and HPC-adjacent work where one job may need:

• a serving process plus a client
• a database plus a worker
• a training step plus checkpoint export and resume handling

Why Docker Compose Does Not Fit Slurm Directly

Docker Compose is good at expressing a small multi-service application on one machine. Slurm solves a different problem: scheduling one batch allocation onto shared cluster resources.

That mismatch is why hpc-compose leaves several Compose features out by design. See Slurm Capability Scope for the exact unsupported-features list.

The omissions are deliberate. The point is not to emulate all of Compose on a cluster. The point is to keep a familiar authoring shape for the subset that maps cleanly to one Slurm job.

The Narrow Execution Model

hpc-compose keeps the execution model explicit: a compose-like spec is planned and rendered on the submission host into one batch script, which Slurm runs as one allocation. See Execution Model for the full spec->sbatch->srun pipeline.

That explicitness gives you a few important properties:

• one inspectable unit of submission
• one obvious place to look when the job fails
• one explicit product boundary instead of hidden orchestration behavior

One Real Example

app-redis-worker.yaml is a good example of the intended shape:

• one Redis service
• one dependent worker service
• TCP readiness gating before the worker starts
• both services living inside the same allocation

That is awkward to hand-roll repeatedly with cluster scripts alone, but it does not justify a full orchestrator. This is the exact middle ground hpc-compose targets.

If you want the smallest possible first run, start with minimal-batch.yaml. If you want the smallest concrete inference flow, start with llm-curl-workflow-workdir.yaml.

Why the Inspectable Path Matters

The authoring flow is designed to answer the practical questions before you launch:

hpc-compose plan -f compose.yaml
hpc-compose plan --show-script -f compose.yaml

That lets you confirm:

• whether the spec is valid
• what service order will run
• what image and cache behavior the planner inferred
• what batch script you are actually handing to Slurm

For a Slurm-first tool, that inspectability matters more than feature breadth.

When Not To Use hpc-compose

Do not use hpc-compose when you need:

• custom container networking
• broad Docker Compose compatibility
• a long-running orchestration control plane
• dynamic cross-node scheduling instead of explicit x-slurm.placement node selectors

If that list rules out your workload, that is not a failure of the tool. It is the intended product boundary.

Read Next

• Slurm and Container Basics
• Runtime Backends
• Execution Model
• Slurm Capability Scope

Slurm and Container Basics

This page is for users who know shell scripts, Python jobs, or Docker images, but are new to Slurm and HPC container runtimes.

It is not a Slurm administration guide. The goal is to explain the vocabulary you will see in generated hpc-compose scripts and in cluster error messages.

The Short Mental Model

The important point is that hpc-compose does not replace Slurm. It writes one inspectable Slurm batch script and uses Slurm to run the planned services inside one allocation. For the full spec->sbatch->srun pipeline, see Execution Model.

Slurm Terms In Plain Language

If you only remember one distinction: sbatch gets the allocation; srun starts work inside it.

A Minimal sbatch Script

A traditional Slurm script often looks like this:

#!/usr/bin/env bash
#SBATCH --job-name=hello-slurm
#SBATCH --partition=<partition>
#SBATCH --time=00:10:00
#SBATCH --cpus-per-task=2
#SBATCH --mem=4G

set -euo pipefail

hostname
python -c 'print("hello from a Slurm job")'

Submit it from a Slurm login node:

sbatch hello.sbatch

sbatch returns a job id. The job may wait in the queue before it starts, and Slurm normally writes batch output to a file such as slurm-<job-id>.out unless the script or site policy sets another output path.

Where hpc-compose Fits

The equivalent hpc-compose starting point is a spec:

name: hello-slurm

x-slurm:
  job_name: hello-slurm
  partition: <partition>
  time: "00:10:00"
  cpus_per_task: 2
  mem: 4G

services:
  app:
    image: python:3.11-slim
    command: python -c "import socket; print('hello from', socket.gethostname())"

Preview the generated Slurm script before submitting:

hpc-compose plan -f compose.yaml
hpc-compose plan --show-script -f compose.yaml

Run it on a supported Slurm login node:

hpc-compose up -f compose.yaml

up runs preflight checks, prepares missing runtime artifacts, renders the batch script, calls sbatch, records tracked job metadata, and follows scheduler/log output.

How YAML Maps To Slurm

hpc-compose translates top-level and service x-slurm fields into #SBATCH directives and srun arguments. For the exact field-by-field mapping and the full command surface (sbatch, srun, render, up, tracked follow-ups), see Spec Reference and CLI Reference. Prefer first-class fields when they exist; use raw submit_args or extra_srun_args only for site-specific options that hpc-compose does not model directly.

When debugging, inspect the generated script:

hpc-compose plan --show-script -f compose.yaml

If a job was submitted but failed before service logs appeared, inspect Slurm state and batch output through:

hpc-compose debug -f compose.yaml

Pyxis And Enroot Basics

Slurm itself is the scheduler. Container support depends on what the cluster installed. The default runtime.backend: pyxis path uses the Pyxis Slurm plugin plus the Enroot unprivileged runtime, and hpc-compose maps each service into a generated srun --container-* launch.

For the Pyxis support check, the Enroot/Apptainer/Singularity/host tooling differences, and how to choose a backend, see Runtime Backends.

Why Shared Storage Matters

hpc-compose prepare can run before the Slurm job starts, but services run later on compute nodes, so the resolved runtime cache must be visible from both places. For why the cache must live on shared storage and the operational cache configuration, see Execution Model and Cache Management.

The same rule applies to host paths mounted through volumes: the compute node must be able to read the path when the service starts.

Small Checks That Explain A Lot

These commands are useful in tiny smoke tests:

hostname
env | grep '^SLURM_' | sort
python -c 'import socket; print(socket.gethostname())'
cat /etc/os-release

Inside a container, cat /etc/os-release should describe the container image. Outside the container, it describes the host. That simple distinction helps diagnose whether a command is running where you expect.

Common Beginner Mistakes

Further Reading

• Slurm Quick Start User Guide
• Slurm sbatch reference
• Slurm job launch design notes
• Slurm containers guide
• NVIDIA Pyxis
• NVIDIA Enroot

Read Next

• Why hpc-compose
• Runtime Backends
• Execution Model
• Slurm Capability Scope

Runtime Backends

runtime.backend selects how each service is launched inside the Slurm step. The default is pyxis.

For a beginner explanation of Slurm steps, Pyxis, Enroot, and shared runtime caches, start with Slurm And Container Basics.

runtime:
  backend: pyxis

Backend Summary

For Pyxis, check support with:

srun --help | grep container-image

For all backends, preflight checks the selected backend tools:

hpc-compose preflight -f compose.yaml

On the first pyxis/Enroot run, prepare imports the image with enroot — download, extract, then squashfs build — which can take several minutes for a multi-GB image; later runs reuse the cached .sqsh. The extraction scratch defaults to the shared cache (<cache_dir>/enroot/tmp); on shared NFS/Lustre/GPFS storage you can redirect it to node-local storage with x-slurm.enroot_temp_dir (or cache.enroot_temp_dir) to avoid slow imports and Stale file handle errors, while the layer cache and final .sqsh stay on the shared cache. See Files and Directories.

When the prepare scratch is node-local, also watch prepare-time bind mounts: x-runtime.prepare.mounts (and enroot prepare-hook mounts) run on the login node, so a mount whose source is on a network/shared filesystem can become a new failure point during prepare. Prefer a dependency-only prepare — install dependencies into the image during prepare (pip install -r requirements.txt, uv pip install, …) and mount your source tree as a runtime volume (services.<name>.volumes) rather than a prepare.mounts entry — so prepare stays independent of network-FS mounts. examples/dev-python-app.yaml shows source-mounted-at-runtime with deps baked in during prepare. preflight checks prepare mount sources (an absolute source is hinted as a possible cluster-workspace/site-storage path needing provisioning), and a prepare command that fails with bind mounts active lists the active mounts and suggests this pattern.

Installing Python packages (PEP 668 / externally-managed images)

How you install dependencies in prepare depends on the base image’s Python:

• pip install works on the official python:*/python:*-slim images (Python from python.org, installed under /usr/local) and on Conda-based images such as pytorch/pytorch:*. The shipped Python examples use these, so a plain pip install --no-cache-dir <pkgs> is fine.
• pip install is blocked on images whose Python comes from the distribution package manager — e.g. apt install python3 on an ubuntu/debian or nvidia/cuda:*-ubuntu* base. These ship an EXTERNALLY-MANAGED marker (PEP 668), so python -m pip install … fails with “externally managed environment”.

For an externally-managed image, do not reach for pip install --break-system-packages. Use one of:

x-runtime:
  prepare:
    commands:
      # Option A — a dedicated venv that can still see the image's system packages
      # (e.g. a CUDA build of torch baked into the base image):
      - python3 -m venv --system-site-packages /opt/venv
      - /opt/venv/bin/pip install --no-cache-dir <your-extra-deps>
      # Option B — uv, installed without pip via its standalone installer, then
      # installing into the system environment (uv does not honor PEP 668):
      - curl -LsSf https://astral.sh/uv/install.sh | sh
      - $HOME/.local/bin/uv pip install --system --no-cache <your-extra-deps>
services:
  trainer:
    # With Option A, run the venv's interpreter so the extra deps are importable:
    command: ["/opt/venv/bin/python", "train.py"]

--system-site-packages keeps framework packages that are baked into the base image (such as a CUDA-matched PyTorch) visible inside the venv, so you only install the extras on top.

Local Mode

up --local, test --local, dev, and tmux are intentionally narrow:

• Linux only
• runtime.backend: pyxis only
• Pyxis-compatible Enroot tooling on the host
• single-host specs only
• no distributed or partitioned placement
• no service-level MPI
• no Slurm arrays or scheduler dependencies

Use local mode to inspect and debug a Pyxis/Enroot single-host launch path. dev adds file-change restart requests to the local supervisor, and tmux tails tracked local service logs in panes. Neither command changes the process-supervision model, and local mode is not a replacement for Slurm distributed execution.

Host Runtime Notes

runtime.backend: host runs service commands directly under srun. It is useful for module-based workflows or nested schedulers that already manage their own software environment.

Because there is no container:

• image is optional
• service volumes are rejected
• x-runtime.prepare and x-enroot.prepare are rejected
• x-slurm.mpi.host_mpi.bind_paths is rejected

Use top-level or service-level x-env for host modules, Spack views, and environment variables.

Related Docs

• Support Matrix
• Slurm And Container Basics
• Execution Model
• Spec Reference
• CLI Reference

Execution Model

This page explains the few runtime rules that matter most when a Compose mental model meets Slurm and HPC runtime backends.

What runs where

The main consequence is simple: image preparation and validation happen before the job starts, but the containers themselves run later inside the Slurm allocation.

Service failure policies inside one job

hpc-compose does not provide a separate long-running orchestrator. Service failure handling happens inside the rendered batch script for the current allocation.

• mode: fail_job keeps fail-fast behavior and stops the job on the first non-zero service exit.
• mode: ignore records the failure but allows the rest of the job to continue.
• mode: restart_on_failure only reacts to non-zero process exits. It does not restart on successful exits, and it does not use cross-attempt or cross-requeue history.

For restart_on_failure, the batch script enforces two limits during one live execution:

• a lifetime cap through max_restarts
• a rolling-window cap through max_restarts_in_window within window_seconds

If a service omits the rolling-window fields, hpc-compose still enables crash-loop protection with window_seconds: 60 and max_restarts_in_window: <resolved max_restarts>.

Use status to inspect the tracked policy state after submission. The text view reports:

state service 'worker': failure_policy=restart_on_failure restarts=1/5 window=1/3@60s last_exit=42 completed=no

Use logs to inspect the corresponding restart messages from the batch script when you need to distinguish lifetime-cap exhaustion from rolling-window exhaustion.

Use per-service x-slurm.hooks when you want host-side notifications around those policy transitions. on: restart runs before a granted relaunch; on: window_exhausted runs when the rolling-window guard blocks another restart. These hooks are best-effort and do not change the service policy outcome.

Which paths must be shared

• The resolved cache directory must be visible from both the login node and the compute nodes. It may come from x-slurm.cache_dir, project settings, or the builtin $HOME/.cache/hpc-compose fallback.
• Relative host paths in volumes, local image paths, and x-runtime.prepare.mounts resolve against the compose file directory.
• Each submitted job writes per-job runtime state under <runtime-root>/<job-id> on the host. <runtime-root> defaults to <submit-dir>/.hpc-compose and can be overridden with x-slurm.runtime_root.
• The active job workspace is mounted into containerized services at /hpc-compose/job. For ordinary runs that workspace is <runtime-root>/<job-id>; for resume-aware attempts it is <runtime-root>/<job-id>/attempts/<attempt>, with top-level paths kept as the latest view.
• Multi-node jobs also populate /hpc-compose/job/allocation/{primary_node,nodes.txt} and export allocation-wide HPC_COMPOSE_NODE... variables plus service-scoped HPC_COMPOSE_SERVICE_NODE... variables.

Use /hpc-compose/job for small shared state inside the allocation, such as ready files, request payloads, logs, metrics, or teardown signals.

Enroot runtime paths

The generated batch script sets three Enroot runtime paths scoped per job under the resolved cache directory:

These paths are created at batch startup and are available inside the batch script and to tooling that reads Enroot environment variables. They are not injected into service containers.

The cache must live on storage shared between login and compute nodes because prepare runs on the login node while services run on compute nodes; node-local /tmp fails because each node sees a different filesystem. For the operational list of invalid cache paths and cache configuration, see Cache Management.

Networking inside the allocation

• Single-node services share the host network on one node.
• In a multi-node job, helper services stay on the allocation’s primary node by default.
• A distributed service may span the full allocation, or services may use x-slurm.placement to select explicit allocation node subsets.
• Partitioned services should use service-scoped metadata such as HPC_COMPOSE_SERVICE_PRIMARY_NODE, HPC_COMPOSE_SERVICE_NODE_COUNT, HPC_COMPOSE_SERVICE_NODELIST, and HPC_COMPOSE_SERVICE_NODELIST_FILE.
• ports, custom Docker networks, and service-name DNS are not part of the model.
• Use depends_on plus readiness when a dependent service must wait for real availability rather than process start.
• Use depends_on with condition: service_completed_successfully when a dependent service should wait for a one-shot stage to exit successfully.

Use 127.0.0.1 only when both sides are intentionally on the same node. For multi-node distributed or partitioned runs, derive rendezvous addresses from allocation or service metadata files and environment variables instead of relying on localhost.

If a service binds its TCP port before it is actually ready, prefer HTTP or log-based readiness over plain TCP readiness.

volumes vs x-runtime.prepare

Recommended default:

• keep active source trees in volumes
• keep slower-changing dependency installation in x-runtime.prepare.commands
• use prepare.mounts only when the prepare step truly needs host files

Command vocabulary

• The normal run is hpc-compose up -f compose.yaml. See Quickstart for the full end-to-end description.
• The tracked follow-up tools are status for scheduler/log summaries, ps for a stable per-service snapshot, and watch when you want to reconnect to the live TUI later.
• The debugging flow is validate, inspect, preflight, and prepare run separately when you need more visibility.

Read Runtime Backends before changing runtime.backend, Runbook for the operational workflow, Examples for starting points, and Spec reference for exact field behavior.

Related Docs

• Why hpc-compose
• Runtime Backends
• Slurm Capability Scope
• Spec Reference

Slurm Capability Scope

This page makes the hpc-compose Slurm boundary explicit. It is a tool for compiling one Compose-like application into one Slurm allocation with one or more srun steps. Those steps can use Pyxis/Enroot, Apptainer, Singularity, or host runtime software. It is not a general frontend for the full Slurm command surface.

First-class support

These capabilities are modeled, validated, and intentionally supported by the planner, renderer, and tracked-job workflow.

Raw pass-through

These capabilities are usable, but hpc-compose does not model or validate their semantics beyond passing them through to Slurm.

Pass-through is appropriate when a site-specific flag is useful but does not justify a first-class schema field. hpc-compose rejects line breaks and null bytes in raw #SBATCH entries so one list entry cannot emit multiple directives, but it does not validate the Slurm semantics of those flags.

Unsupported or out of scope

These capabilities are intentionally outside the product seam.

Non-goals

hpc-compose should not grow into a generic Slurm administration layer. In particular, it will not broaden into sacctmgr, reservation management, federation control, or generic scontrol mutation. Those are real Slurm features, but they do not fit the “one application, one allocation, tracked runtime workflow” seam this tool is built around.

Related Docs

• Why hpc-compose
• Execution Model
• Runtime Backends
• Spec Reference

Examples

These examples are the fastest way to understand the intended hpc-compose workflows and adapt them to a real application.

There are two starting points:

• built-in starter templates generated by hpc-compose new
• repository example files copied directly from examples/

Use the CLI recommendation flow when you want a ranked starting point, or the coverage map when you want to inspect every shipped example by workflow or tag:

hpc-compose examples recommend
hpc-compose examples recommend 'vllm worker'
hpc-compose examples recommend 'multi-node training' --tag gpu
hpc-compose examples list --tag mpi
hpc-compose examples search 'vllm worker'

hpc-compose examples recommend is static and authoring-only: it uses the checked-in example registry, tags, and prerequisite notes; it does not inspect the cluster, contact Slurm, or submit jobs. Each result explains why it matched and prints safe next commands such as hpc-compose new, cp, plan, and plan --show-script.

Before launching anything, run the safe authoring path first:

hpc-compose new --template minimal-batch --name my-app --output compose.yaml
hpc-compose plan -f compose.yaml
hpc-compose plan --show-script -f compose.yaml

If you are reading from a source checkout, you can run the same static checks directly against examples/minimal-batch.yaml.

Some repository examples keep an explicit ${CACHE_DIR:-/cluster/shared/hpc-compose-cache} for portability, while starter examples rely on the settings/builtin cache default. Before running on a real cluster, configure a shared path visible from both the submission host and the compute nodes:

export CACHE_DIR=/cluster/shared/hpc-compose-cache
mkdir -p "$CACHE_DIR"
test -w "$CACHE_DIR"

Start Here: The Four Promoted Examples

These four examples are the intended conversion funnel.

minimal-batch.yaml

• Demonstrates: one service, no dependencies, no image prepare step
• Expected prerequisites: any machine for plan; a Linux Slurm login node plus the selected runtime backend for up
• Cluster run, Linux Slurm login node only: hpc-compose up -f examples/minimal-batch.yaml
• Success signal: the batch log prints Hello from Slurm!

app-redis-worker.yaml

• Demonstrates: multi-service startup ordering plus TCP readiness inside one allocation
• Expected prerequisites: a normal Slurm + Enroot submission host and shared CACHE_DIR
• Cluster run, Linux Slurm login node only: hpc-compose up -f examples/app-redis-worker.yaml
• Success signal: worker.log shows a successful Redis PING followed by repeated INCR jobs calls

llm-curl-workflow-workdir.yaml

• Demonstrates: one GPU-backed LLM service plus one client service in the same job
• Expected prerequisites: a GGUF model at $HOME/models/model.gguf, a GPU-capable Slurm target, and shared CACHE_DIR
• Cluster run, Linux Slurm login node only: hpc-compose up -f examples/llm-curl-workflow-workdir.yaml
• Success signal: curl_client.log contains a JSON response from /v1/chat/completions

training-resume.yaml

• Demonstrates: checkpoint export, resume-aware reruns, and attempt-aware training state
• Expected prerequisites: shared storage for x-slurm.resume.path plus shared CACHE_DIR
• Cluster run, Linux Slurm login node only: hpc-compose up -f examples/training-resume.yaml
• Success signal: results/<job-id>/ contains exported checkpoints and later attempts resume from the previously saved epoch

Beginner Ladder

Use this ordering when you are new to the project:

For a guided version of the first five concepts, run hpc-compose evolve --output compose.yaml. The progressive-complexity lesson walks through minimal, second-service, readiness, failure-policy, and multi-node-placement as one evolving valid spec.

Built-In Starter Templates

Use built-in templates when you want hpc-compose to write a fresh compose.yaml with your application name filled in for you.

hpc-compose new --list-templates
hpc-compose new --describe-template minimal-batch
hpc-compose new --template minimal-batch --name my-app --output compose.yaml
hpc-compose new --template minimal-batch --name my-app --cache-dir '<shared-cache-dir>' --output compose.yaml

If the workflow you want is not listed by --list-templates, copy the closest repository example directly from examples/.

Broader Example Matrix

The matrix below covers the broader set of runnable examples beyond the four promoted starts. “Built-in template” means hpc-compose new --template <name> can scaffold it; “repository file” means copy the YAML from examples/ directly. Generate the same coverage map from the CLI with hpc-compose examples coverage --format markdown.

Which Example Should I Start From?

Run hpc-compose examples recommend with no query for the default beginner path, or pass a short workflow description when you already know the shape you need:

hpc-compose examples recommend
hpc-compose examples recommend 'checkpoint resume training'
hpc-compose examples recommend 'workflow engine bridge'
hpc-compose examples recommend 'separate rendezvous jobs' --format json

The recommendation output is the maintained chooser. It reuses the same registry metadata that feeds the coverage table below, so new examples, tags, and prerequisite notes only need to be updated in one place.

Companion notes for the more involved examples live alongside the example assets:

• examples/llm-curl/README.md
• examples/llama-uv-worker/README.md
• examples/vllm-uv-worker/README.md
• examples/models/README.md

Development Workflow Recipe

examples/dev-python-app.yaml mounts examples/app/ and runs a long-lived Python process, so it is best for hot reload:

hpc-compose dev -f examples/dev-python-app.yaml
hpc-compose tmux -f examples/dev-python-app.yaml --no-attach

examples/dev-python-smoke.yaml keeps the same mounted-source shape but uses a finite command, so it is suitable for smoke tests:

hpc-compose test --local -f examples/dev-python-smoke.yaml
hpc-compose test --submit --time 00:01:00 -f examples/dev-python-smoke.yaml

Adaptation Checklist

1. Copy the closest repository example to your own compose.yaml, or run hpc-compose new --template <name> --name my-app --output compose.yaml when a matching built-in template exists.
2. Configure a cache path visible from both the login node and compute nodes through hpc-compose setup --cache-dir, x-slurm.cache_dir, or [defaults.cache] / [profiles.<name>.cache].
3. Override CACHE_DIR before running repository examples that use ${CACHE_DIR:-...}, or replace the default cache path in your copied file.
4. Replace the example image, command, environment, and volumes with your workload.
5. Keep active source in volumes and keep slower-changing dependency installation in x-runtime.prepare.commands.
6. Add readiness to services that must be reachable before dependents continue.
7. Adjust top-level or per-service x-slurm settings for your cluster.
8. Run hpc-compose plan -f compose.yaml before the first run, and hpc-compose debug -f compose.yaml --preflight if that run fails.
9. Run cluster up only from a supported Linux Slurm submission host with the selected runtime backend available.

Related Docs

• Guided Authoring Tutorial
• Task Guide
• Migrate a docker-compose.yaml
• Spec Reference
• Runbook

Guided Authoring Tutorial

hpc-compose evolve is an interactive authoring tutorial. It starts from a minimal valid spec and progressively rewrites the same output file through increasingly realistic HPC workflow features.

The command is safe to run on a laptop or login node:

• it validates and plans candidate specs,
• it writes only the selected compose file,
• it does not prepare images,
• it does not call sbatch,
• it does not run preflight.

Canonical Lesson

evolve currently ships one lesson:

hpc-compose evolve --describe-lesson progressive-complexity

The progressive-complexity path contains five valid snapshots:

The final step can validate anywhere, but running it requires a Slurm target that can grant a two-node allocation and a runtime backend available on that cluster.

Interactive Flow

Start the tutorial:

hpc-compose evolve --output compose.yaml

At each step, the command prints:

• a short explanation,
• the concepts being introduced,
• a compact diff from the last accepted spec,
• and the validation summary for the candidate.

Controls:

• Enter, y, or a accepts the step and writes compose.yaml.
• s skips the current step.
• q quits after the last accepted valid spec.
• ? prints prompt help.

Transcript Example

$ hpc-compose evolve --output compose.yaml
Step 1/5: Minimal batch spec
Accept this step? [Y/a/s/q/?]
wrote /path/to/compose.yaml

Step 2/5: Add a dependent service
Accept this step? [Y/a/s/q/?]
wrote /path/to/compose.yaml

Step 3/5: Gate on readiness
Accept this step? [Y/a/s/q/?]
wrote /path/to/compose.yaml

Inspect the accepted readiness-gated spec:

hpc-compose plan -f compose.yaml

Then continue the tutorial to failure policies and multi-node placement:

Accept this step? [Y/a/s/q/?]

For automation or docs examples, accept through a specific step noninteractively:

hpc-compose evolve --yes --until readiness --format json --output compose.yaml

Non-Goals

• evolve does not mutate arbitrary existing specs.
• evolve is not a full-screen TUI.
• evolve does not submit jobs.

For a fresh single-template scaffold, use hpc-compose new. For choosing among the broader runnable examples, use Examples.

Related Docs

• Examples
• Task Guide
• Migrate a docker-compose.yaml
• CLI Reference
• Spec Reference

Task Guide

Use this page when you know what you want to do, but not yet which command or example should be your starting point.

First run

• Read Quickstart.
• Run hpc-compose evolve --output compose.yaml if you want a guided progression from minimal through multi-node-placement.
• Run hpc-compose new --list-templates if you want to inspect the built-in starter templates before choosing one.
• Run hpc-compose examples recommend for a static, no-Slurm starting-point recommendation with match reasons and safe next commands. Add a workflow description, such as hpc-compose examples recommend 'vllm worker', when you want registry-backed recommendations for a narrower shape.
• Run hpc-compose examples list or hpc-compose examples search 'vllm worker' when you want to browse the broader example coverage map by workflow or tag.
• Start from minimal-batch with hpc-compose new --template minimal-batch --name my-app --output compose.yaml.
• Before running on a cluster, configure a shared cache with hpc-compose setup --cache-dir '<shared-cache-dir>' or explicit x-slurm.cache_dir. If you copy a repository example that uses CACHE_DIR, override it for your cluster before running.
• Run hpc-compose plan -f compose.yaml before the first real run. Add --show-script when you want to inspect the generated launcher without writing a file.
• Run hpc-compose up -f compose.yaml only from a supported Linux Slurm submission host.

Remember directory/data/env settings once

• Run hpc-compose setup to create or update the project-local settings file (.hpc-compose/settings.toml).
• Use hpc-compose --profile dev up so compose path, env files, env vars, and binary paths come from the selected profile.
• Run hpc-compose context --format json to inspect resolved paths plus value sources. Interpolation variables are scoped to names referenced by the compose file and sensitive-looking values are redacted unless you add --show-values.
• Use --settings-file <PATH> when you need an explicit settings file instead of upward discovery.

Migrate from Docker Compose

• Read Docker Compose Migration.
• Replace build: with image: plus x-runtime.prepare.commands.
• Replace service-name networking with 127.0.0.1 or explicit allocation metadata where appropriate.

Pick a starting example

• Browse the annotated catalog and chooser in Examples; it owns the per-example filename, tag, and prerequisite map.
• Run hpc-compose examples recommend '<workflow description>' for a registry-backed starting point, e.g. 'multi-service app', 'multi-node training', 'checkpoint resume training', or 'vllm worker'.

Single-node multi-service app

• Use Execution Model to confirm which services can rely on localhost.
• Add depends_on and readiness only where ordering really matters.

Multi-node distributed training

• Use generated distributed metadata such as HPC_COMPOSE_DIST_RDZV_ENDPOINT, HPC_COMPOSE_DIST_NODE_RANK, and HPC_COMPOSE_DIST_NPROC_PER_NODE instead of Docker-style service discovery.
• Put cluster-specific NCCL/UCX/OFI fabric variables in .hpc-compose/cluster.toml under [distributed.env] so specs stay portable.

Checkpoint and resume workflows

• See Artifacts and Resume for the export-vs-resume split.
• Keep the canonical resume source in x-slurm.resume.path, not in exported artifact bundles.

LLM serving workflows

• Use volumes for model directories and fast-changing code.
• Use x-runtime.prepare.commands for slower-changing dependencies.

Debug cluster readiness

• Run hpc-compose validate -f compose.yaml.
• Run hpc-compose validate -f compose.yaml --strict-env when default interpolation fallbacks should be treated as failures.
• Run hpc-compose plan --verbose -f compose.yaml.
• Run hpc-compose preflight -f compose.yaml.
• Run hpc-compose debug -f compose.yaml --preflight after a failed tracked run.
• Run hpc-compose doctor readiness -f compose.yaml --service <name> to inspect the normalized readiness probe, or add --run when the target service, tunnel, or log file is already reachable from the current host.
• Read Troubleshooting.

Cache and artifact management

• Use hpc-compose cache list to inspect imported/prepared artifacts.
• Use hpc-compose cache inspect -f compose.yaml to see per-service reuse expectations.
• Use hpc-compose --profile dev cache prune --age 14 when you want age-based cleanup to follow the active context cache dir.
• Use hpc-compose cache prune --age 7 --cache-dir '<shared-cache-dir>' when you want a direct cache cleanup that does not depend on compose resolution.
• Use hpc-compose artifacts -f compose.yaml after a run to export tracked payloads.

Find and clean tracked runs

• Use hpc-compose jobs list to scan the current repo tree for tracked runs.
• Use hpc-compose ps -f compose.yaml when you want a one-shot per-service runtime table.
• Use hpc-compose watch -f compose.yaml to reconnect to the live watch UI for the latest tracked job.
• Use hpc-compose jobs list --disk-usage when you need a quick size estimate before deleting old state.
• Use hpc-compose clean -f compose.yaml --dry-run --age 7 to preview what a cleanup would remove.
• Use hpc-compose clean -f compose.yaml --all --format json when automation needs a stable cleanup report for one compose context, including effective latest IDs plus stale-pointer diagnostics.

Automation and scripting with JSON output

• Prefer --format json for machine-readable output on non-streaming commands such as new, plan, validate, render, prepare, preflight, config, inspect, debug, status, ps, stats, score, artifacts, down, cancel, setup, cache list/cache inspect/cache prune, clean, and context. For up, --format json requires --detach or --dry-run.
• Include context --format json when automation needs resolved compose path, binaries, referenced interpolation vars, and runtime path roots.
• Use hpc-compose stats --format jsonl or --format csv when downstream tooling wants row-oriented metrics.
• Use --format json for machine-readable output on non-streaming commands. Streaming commands such as logs --follow, watch, and completions keep their native text or script output.

Related Docs

• Examples
• Guided Authoring Tutorial
• Migrate a docker-compose.yaml
• CLI Reference
• Runbook

Migrate a docker-compose.yaml

This guide helps you convert an existing docker-compose.yaml into an hpc-compose spec for Slurm clusters using Pyxis/Enroot, Apptainer, Singularity, or host runtimes.

At a glance

Side-by-side: web app + Redis

Docker Compose

version: "3.9"
services:
  redis:
    image: redis:7
    ports:
      - "6379:6379"
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 5s
      timeout: 3s
      retries: 5

  app:
    build: .
    ports:
      - "8000:8000"
    depends_on:
      redis:
        condition: service_healthy
    environment:
      REDIS_HOST: redis
    volumes:
      - ./app:/workspace
    working_dir: /workspace
    command: python -m main

hpc-compose

version: "1"
name: my-app

x-slurm:
  job_name: my-app
  time: "01:00:00"
  mem: 8G
  cpus_per_task: 4
  cache_dir: /cluster/shared/hpc-compose-cache

services:
  redis:
    image: redis:7
    command: redis-server --save "" --appendonly no
    readiness:
      type: tcp
      host: 127.0.0.1
      port: 6379
      timeout_seconds: 30

  app:
    image: python:3.11-slim
    depends_on:
      redis:
        condition: service_healthy
    environment:
      REDIS_HOST: 127.0.0.1
    volumes:
      - ./app:/workspace
    working_dir: /workspace
    command: python -m main
    x-runtime:
      prepare:
        commands:
          - pip install --no-cache-dir redis fastapi uvicorn

Key changes

1. version: "3.9" → version: "1" or remove the field. hpc-compose uses this as its own spec schema version, not a Docker Compose compatibility version.
2. build: . → image: python:3.11-slim + x-runtime.prepare.commands for dependencies.
3. ports → Removed. Services communicate via 127.0.0.1 because they run on the same node.
4. REDIS_HOST: redis → REDIS_HOST: 127.0.0.1. No DNS service names; use localhost.
5. healthcheck → readiness with type: tcp.
6. Added x-slurm block for Slurm resource allocation (time, memory, CPUs).
7. Configured a shared cache for image storage, either through x-slurm.cache_dir as shown or project settings.

Key differences

Networking

Docker Compose creates isolated networks where services find each other by name. In hpc-compose, helper services on the same node share the host network directly, and multi-node distributed steps must use explicit rendezvous addresses. Replace service hostnames with 127.0.0.1 only when both sides intentionally stay on one node. For multi-node runs, derive the rendezvous host from /hpc-compose/job/allocation/primary_node or HPC_COMPOSE_PRIMARY_NODE.

Building images

Docker Compose uses build: to run a Dockerfile. hpc-compose uses x-runtime.prepare.commands instead:

# Docker Compose
app:
  build:
    context: .
    dockerfile: Dockerfile

# hpc-compose
app:
  image: python:3.11-slim
  x-runtime:
    prepare:
      commands:
        - pip install --no-cache-dir -r /tmp/requirements.txt
      mounts:
        - ./requirements.txt:/tmp/requirements.txt

Prefer volumes for fast-changing source code and x-runtime.prepare.commands for slower-changing dependencies. x-enroot.prepare remains accepted as a Pyxis/Enroot compatibility spelling, but new specs should use x-runtime.prepare.

Health checks vs readiness

Docker Compose uses healthcheck with a test command, interval, timeout, and retries. hpc-compose now accepts a constrained healthcheck subset and normalizes it into readiness:

# TCP: wait for a port to accept connections
readiness:
  type: tcp
  host: 127.0.0.1
  port: 6379
  timeout_seconds: 30

# HTTP: wait for an endpoint to return an expected status
readiness:
  type: http
  url: http://127.0.0.1:8080/health
  status_code: 200
  timeout_seconds: 30

# Log: wait for a pattern in service output
readiness:
  type: log
  pattern: "Server started"
  timeout_seconds: 60

# Sleep: fixed delay
readiness:
  type: sleep
  seconds: 5

Supported healthcheck migration patterns:

• ["CMD", "nc", "-z", HOST, PORT]
• ["CMD-SHELL", "nc -z HOST PORT"]
• recognized curl probes against http:// or https:// URLs
• recognized wget --spider probes against http:// or https:// URLs

Still unsupported:

• arbitrary custom command probes
• interval
• retries
• start_period

Resource allocation

Docker Compose uses deploy.resources or top-level cpus/mem_limit. hpc-compose uses Slurm-native resource settings:

x-slurm:
  time: "02:00:00"
  mem: 32G
  cpus_per_task: 8
  gpus: 1

services:
  app:
    x-slurm:
      cpus_per_task: 4
      gpus: 1

Restart policies

Docker Compose supports restart: always, on-failure, etc. hpc-compose does not accept the Compose restart: key, but it does support per-service restart behavior through services.<name>.x-slurm.failure_policy.

services:
  app:
    image: python:3.11-slim
    x-slurm:
      failure_policy:
        mode: restart_on_failure
        max_restarts: 3
        backoff_seconds: 5
        window_seconds: 60
        max_restarts_in_window: 3

restart_on_failure retries only on non-zero exits. It enforces both a lifetime restart cap and a rolling-window crash-loop cap during one live batch-script execution. If you omit the rolling-window fields, hpc-compose defaults to window_seconds: 60 and max_restarts_in_window: <resolved max_restarts>. Use mode: fail_job (default) for fail-fast behavior, or mode: ignore for non-critical sidecars.

Practical mapping:

• Compose restart: "no" -> omit failure_policy or use mode: fail_job
• Compose restart: on-failure[:N] -> use mode: restart_on_failure with max_restarts: N when you want a similar lifetime retry budget
• Compose restart: always / unless-stopped -> no direct equivalent; hpc-compose intentionally keeps restart handling bounded within one batch job

The rolling-window fields have no direct Docker Compose equivalent. They exist to stop fast crash loops inside one Slurm allocation without giving up a larger lifetime retry budget for transient failures.

What to do about unsupported features

Migration checklist

1. Replace Compose version: — Use version: "1" or omit the field; values like "3.9" are rejected by hpc-compose.
2. Remove build: — Replace with image: pointing to a base image. Move dependency installation to x-runtime.prepare.commands.
3. Remove ports: — Use host-network semantics instead of container port publishing.
4. Remove networks: / network_mode: — There is no Docker-style overlay network or service-name DNS layer.
5. Remove Compose restart: — use services.<name>.x-slurm.failure_policy when you need per-service restart behavior.
6. Remove deploy: — Use x-slurm for resource allocation.
7. Replace service hostnames — Change any service-name references (e.g. redis, postgres) to 127.0.0.1 for same-node helpers, or to explicit allocation metadata for distributed runs.
8. Replace healthcheck: — Convert to readiness: with type: tcp, type: http, type: log, or type: sleep.
9. Add x-slurm: — Set time, mem, cpus_per_task, and optionally gpus, partition, account.
10. Set cache storage — Point x-slurm.cache_dir or setup --cache-dir to shared storage visible from login and compute nodes.
11. Validate — Run hpc-compose validate -f compose.yaml to check the converted spec.
12. Inspect — Run hpc-compose inspect --verbose -f compose.yaml to confirm the planner understood your intent.

Related Docs

• Examples
• Guided Authoring Tutorial
• Task Guide
• Execution Model
• Spec Reference

Operate a Real Cluster Run

This runbook is the normal real-cluster flow for adapting a hpc-compose spec on a supported Linux Slurm submission host.

If you are new to Slurm, read Slurm And Container Basics first. If you are adapting to HAICORE@KIT, read HAICORE Guide alongside this runbook.

Commands below assume hpc-compose is on your PATH. If you are running from a local checkout, replace hpc-compose with target/release/hpc-compose.

Compose-aware commands accept -f / --file. When omitted, hpc-compose uses the active context compose file from .hpc-compose/settings.toml, then falls back to compose.yaml in the current directory. Global context flags are available everywhere:

• --profile <NAME> selects a profile from .hpc-compose/settings.toml.
• --settings-file <PATH> uses an explicit settings file instead of upward auto-discovery.

Read Slurm And Container Basics, Execution Model, Runtime Backends, and Support Matrix before adapting a workflow to a new cluster.

Before You Start

Make sure you have:

• a Linux submission host with srun and sbatch,
• the runtime backend selected by runtime.backend,
• scontrol when x-slurm.nodes > 1,
• Pyxis support in srun when runtime.backend: pyxis (srun --help should mention --container-image),
• shared storage for the resolved cache directory,
• local source trees or local .sqsh / .sif images in place,
• registry credentials when your cluster or registry requires them.

Backend-specific requirements are listed in Runtime Backends. Cluster profile generation and MPI smoke probes are covered in Cluster Profiles.

The Operational Spine

For a new spec on a real cluster, work the numbered steps below in order:

1. Choose a starter from Examples, or run hpc-compose new --template <name> --name my-app --output compose.yaml. See Choose A Starting Example.
2. Run hpc-compose setup once and verify resolved values with hpc-compose context --format json. See Project-Local Settings.
3. Choose the cache directory early. See Choose A Cache Directory Early.
4. Adapt the example and adjust cluster-specific resource settings. See Adapt The Example.
5. Validate the spec. See Validate The Spec.
6. Plan the run. See Plan The Run.
7. Launch with up. See Normal Run: Use up.
8. When debugging cluster readiness, prepare, or rendering, break out preflight, prepare, and render separately. See steps 6–8.
9. Inspect the tracked run. See Inspect A Tracked Run.
10. Manage cache and old state. See Manage Cache And Old State.

If a run fails, start with hpc-compose debug -f compose.yaml --preflight, then follow the First Triage flow in Troubleshooting.

For a minimal cluster smoke test from a checkout, set CACHE_DIR to shared storage and run scripts/cluster_smoke.sh. It validates, preflights, and renders by default; set HPC_COMPOSE_SMOKE_SUBMIT=1 only when you intentionally want it to launch the smoke job.

Project-Local Settings

hpc-compose can discover .hpc-compose/settings.toml by walking upward from the current directory. You can also pin a file with --settings-file.

Typical setup flow:

hpc-compose setup
hpc-compose context
hpc-compose --profile dev context --format json

Non-interactive setup is available for scripting:

hpc-compose setup --profile-name dev --compose-file compose.yaml --env-file .env --env-file .env.dev --cache-dir '<shared-cache-dir>' --default-profile dev --non-interactive

Settings file shape:

version = 1
default_profile = "dev"

[defaults]
compose_file = "compose.yaml"
env_files = [".env"]
login_host = "login01.hpc.example.edu"
login_user = "<username>"

[defaults.env]
CACHE_DIR = "/cluster/shared/hpc-compose-cache"

[defaults.cache]
dir = "/cluster/shared/hpc-compose-cache"

[profiles.dev]
compose_file = "compose.yaml"
env_files = [".env", ".env.dev"]

[profiles.dev.env]
RESUME_DIR = "/shared/$USER/runs/my-run"
MODEL_DIR = "$HOME/models"

[profiles.dev.cache]
dir = "/cluster/shared/dev-hpc-compose-cache"

[resource_profiles.cpu-small]
time = "00:30:00"
cpus_per_task = 4
mem = "16G"

[resource_profiles.gpu-small]
partition = "gpu"
time = "01:00:00"
gpus = 1
cpus_per_task = 8
mem = "32G"

Resolution precedence is fixed:

1. CLI flags
2. selected profile values
3. shared settings defaults
4. built-in CLI defaults

Use context whenever you want to inspect effective compose path, binaries, interpolation variables, runtime paths, and per-field sources.

Resource profiles are referenced from YAML with x-slurm.resources: gpu-small. They are Slurm resource defaults, not the same thing as the global --profile setting selector, and explicit x-slurm values in the spec override profile defaults.

login_host is the SSH login host. It is the default SSH destination for hpc-compose up --remote when you do not pass --remote=<host>, and it also names the host shown in notebook, reach, and pull connection/tunnel hints and in the machine-readable hpc-compose notebook --format json output. A profile’s login_host overrides the shared default.

login_user is the SSH username applied to a bare login host, so the resolved destination becomes user@host. A profile’s login_user overrides the shared default. The login user for up --remote is resolved with this precedence: an explicit user@ already present in --remote=<dest> or login_host wins; then the HPC_COMPOSE_REMOTE_USER environment variable; then settings login_user (profile over defaults); then the User from your ~/.ssh/config. Persist both values with hpc-compose setup --profile-name <name> --login-host <host> --login-user <user> (written into [profiles.<name>]) or edit settings.toml directly.

If the login node requires an OTP/2FA on every SSH session, use SSH connection multiplexing (ControlMaster/ControlPersist) so you authenticate once and reused tunnels skip the prompt — see Run a Notebook or IDE Session.

An editor schema for settings.toml is available:

hpc-compose schema --kind settings

For TOML editor integration, write that schema to a file (hpc-compose schema --kind settings > hpc-compose-settings.schema.json) and point your TOML language server at the local path.

Choose A Starting Example

The maintained selection guide is Examples. It includes:

• four promoted beginner paths,
• a novice ladder from authoring to distributed workloads,
• the full repository example matrix,
• companion notes for LLM worker examples,
• an adaptation checklist.

Keep docs/src/examples.md as the single source of example selection truth. The embedded YAML source appendix is Example Source.

1. Choose A Cache Directory Early

Set the cache default to a path visible from both the login node and compute nodes:

[profiles.dev.cache]
dir = "/cluster/shared/hpc-compose-cache"

Or set x-slurm.cache_dir directly in the spec when the cache path should travel with that file:

x-slurm:
  cache_dir: /cluster/shared/hpc-compose-cache

Quick recipe:

export CACHE_DIR=/cluster/shared/hpc-compose-cache
mkdir -p "$CACHE_DIR"
test -w "$CACHE_DIR"

Rules:

• Do not use /tmp, /var/tmp, /private/tmp, or /dev/shm.
• If cache_dir is unset in the spec, resolution checks profile cache settings, then defaults cache settings, then $HOME/.cache/hpc-compose.
• The default may work on some clusters, but a shared project/work/scratch path is safer.
• Validation can accept unsafe local paths; preflight reports them as policy errors.

More cache details are in Cache Management.

2. Adapt The Example

Start with the nearest example and then change:

• image
• command / entrypoint
• volumes
• environment
• x-slurm resource settings
• x-runtime.prepare commands for dependencies or tooling

Recommended pattern:

• Put fast-changing application code in volumes.
• Put slower-changing dependency installation in x-runtime.prepare.commands.
• Add readiness only to services that other services truly depend on.

3. Validate The Spec

hpc-compose validate -f compose.yaml
hpc-compose validate -f compose.yaml --strict-env

Use validate first when changing field names, dependency shape, command/entrypoint form, paths, x-slurm, x-runtime, or compatibility x-enroot blocks.

If validate fails, fix that before doing anything more expensive. Use --strict-env when missing interpolation variables should fail instead of consuming ${VAR:-default} or ${VAR-default} fallbacks.

4. Plan The Run

hpc-compose plan -f compose.yaml
hpc-compose plan --verbose -f compose.yaml
hpc-compose plan --show-script -f compose.yaml

Check:

• service order,
• allocation geometry and service step geometry,
• normalized image references,
• host-to-container mount mappings,
• resolved environment values,
• runtime artifact paths,
• cache hit/miss expectations.

plan is purely static: it parses, validates, builds the normalized runtime plan, and can print the generated script to stdout, but it does not run preflight, prepare images, call sbatch, or write hpc-compose.sbatch. Add --explain for planner hints about cache paths, missing artifacts, resume/artifact settings, and the next command. plan --verbose can print secrets from resolved environment values.

5. Normal Run: Use up

hpc-compose up -f compose.yaml

up is the preferred end-to-end cluster flow. It runs preflight unless disabled, prepares images unless skipped, renders the script, calls sbatch, records tracked job metadata, polls scheduler state, and streams logs. It also uses a spec-scoped lock under .hpc-compose/locks/ so two concurrent up invocations against the same compose file do not race through prepare/render/submit.

Useful options:

• --script-out path/to/job.sbatch keeps a copy of the rendered script.
• --force-rebuild refreshes imported and prepared artifacts.
• --skip-prepare reuses existing prepared artifacts.
• --no-preflight skips the preflight phase.
• --detach submits or launches, records tracking metadata, and returns without watching.
• --format text|json is accepted with --detach or --dry-run.
• --watch-queue waits in line-oriented queue output until the Slurm job reaches RUNNING, then opens the normal watch view.
• --queue-warn-after <DURATION> warns once when --watch-queue stays PENDING longer than the threshold; the default is 10m, and 0 disables the warning.
• --watch-mode auto|tui|line selects the live output mode.
• --hold-on-exit never|failure|always controls whether the TUI stays open after the job reaches a terminal scheduler state.
• --resume-diff-only prints resume-sensitive config diffs without launching.
• --allow-resume-changes confirms intentional resume-coupled config drift.

up --local is Linux + Pyxis-only and single-host. See Runtime Backends.

Array jobs should be submitted with up --detach; use SLURM_ARRAY_TASK_ID in the service command and output patterns such as %A_%a for task-specific logs. Scheduler dependencies declared with x-slurm.after_job or x-slurm.dependency are passed to sbatch --dependency=... at submit time. Arrays and scheduler dependencies are not supported by up --local.

For conditional submission on a busy partition, use when:

hpc-compose when -f compose.yaml --partition gpu8 --free-nodes 4 --poll-interval 120s
hpc-compose when -f compose.yaml --after-job 12345
hpc-compose when -f compose.yaml --between 22:00-06:00

when is a foreground monitor. Interrupt it with Ctrl-C to stop waiting before the job is submitted. It runs preflight, image preparation, and script rendering before the wait begins, so submission is immediate once the conditions match; use --skip-prepare only when the required runtime artifacts already exist. --detach applies after submission: it still waits in the foreground for conditions, then returns after tracking metadata is written instead of opening the watch view.

Idle-node checks are advisory, not reservations. Another user can still submit first, and Slurm may queue the job after when calls sbatch. Keep polling gentle on shared login nodes: the default --poll-interval is 60s (minimum 5s); reserve very short intervals for brief, intentional watches.

For interactive development inside one allocation, use alloc:

hpc-compose alloc -f compose.yaml
hpc-compose run app -- python -m pytest

Inside the allocation shell, run SERVICE -- CMD reuses the active allocation with srun instead of submitting a new sbatch job. alloc exports HPC_COMPOSE_* metadata for the compose file, cache directory, runtime backend, and allocated nodes. For interactive notebook sessions inside an allocation, see Notebook.

5b. Submit From Your Laptop With up --remote

hpc-compose up runs on a Linux Slurm login node. macOS (and any host without Slurm) is authoring-only, so to submit from a laptop, delegate the run to a login node over SSH:

# Uses the configured login_host (with login_user, if set, as user@host):
hpc-compose up --remote -f compose.yaml
# Or target a specific host or ~/.ssh/config alias:
hpc-compose up --remote=login01 -f compose.yaml
# Or pass the SSH user inline:
hpc-compose up --remote=alice@login01 -f compose.yaml

The SSH destination comes from --remote=<dest> when given, otherwise from login_host. The login user follows the precedence documented in Project-Local Settings: an inline user@ wins, then HPC_COMPOSE_REMOTE_USER, then settings login_user (profile over defaults), then your ~/.ssh/config User.

What --remote stages

--remote rsyncs the compose project to a per-project staging directory on the login node (~/.hpc-compose-remote/<project>), including project settings such as .hpc-compose/settings.toml and .hpc-compose/cluster.toml while excluding tracked job/runtime state. It then runs hpc-compose up there over SSH, streaming the output back and propagating the remote exit code. Behavioral up flags such as --detach, --dry-run, --no-preflight, --skip-prepare, --force-rebuild, --allow-resume-changes, --resume-diff-only, --format, --print-endpoints, --watch-mode line, and --hold-on-exit are forwarded; without --detach the default remote run streams in line mode.

The staged root is the settings base: the directory that contains .hpc-compose/settings.toml. Place that file (or run hpc-compose setup) at the repo root so your whole source tree is staged. If your compose file lives in a subdirectory (for example hpc/haicore/compose.yaml) and there is no repo-root settings file, only that subdirectory is staged and the rest of your source tree is hidden from the job; hpc-compose prints a warning when it stages only a subdir.

--remote stages your repo only. It does not allocate cluster workspaces (for example ws_allocate) or create site storage directories — provision those yourself first, or a missing host bind-mount path blocks preflight. See Repo staging vs cluster workspace provisioning.

Auto-installing hpc-compose on the login node

Before the (potentially expensive) rsync, up --remote probes the login node for hpc-compose — on PATH or in ~/.local/bin — and reads its version. If the remote binary is missing or older than your local version, up --remote downloads and installs the newest release into ~/.local/bin with the official installer (curl -fsSL https://raw.githubusercontent.com/NicolasSchuler/hpc-compose/main/install.sh | sh), reusing the same multiplexed SSH connection so an OTP login node prompts only once. No root is needed, and the release tarball is checksum-verified. The delegated command runs the resolved absolute binary path, so an install in ~/.local/bin that is not on the non-interactive SSH PATH still works.

Control this with --remote-install <auto|never|force> (default auto) or the HPC_COMPOSE_REMOTE_INSTALL environment variable:

• auto (default): install only when the remote binary is missing or older than your local version.
• force: always reinstall the newest release before delegating.
• never: only probe. If the remote binary is missing or old, fail with an actionable error that prints the manual install command. Use this on locked-down or air-gapped login nodes.

If the install fails (for example, the login node has no outbound network), hpc-compose prints the manual install one-liner and a clear error. Set HPC_COMPOSE_REMOTE_INSTALL_URL to point the installer at a mirror.

Connection details and first run

Connection details belong in your ~/.ssh/config (port, identity, jump host), so --remote=<host> stays a bare host or alias. For an ad-hoc host not in your config, set HPC_COMPOSE_REMOTE_SSH_OPTS (whitespace-split ssh flags, e.g. -p 2222 -i ~/.ssh/cluster). Every connection reuses one SSH ControlMaster, so a login node that requires an OTP/2FA prompts only once within ControlPersist.

On the first remote run (or after cache eviction) the login-node prepare step imports your image with enroot — a multi-GB download plus extract and squashfs build — which can take several minutes; later runs reuse the cache. See Prepare Images Separately When Needed.

This is a thin delegation: it re-stages the project on each run and does not maintain a persistent login session. It is not up --local (that launches on the current host); --remote and --local cannot be combined.

Inspect a remote run from your laptop

The follow-up commands take the same --remote flag, so the metrics/logs workflow stays laptop-native — you don’t have to SSH into the staged checkout or know its internal paths. After a successful up --remote, hpc-compose prints the exact commands to run (fill in the Slurm job id it reported):

hpc-compose stats --remote=alice@login01 -f compose.yaml --job-id <job-id>   # GPU util / memory / power
hpc-compose logs  --remote=alice@login01 -f compose.yaml --job-id <job-id>
hpc-compose score --remote=alice@login01 -f compose.yaml --job-id <job-id>
hpc-compose pull  --remote=alice@login01 -f compose.yaml --job-id <job-id>

These reuse the same host/login-user/staging context as up --remote: they SSH into the existing remote stage (no re-sync) and stream the output back, reusing the same SSH ControlMaster so an OTP node still prompts only once. They require the project to have been staged by a prior up --remote. pull --remote prints the same rsync command from the login-node context; run that printed command from your laptop to copy the artifact bundle locally.

6. Run Preflight When Debugging Cluster Readiness

hpc-compose preflight -f compose.yaml
hpc-compose preflight --verbose -f compose.yaml
hpc-compose preflight -f compose.yaml --strict

preflight checks selected-backend tools, Slurm tools, cache path policy, local mounts/images, registry credentials, cluster profile compatibility, distributed-readiness hazards, metrics collector tools, and resume path safety.

Generate a cluster capability profile on the target login node when you want validation and preflight to catch partition/backend/QOS/GPU/MPI mismatches earlier:

hpc-compose doctor cluster-report

See Cluster Profiles for generated profile details, site policy packs, and MPI smoke probes.

7. Prepare Images Separately When Needed

hpc-compose prepare -f compose.yaml
hpc-compose prepare -f compose.yaml --force-rebuild

Use this when you want to build or refresh prepared images before submission, confirm cache reuse behavior, or debug preparation separately from job submission.

prepare needs the selected runtime backend tools, but it does not call sbatch.

8. Render The Batch Script

hpc-compose render -f compose.yaml --output /tmp/job.sbatch

This is useful when debugging generated srun arguments, mounts, environment passing, launch order, and readiness waits.

9. Inspect A Tracked Run

hpc-compose jobs list
hpc-compose status -f compose.yaml
hpc-compose status -f compose.yaml --array
hpc-compose ps -f compose.yaml
hpc-compose watch -f compose.yaml
hpc-compose replay -f compose.yaml --speed 10
hpc-compose logs -f compose.yaml --service app --follow
hpc-compose stats -f compose.yaml --format jsonl

Use Runtime Observability for tracked state, replay, logs, metrics, and machine-readable output. For a failed run, start with the First Triage flow in Troubleshooting. Use Artifacts and Resume for artifact bundles and resume-aware attempts.

10. Manage Cache And Old State

Cache Management owns cache inspection, pruning, and cleanup of old tracked runs (cache prune, jobs list --disk-usage, clean --age). For first triage of a failed run, see Troubleshooting.

What Changed And What Should I Run?

Related Docs

• Monitor a Run
• Manage the Cache and Clean Up
• Troubleshoot a Failed Run
• Develop and Smoke-Test Locally
• Onboard a Cluster Site
• Notebook

Monitor a Run

Status, watch, logs.

After a real submission, hpc-compose writes per-job runtime artifacts under:

<runtime-root>/<job-id>/

<runtime-root> defaults to <submit-dir>/.hpc-compose and can be overridden with x-slurm.runtime_root. The tracked submission record lives next to the compose file under .hpc-compose/jobs/<job-id>.json, and together those paths let follow-up commands reconnect without resubmitting.

Common Commands

hpc-compose status -f compose.yaml
hpc-compose ps -f compose.yaml
hpc-compose watch -f compose.yaml
hpc-compose watch -f compose.yaml --hold-on-exit always
hpc-compose replay -f compose.yaml --speed 10
hpc-compose watch -f compose.yaml --watch-mode line
hpc-compose logs -f compose.yaml --follow
hpc-compose logs -f compose.yaml --grep 'error|oom' --since 30m
hpc-compose stats -f compose.yaml
hpc-compose stats -f compose.yaml --accounting
hpc-compose inspect -f compose.yaml --rightsize
hpc-compose score 12345
hpc-compose germinate -f compose.yaml
hpc-compose sweep status -f compose.yaml
hpc-compose sweep list -f compose.yaml
hpc-compose diff 12345 12346 -f compose.yaml

Use --format json on non-streaming commands when automation needs stable fields. stats also supports --format csv and --format jsonl.

Watch UI

On an interactive terminal, watch and the default up follow mode open a live view with service state on the left and log output on the right. The UI automatically switches to a compact single-column view on narrow or short terminals. It keeps a detailed status view while the job runs and, by default, holds the final screen on failures so the failing service, final scheduler state, and next diagnostic commands stay visible.

Keybindings:

Log lines are colored by inferred severity: lines mentioning error/fatal/panic show in red and warn/warning in yellow (subject to the active color policy).

Use --hold-on-exit never|failure|always on up or watch to control whether the final TUI stays open after a terminal scheduler state. When the view is held, press d, l, or s to print the exact debug, logs, or stats command after leaving the alternate screen.

The r restart action writes a request consumed by the local Pyxis/Enroot supervisor, the same mechanism hpc-compose dev uses for file-watch reloads; it applies to local supervised jobs and is reported as unavailable for Slurm batch jobs. Run hpc-compose dev --tui to get this live view during a dev session: file-watching keeps reloading changed services in the background while the watch UI (including r for an on-demand restart) runs in the foreground. Without --tui, dev keeps its line-oriented output, which is friendlier for CI and logs.

The watch and replay views repaint only the rows that change between refreshes, which keeps the display flicker-free and minimizes bytes sent over SSH. Two environment variables tune the live view:

These display preferences can also be set per-project in .hpc-compose/settings.toml under a [watch] section; environment variables take precedence over the file:

[watch]
sort = "triage"          # spec | triage
wrap = true
refresh_ms = 500         # 100–60000
metrics_refresh_ms = 2000 # 500–600000
mouse = false

Use hpc-compose up --watch-queue when you want explicit queue polling before the watch view opens. It prints queue state changes, pending reason, and expected start time when Slurm exposes them; --queue-warn-after <DURATION> controls the one-time long-pending warning.

Use --watch-mode line when you are recording output, using a screen reader, running in CI, or working in a terminal where alternate-screen UIs are inconvenient. Line mode preserves detailed scheduler and log updates without alternate-screen control codes.

Replay

hpc-compose replay reconstructs a best-effort execution timeline after the run. It reuses the watch-style view, but reads only artifacts that already exist under the tracked job directory. This makes it useful for rewinding to the time a service failed, comparing the nearest prior metrics sample, or sharing a deterministic text/JSON summary without querying Slurm again.

hpc-compose replay -f compose.yaml
hpc-compose replay -f compose.yaml --speed 10
hpc-compose replay -f compose.yaml --job-id 12345 --service trainer
hpc-compose replay -f compose.yaml --format json

Replay controls:

A timeline scrubber under the header shows the playback cursor and reconstructed event ticks between the start and end of the run.

Replay data sources:

Use --format json when notebooks, dashboards, or experiment records need the reconstructed events, frame summaries, artifact paths, and fidelity notes.

Checkpoints

hpc-compose checkpoints reports the attempt and requeue history of a tracked job from LOCAL tracked state only. It contacts no scheduler and reads nothing from the cluster filesystem, so it is safe to run from a laptop against a synced tracked directory.

hpc-compose checkpoints -f compose.yaml
hpc-compose checkpoints --job-id 12345
hpc-compose checkpoints --format json

The history derives from the per-attempt state.json files written under .hpc-compose/<job>/attempts/<n>/. These per-attempt directories are produced only when x-slurm.resume is configured and the job is requeued: each requeue records a new 0-based attempt index (attempts = highest index + 1, requeues = attempts - 1). A non-resume job has no attempts/ directory and writes a single top-level state.json, which checkpoints reports as one attempt with zero requeues and no per-attempt index.

For each attempt, the command reports the earliest service start, the latest service finish, the derived duration, the job status, and the job exit code. A missing or unreadable per-attempt state.json is skipped and surfaced under degraded[] rather than failing the command, and a gap in the 0-based attempt indices (for example, an early attempt reaped by retention) is flagged as a truncated history so requeue counts are not silently miscounted.

--format json emits one object: {job_id, compose_file, submitted_at, resume_configured, attempts, requeues, current_attempt, is_resume, resume_dir, entries[], degraded[]}. This is distinct from the artifacts --bundle checkpoints export, which copies model checkpoint files rather than describing attempt history. See Artifacts and Resume for the attempt directory layout.

Logs

Runtime logs live under:

<runtime-root>/<job-id>/logs/<service>.log

Unless x-slurm.output is set, real submissions also write the top-level batch log under <runtime-root>/logs/hpc-compose-<job-id>.out. Check the batch log first when a job fails before any service log appears.

Service names containing non-alphanumeric characters are encoded in log filenames. Prefer [a-zA-Z0-9_-] in service names for readability.

Each service log is bracketed by timestamped lifecycle markers so a run does not look stuck before it produces output. A [hpc-compose] <ts> service <name>: container starting via srun … line is written just before the container launch (which is where srun scheduling and the first-use image extract happen), and a [hpc-compose] <ts> service <name>: command exited rc=<code> line is written when the command finishes. The gap between the start marker and the command’s own first line is the container-launch time, not a hang.

Use --grep <pattern> to print only matching raw log lines across selected service logs. Use --since <duration> for coarse time-bounded initial output, for example 30s, 15m, 2h, 1d, or 1h30m. Because service logs do not include line timestamps, --since filters by each log file’s modification time rather than by individual line time. Follow mode still starts from the current end of each selected log and applies --grep to appended lines.

Event Hooks

Per-service x-slurm.hooks can run host-side observability scripts when restart_on_failure accepts a restart or when the rolling restart window blocks a crash loop. Hook stdout/stderr is appended to that service’s log, and non-zero hook exits are logged without changing the restart or failure outcome.

Use on: restart for retry notifications and on: window_exhausted for crash-loop alerts. Event hooks receive service identity, exit code, Slurm attempt, and restart-window counters through HPC_COMPOSE_* environment variables; see Spec reference for the full list.

Metrics

When x-slurm.metrics is enabled, sampler files are written under:

<runtime-root>/<job-id>/metrics/
  meta.json
  gpu.jsonl
  gpu_processes.jsonl
  slurm.jsonl
  diagnostics/

The sampler can collect GPU snapshots through nvidia-smi and job-step CPU/memory snapshots through sstat. Collector failures are best-effort: missing nvidia-smi, missing sstat, or unsupported queries do not fail the batch job itself.

Add --accounting to stats when you need post-run sacct rollups for reporting. The accounting summary includes allocated CPU-hours, total CPU-hours when available, allocated GPU-hours, allocation-based memory byte-seconds, and observed maximum RSS. Memory byte-seconds are labeled as allocation-based because Slurm’s standard accounting fields do not reliably provide true per-line memory-seconds across all clusters.

Use hpc-compose inspect --rightsize -f compose.yaml after a tracked Slurm run to convert those observations into conservative resource suggestions. The assistant requires tracked submission metadata and compares explicit requests such as x-slurm.mem, x-slurm.time, x-slurm.gpus, and service x-slurm.cpus_per_task against sacct, sstat, and nvidia-smi sampler evidence. It only reports suggestions; it does not rewrite the compose file.

Use hpc-compose score <job-id> after a tracked Slurm run when you want a compact efficiency grade. The score reuses sampler history, sacct, sstat, and right-sizing recommendations, then reports GPU utilization, memory utilization, active compute-time versus requested walltime, and a best-effort kWh estimate. Energy uses sampled GPU power when available, otherwise falls back to power limits or configured TDP assumptions through --gpu-tdp-w, --cpu-watts-per-core, and --pue; it does not claim carbon intensity or emissions.

Use hpc-compose experiment show <job-id> when you want all of that in one read-only object. A single call aggregates scheduler status, the post-run efficiency score, the artifact manifest, and submit-time provenance, so a notebook or experiment tracker can capture one run with one command (hpc-compose experiment show <job-id> --format json). It is static-safe: it contacts the scheduler only as much as status and score already do, writes nothing, and opens no connection. For each service with TCP or HTTP readiness it emits a per-service ssh -L tunnel hint, and next_commands carries SSH ControlMaster/ControlPath/ControlPersist multiplexing guidance so an OTP/2FA login node prompts you only once. Legacy records without provenance, non-terminal jobs without a complete efficiency report, and runs without an artifact manifest still produce a valid object with those fields omitted.

For a short canary run before a full run, use hpc-compose germinate; see Right-Size With Canary Runs.

Sweep Manifests

Sweep submission and monitoring (sweep submit, sweep status, sweep list) are covered in Hyperparameter Sweeps. Sweep-trial records do not replace normal latest.json or latest-run.json, so hpc-compose status, watch, and logs continue to target ordinary runs unless you pass an explicit job id.

Diffing Runs

Use hpc-compose diff <job-id-1> <job-id-2> to compare two tracked submissions. The compact text view highlights outcome, resource, and config changes; --format json returns the full uncapped diff for notebooks or experiment records. Older tracked jobs without config snapshots still compare outcome metadata and report a note that config comparison is unavailable.

N-Way Comparison Matrix

To compare more than two runs at once, drop the positional job ids and pass either --jobs a,b,c (an explicit comma-separated list of tracked job ids) or --across <sweep-id> (every submitted trial of a sweep; unsubmitted trials are skipped with a note). The result is a matrix with one column per run and one row per field that differs in at least one run — fields identical across every run are collapsed and omitted, so the output stays focused on what actually changed. The same outcome, provenance, resource, and config sections as the pairwise diff are projected across all runs.

Choose the output with --matrix-format text|csv|json (default text). --matrix-format csv emits a section,field,<job_id>... table for spreadsheets, while --matrix-format json serializes the full uncapped matrix (the text view caps the config section at 25 rows). This is a pure read-only projection over already-persisted records; like pairwise diff, it opens no connection and only probes the scheduler as much as status does.

hpc-compose diff --jobs 12345,12346,12347 --matrix-format json
hpc-compose diff --across sweep-1700000000-1234 --matrix-format csv

Related Docs

• Operate a Real Cluster Run
• Troubleshoot a Failed Run
• Manage the Cache and Clean Up
• Artifacts and Resume
• Hyperparameter Sweeps
• Right-Size With Canary Runs

Manage the Cache and Clean Up

The resolved cache directory stores imported and prepared runtime artifacts. It comes from explicit x-slurm.cache_dir, then profile/default settings, then $HOME/.cache/hpc-compose. For real cluster runs, it must be visible from both the submission host and compute nodes; see Execution Model for why prepared artifacts must live on shared storage.

Choose A Cache Path

Use a project scratch, work, or shared filesystem path:

export CACHE_DIR=/cluster/shared/hpc-compose-cache
mkdir -p "$CACHE_DIR"
test -w "$CACHE_DIR"

You can record that path in project settings instead of every compose file:

hpc-compose setup --profile-name dev --cache-dir "$CACHE_DIR" --default-profile dev --non-interactive

Do not use /tmp, /var/tmp, /private/tmp, or /dev/shm. Validation may accept those strings, but preflight reports them as unsafe because compute nodes must reuse artifacts prepared before submission.

Inspect Cache State

hpc-compose cache list
hpc-compose cache inspect -f compose.yaml
hpc-compose cache inspect -f compose.yaml --service app

Use cache inspect to answer:

• which artifact is being reused
• whether a prepared image came from a cached manifest
• whether a service rebuilds on every prepare because prepare mounts are present

Staged-Input Cache (Datasets/Models)

Staged datasets and models live in a content-addressed store under the same shared cache root, at cache_dir/datasets/<key> and cache_dir/models/<key>. The key is derived from the input spec (its source URI and pinned revision), so identical staged inputs are materialized once and reused on every later run. Each staged directory carries a sidecar manifest (<key>.dataset.json or <key>.model.json) so cache list and cache prune cover staged inputs alongside image artifacts.

The store itself never fetches anything: it is a pure on-disk store, and the actual fetch and materialization (network) is approval-gated and introduced by the hf:// stage-in work, not run automatically by cache, plan, or prepare.

Prune Cache Entries

Prune old entries by age:

hpc-compose --profile dev cache prune --age 14 --yes

Prune artifacts not referenced by the current plan:

hpc-compose cache prune --all-unused -f compose.yaml --yes

Prune one cache directory directly:

hpc-compose cache prune --age 7 --cache-dir '<shared-cache-dir>' --yes

--age and --all-unused are mutually exclusive.

Rendezvous Records

Cross-job rendezvous records live under the same shared cache root and are pruned separately (rendezvous list, rendezvous prune). See Cross-Job Rendezvous for placement, TTL, and ownership rules.

Clean Up Old Tracked Runs

Tracked job metadata and logs accumulate in .hpc-compose/. Preview disk usage and cleanup before deleting:

hpc-compose jobs list --disk-usage
hpc-compose clean -f compose.yaml --age 7 --dry-run
hpc-compose clean -f compose.yaml --age 7

After Upgrading

Cache keys include the tool version, so upgrading hpc-compose invalidates existing cached artifacts. Expect a full rebuild on the next prepare or up, then optionally prune old entries:

hpc-compose cache prune --age 0 --yes

Related Docs

• Operate a Real Cluster Run
• Monitor a Run
• Troubleshoot a Failed Run
• Execution Model
• Cross-Job Rendezvous
• CLI Reference

Troubleshoot a Failed Run

Use this page when the safe authoring path worked but the first real cluster run failed.

For background on Slurm allocations, sbatch, srun, Pyxis, and Enroot, see Slurm And Container Basics. For HAICORE-specific storage and runtime checks, see HAICORE Guide.

First Triage

hpc-compose validate -f compose.yaml
hpc-compose validate -f compose.yaml --strict-env
hpc-compose plan --verbose -f compose.yaml
hpc-compose lint --fix --dry-run -f compose.yaml
hpc-compose debug -f compose.yaml --preflight

plan --verbose can print resolved environment values and final mount mappings. Treat its output as sensitive when the spec contains secrets. validate and lint emit “Did you mean …” suggestions for misspelled service keys and dependency conditions. lint --fix --dry-run previews auto-fixes (for example, making an implicit depends_on condition explicit) without writing. debug is read-only unless --preflight is passed; with --preflight, it reruns prerequisite checks and includes those findings in the triage report.

Common Symptoms

Readiness Issues

Use depends_on with condition: service_healthy when a dependent must wait for a dependency’s readiness probe. Plain list form means service_started.

Use condition: service_completed_successfully for one-shot DAG stages where the next service should start only after the previous stage exits with status 0, such as preprocess -> train -> postprocess.

When a TCP port opens before the service is fully usable, prefer HTTP or log-based readiness over TCP readiness.

Inspect the normalized readiness probe without starting or submitting anything:

hpc-compose doctor readiness -f compose.yaml --service api

If the service is already running, tunneled, or otherwise reachable from the current host, run the same probe host-side:

hpc-compose doctor readiness -f compose.yaml --service api --run
hpc-compose doctor readiness -f compose.yaml --service api --run --log-file .hpc-compose/<job-id>/logs/api.log

doctor readiness --run does not launch services, prepare images, or call Slurm. It only checks the selected readiness target from the current host, which makes it useful before testing a dependent service or while debugging an already tracked run.

For hpc-compose test, readiness failures are terminal smoke-test failures. A service with configured readiness must become healthy and then complete successfully; ignored sidecars are still expected to pass in a smoke spec.

Preview A Run

Use plan for the static preview. It never prepares images, runs preflight, calls sbatch, or writes hpc-compose.sbatch:

hpc-compose plan --show-script -f compose.yaml

Use up --dry-run only when you intentionally want to exercise preflight, prepare, and render without calling sbatch:

hpc-compose up --dry-run -f compose.yaml

Clean Old Tracked Runs

Cleaning up accumulated tracked job metadata and logs is covered in Manage the Cache and Clean Up.

Related Docs

• Operate a Real Cluster Run
• Monitor a Run
• Manage the Cache and Clean Up
• Develop and Smoke-Test Locally
• Slurm And Container Basics
• HAICORE@KIT Guide

Develop and Smoke-Test Locally

test, dev, and tmux are the local-development command layer. They reuse the same prepare, render, local supervisor, runtime state, and tracking paths as up, so a run started by one command remains visible to status, ps, logs, stats, watch, and debug.

Smoke-Test Specs

Use test for finite specs that prove a workflow starts, satisfies readiness gates, and exits cleanly:

hpc-compose test --local -f examples/dev-python-smoke.yaml
hpc-compose test --submit --time 00:01:00 --timeout 180s -f compose.smoke.yaml
hpc-compose test --submit --format json -f compose.smoke.yaml

test requires exactly one execution mode:

• --local runs the rendered local supervisor on the current host.
• --submit calls sbatch; it defaults to --time 00:01:00 and --timeout 180s. This is a real-scheduler operation that consumes an allocation, so it needs explicit user approval before running.

A smoke test passes only when every service:

• appears in tracked runtime state,
• launched at least once,
• passed readiness when readiness is configured,
• completed successfully.

Services with failure_policy.mode: ignore still have to complete successfully for test to pass. That makes smoke tests stricter than production runs by design: ignored sidecars are useful operationally, but they should not silently hide a broken spec test.

Making Long-Running Specs Finite

Production services often run forever. For smoke tests, create a finite variant of the spec or override the service command in a copied file:

services:
  app:
    image: python:3.11-slim
    working_dir: /workspace
    volumes:
      - ./app:/workspace
    command:
      - python
      - -c
      - "import main; print('smoke ok', flush=True)"

Keep the same image, mounts, environment, dependencies, and readiness where possible. Change only the command or entrypoint needed to prove startup and exit. If a dependent service uses condition: service_healthy, keep the upstream readiness probe real enough to catch wiring mistakes.

Hot Reload

dev is local-only:

hpc-compose dev -f examples/dev-python-app.yaml
hpc-compose dev -f compose.yaml --watch-paths ./src --debounce-ms 500

It infers watch roots from host directories mounted through service volumes. File mounts, container-only paths, cache paths, missing paths, and non-directory paths are ignored. --watch-paths adds an explicit directory and restarts every service when it changes.

File changes write restart requests into the tracked run’s dev control directory. The local supervisor handles those requests as development restarts, so readiness and completion state reset for the affected service without consuming the restart counters used by failure_policy.mode: restart_on_failure (max_restarts/max_restarts_in_window).

By default, Ctrl-C stops the local supervisor. Add --keep-running when you want to leave the tracked local run alive after exiting the watch loop.

Tmux Dashboard

tmux is a log dashboard, not a process supervisor:

hpc-compose tmux -f compose.yaml
hpc-compose tmux -f compose.yaml --job-id local-123
hpc-compose tmux -f compose.yaml --session demo --no-attach

Without --job-id, it launches a new local run. With --job-id, it attaches to an existing tracked local run. Each pane tails one service log with tail -F, and pane titles use service names. Use --no-attach when running from a non-interactive terminal or CI smoke check.

Shared Local Constraints

up --local, test --local, dev, and tmux share the same current constraints:

• Linux hosts only
• runtime.backend: pyxis only
• Pyxis-compatible Enroot tooling on the host
• single-host specs only
• no distributed or partitioned placement
• no service-level MPI
• no Slurm arrays or scheduler dependencies

Use these commands to author and debug single-host launch behavior. Use test --submit or up on a Slurm login node for real scheduler behavior, or use the Local Slurm Dev Cluster from a source checkout when you want a throwaway real sbatch smoke test without a cluster login.

Example Recipe

The source-mounted app in examples/dev-python-app.yaml is intentionally long-running, so it is a good dev target:

hpc-compose dev -f examples/dev-python-app.yaml
hpc-compose tmux -f examples/dev-python-app.yaml --no-attach

The companion examples/dev-python-smoke.yaml keeps the same mounted source pattern but uses a finite command:

hpc-compose test --local -f examples/dev-python-smoke.yaml
hpc-compose test --submit --time 00:01:00 -f examples/dev-python-smoke.yaml

Related Docs

• Operate a Real Cluster Run
• Local Slurm Dev Cluster
• Monitor a Run
• Troubleshoot a Failed Run
• Manage the Cache and Clean Up
• CLI Reference

Local Slurm Dev Cluster

The local Slurm dev cluster is source-checkout tooling for running hpc-compose against a real throwaway scheduler on a laptop. It starts one privileged Docker/Podman container with slurmctld, slurmd, slurmdbd, MariaDB, and the current checkout’s hpc-compose binary.

Use it when you want a real scheduler smoke test before moving to a shared cluster. It is not a dry-run: scripts/devcluster.sh run ... calls real sbatch inside the local container. The job consumes only the local throwaway Slurm node.

Preview Levels

Use plan first for fast static feedback. Use up --dry-run when you want the same preflight and preparation path as submission but no sbatch. Use the dev cluster when you specifically want to exercise hpc-compose’s real up -> sbatch -> slurmd -> sacct path without a cluster login.

Requirements

• A source checkout of this repository. Release archives install the CLI and manpages, not the dev-cluster wrapper and Dockerfile.
• docker compose or podman compose, with the engine running.
• Support for privileged containers. The local node needs cgroup access for slurmd; treat it as a disposable developer machine workflow.

Quickstart

From the repository root:

scripts/devcluster.sh up
scripts/devcluster.sh sinfo
scripts/devcluster.sh run dev-cluster/specs/hello.yaml
scripts/devcluster.sh down

To smoke-test another project tree with the same local Slurm node:

scripts/devcluster.sh up --project /path/to/project
scripts/devcluster.sh run compose.yaml
scripts/devcluster.sh down

Specs run in the dev cluster should use runtime.backend: host. That keeps the local loop tractable and avoids nesting Pyxis/Enroot or Apptainer inside Docker/Podman. If your production spec uses a container backend, keep a small host-backend smoke variant for local scheduler validation and revalidate the container runtime on the real cluster.

Automated Check

Maintainers can run the checked-in real-scheduler suite with:

DEVCLUSTER_E2E_DOWN=1 scripts/devcluster_e2e.sh

The script boots the cluster, runs every spec under dev-cluster/specs, asserts that each spec has an explicit expected outcome, and verifies scheduler-backed commands such as status, ps, logs, and score where applicable. CI runs the same harness as a separate Dev Cluster E2E job with a cached image build.

Scope

Validated locally:

• sbatch submission against a real controller
• service ordering and readiness gates
• multi-service composition inside one allocation
• terminal accounting through sacct
• scheduler-facing observability for tracked runs
• expected failure propagation for negative smoke specs
• sbatch --array fan-out with per-task accounting and status --array
• the restart_on_failure supervisor draining to COMPLETED through real restarts
• cancel driving a running job to the CANCELLED terminal state, with tracked-state teardown
• artifact teardown collection resolved by pull/artifacts against a real manifest
• scheduler inter-job dependencies (after_job holds a consumer until the producer ends)
• failure_policy: ignore and depends_on: service_completed_successfully ordering
• tracked-state readers over a real run (experiment, replay, debug, checkpoints, jobs, clean)
• the host-backend resume dir resolving to a real on-node path
• alloc + run reusing one allocation via srun

Still validate on the real cluster:

• Pyxis/Enroot, Apptainer, or Singularity runtime behavior
• GPU execution
• site-specific modules, filesystems, partitions, and accounting policy
• multi-node network and placement behavior

Related Docs

• Quickstart
• Develop and Smoke-Test Locally
• Operate a Real Cluster Run
• Troubleshoot a Failed Run

Run a Notebook or IDE Session

hpc-compose notebook launches a tracked interactive server — JupyterLab or VS Code (code tunnel) — as a single-service Slurm job, waits for it to become ready, and prints the connection URL. The session is a normal tracked job: manage it with hpc-compose status and stop it with hpc-compose cancel.

Use it when you want an interactive IDE or notebook on a compute node (for example, on a GPU partition) without hand-writing sbatch glue.

Kinds

Quickstart

# JupyterLab on one GPU, with your project mounted
hpc-compose notebook --kind jupyter --gpus 1 \
  --volume ./project:/workspace --working-dir /workspace

# VS Code tunnel (supply an image containing the `code` CLI)
hpc-compose notebook --kind vscode --image ghcr.io/example/code:1 --gpus 1

After readiness, hpc-compose prints the URL. For Jupyter on Slurm it also prints a ready-to-copy SSH command:

Open: http://127.0.0.1:8888/lab?token=<generated>

On your laptop, forward the port:
  ssh -N -o ControlMaster=auto -o ControlPath=~/.ssh/cm-%r@%h:%p -o ControlPersist=10m -L 8888:<compute-node>:8888 <login-node>
then open the URL above in your browser.
The ControlMaster options reuse one authenticated connection, so a login node that requires an OTP/2FA only prompts on the first connection within ControlPersist.

The printed command already carries the SSH connection-multiplexing options (the same ones reach, pull, and experiment emit), so a login node that requires an OTP/2FA prompts only on the first connection of your session.

For VS Code, open the printed vscode.dev link directly in a browser — no tunnel is required.

Login nodes that require an OTP / 2FA

If your login node demands a one-time password on every SSH session, keep SSH connection multiplexing enabled so you authenticate once and every later tunnel (and rsync/scp) reuses the master connection. The printed Jupyter tunnel command already includes these options; the equivalent persistent ~/.ssh/config form is:

# ~/.ssh/config
Host <login-node>
    ControlMaster auto
    ControlPath ~/.ssh/cm-%r@%h:%p
    ControlPersist 10m

Establish the master once (entering the OTP), then the forward runs without re-authenticating until ControlPersist expires:

ssh -fN <login-node>                          # OTP entered here, once
ssh -L 8888:<compute-node>:8888 <login-node>  # reuses the master — no OTP

hpc-compose only prints the tunnel command; it never opens a connection or stores credentials, so the OTP step stays entirely under your control.

Local mode

--local runs the server on the current host (login node or workstation) through the same local supervisor used by dev. The printed URL points at 127.0.0.1 directly:

hpc-compose notebook --kind jupyter --local --volume ./src:/workspace

Local mode requires a Linux host with Pyxis-compatible Enroot tooling, like the rest of the local-development command layer.

Managing the session

The notebook is a tracked job, so the standard commands work:

hpc-compose status -f <compose>          # scheduler + service state
hpc-compose logs -f <compose> --follow   # tail the notebook log
hpc-compose cancel -f <compose>          # stop and release the allocation

By default notebook detaches after printing the URL (the job keeps running). Pass --follow to stream logs in the foreground instead.

Security

For Jupyter, hpc-compose generates a random auth token and embeds it in the printed URL, so the link is unguessable but self-contained. Override it with --token if you prefer. Do not share the printed URL: it grants access to the notebook session.

For VS Code, code tunnel performs GitHub device-flow authentication the first time; --accept-server-license-terms is passed automatically.

Authoring notes

• Images and users. jupyter/scipy-notebook runs as the non-root jovyan user. Bind-mounted host directories must be writable by that user (typically uid 1000). Use --working-dir to point at your mounted workspace and adjust ownership on the host if needed.
• VS Code images. There is no universal default code image; supply one with --image that contains the VS Code CLI.
• Readiness. hpc-compose waits for a log pattern (/lab?token= for Jupyter, vscode.dev/tunnel/ for VS Code) before printing the URL. Use --ready-timeout (default 10m) to bound the wait; first-run image pulls happen during prepare, before the readiness clock starts.
• Declarative counterpart. The same workflow is available as a compose file via the jupyter template (hpc-compose new --template jupyter), so you can commit it to a repo and launch with hpc-compose up.

Related Docs

• Use Secrets
• Development Workflow
• Examples
• CLI Reference

Use Secrets

hpc-compose resolves named secrets from local files or environment variables and feeds them into the interpolation map as first-class, redacted values. This keeps secrets out of the rendered batch script’s environment: block authoring surface and ensures they are hidden in config/context/inspect output.

Declaring secrets

Add a top-level secrets: block mapping a secret name to exactly one source:

secrets:
  hf_token:
    file: ./secrets/hf.txt       # value = file contents (trimmed)
  db_password:
    env: DB_PASSWORD             # value = named environment variable

• file: reads the value from a file relative to the compose file directory.
• env: reads the value from the named variable in the resolved environment (process env, .env, or settings env_files).
• Each secret must set exactly one of file or env.

Using secrets

Reference a secret anywhere interpolation works — most commonly in a service environment: block:

services:
  trainer:
    image: pytorch/pytorch:2.12.1-cuda13.2-cudnn9-runtime
    environment:
      HF_TOKEN: ${hf_token}
      DB_PASSWORD: ${db_password}
    command: python -m train

The resolved value flows through the normal environment: render path (--container-env= + the launcher env array) into the container. No new mount machinery is required.

Redaction

A value resolved through secrets: is tagged as a secret source. It is always redacted in diagnostic output regardless of its name:

$ hpc-compose config -f compose.yaml
...
    environment:
      HF_TOKEN: <redacted>
      MODEL: llama

Name-based redaction also still applies to any sensitive-named value, even when it was not declared as a secret. A name triggers redaction when (after upper-casing) it contains any of these substrings:

SECRET   TOKEN     PASSWORD   PASSWD
API_KEY  ACCESS_KEY  PRIVATE_KEY  CREDENTIAL
AUTH     COOKIE    SESSION    BEARER

Matching is a case-insensitive substring test, so names such as SESSION_DIR, AUTH_MODE, or MY_API_KEY_PATH are redacted too. Pass --show-values on config or context to reveal secrets when you have a legitimate need:

hpc-compose config -f compose.yaml --show-values
hpc-compose context   # table of interpolation vars; secret-sourced values show SOURCE 'secret' and are redacted

The raw secret value never appears in config, context, or inspect output by default. inspect does not expose a --show-values escape hatch; use config --show-values or context --show-values for trusted local diagnostics.

Secrets at rest

Redaction only governs diagnostic output. The rendered Slurm script and the persisted job state can carry resolved secret values, because the environment is materialized so the job can run. These files are written owner-only (mode 0600); even so, keep secret-bearing compose specs, rendered scripts, and tracked state under a non-group-readable directory (for example a private cache or scratch path), and avoid committing them to shared or version-controlled locations.

Resolution order

Secrets are resolved after process environment variables and declared with the secret source. Declaring a secret is authoritative for its name; an explicit declaration overrides a same-named variable from a lower-precedence source. For env: sources, the named variable is read from the full resolved environment (including .env and settings env_files).

What is not included

hpc-compose ships local file: and env: sources only. Backend integrations (HashiCorp Vault, AWS Secrets Manager, GCP Secret Manager) are intentionally deferred — they would require either shelling out to the vault/gcloud CLIs or adding a client crate, which conflicts with the project’s minimal-dependency stance. You can bridge to them today by writing the fetched value into a file or exporting it as an environment variable, then referencing it through secrets:.

File-mount injection to /run/secrets/<name> (Docker Compose semantics) is also deferred; env-var injection through environment: covers the common case.

Related Docs

• Run a Notebook or IDE Session
• Wire Up CI
• Spec Reference
• Troubleshooting

Run Hyperparameter Sweeps

hpc-compose sweep turns one compose file with an embedded sweep block into many independent tracked Slurm jobs. Each trial is a normal sbatch submission with its own allocation, rendered script, job record, and scheduler state. The sweep manifest ties those jobs together for listing and aggregate status.

Quickstart

Start from a spec that can run with ordinary defaults, then add a top-level sweep block:

name: training-sweep

x-slurm:
  time: "00:20:00"
  cache_dir: ${CACHE_DIR:-/cluster/shared/hpc-compose-cache}

sweep:
  parameters:
    lr: [0.001, 0.01, 0.1]
    batch_size: [32, 64]
  matrix: full

services:
  trainer:
    image: python:3.11-slim
    environment:
      LR: "${lr:-0.001}"
      BATCH_SIZE: "${batch_size:-32}"
    command: ["python", "train.py"]

Preview the expansion first:

hpc-compose sweep submit -f examples/training-sweep.yaml --dry-run

Then submit the trials:

hpc-compose sweep submit -f examples/training-sweep.yaml
hpc-compose sweep status -f examples/training-sweep.yaml
hpc-compose sweep list -f examples/training-sweep.yaml

Matrix Modes

matrix: full expands the full Cartesian product over sorted parameter names, so the example above produces six trials in stable t000, t001, … order.

Random sampling selects without replacement:

sweep:
  parameters:
    lr: [0.001, 0.01, 0.1]
    batch_size: [32, 64]
  matrix:
    random: 5
    seed: "paper-table-2"

With a seed, the selected trials are stable across machines. Without a seed, sweep submit derives one from the new sweep id and persists it in the manifest.

Interpolation Rules

Sweep parameter names are interpolation variable names. Values may be scalar strings, numbers, or booleans. For each trial, those variables override values from the environment and settings before planning, preparing, and rendering.

Reserved variables are also available:

Normal commands still treat sweep as metadata. If plan, up, or render encounters ${lr} without a default, it fails unless lr is provided in the environment or settings. Use defaults such as ${lr:-0.001} when the base spec should remain runnable, and use sweep submit --dry-run as the validation path for missing sweep-only variables.

Replicates

Set replicates: N to submit N seeded trials per parameter config. This is sweep sugar for repeating each combination so noise can be averaged out:

sweep:
  parameters:
    lr: [0.001, 0.01]
  matrix: full
  replicates: 3
  objective:
    direction: minimize
    log_pattern: 'final loss=([0-9.]+)'

With replicates: 1 (the default) the expansion is byte-identical to a non-replicated sweep: trial ids stay t000, t001, … and no replicate seed is injected. With replicates > 1 each config c fans out into t{c:03}r0 … t{c:03}r{N-1} (for example t000r0, t000r1, t000r2), each its own Slurm allocation. The example above submits 2 configs × 3 replicates = 6 trials.

Each replicate gets a deterministic seed exposed as HPC_COMPOSE_SWEEP_SEED, derived as the hex SHA-256 digest of <sweep_id>:<config_key>:<replicate> (where config_key is the name=value;… join of the config’s sorted variables). Re-expanding the same sweep block with the same sweep id always reproduces the same seed, so a training script can feed HPC_COMPOSE_SWEEP_SEED to its RNG and recover the same run. HPC_COMPOSE_SWEEP_REPLICATE carries the zero-based replicate index.

sweep status, sweep observe, and sweep results group the trials of each config and report a mean±std(n) rollup (population standard deviation, so n=1 reports std=0). Crucially, best_trial ranks on the per-config group mean, not the single luckiest replicate, and sweep observe reports the winning config’s mean objective:

replicate rollup (mean+/-std over n replicates per config):
  lr=0.001: mean=0.034000 std=0.002160 n=3 (3 replicate(s))
  lr=0.01:  mean=0.041000 std=0.001414 n=3 (3 replicate(s))
best config: t000r0 (mean objective=0.034)

The fanout guard below counts materialized runs (combinations × replicates), so a 40-config matrix with replicates: 3 is 120 runs and is rejected without --max-trials.

Fanout Guard

By default, submitted sweeps are capped at 100 trials. Larger matrices fail before calling sbatch:

hpc-compose sweep submit -f examples/training-sweep.yaml

Raise the explicit ceiling when the fanout is intentional:

hpc-compose sweep submit -f examples/training-sweep.yaml --max-trials 500

The guard applies to real submissions. Dry runs can inspect any matrix size.

Status Output

sweep status loads the manifest, queries the tracked state for submitted jobs, and aggregates:

• completed
• failed
• running
• pending
• unknown
• missing_tracking
• submit_failed

Use JSON for notebooks, dashboards, or CI automation:

hpc-compose sweep submit -f examples/training-sweep.yaml --format json
hpc-compose sweep status -f examples/training-sweep.yaml --format json
hpc-compose sweep status -f examples/training-sweep.yaml --sweep-id sweep-123 --format json
hpc-compose sweep list -f examples/training-sweep.yaml --format json

The JSON includes the sweep id, manifest path, matrix mode, persisted seed, trial variables, job ids, record paths, and per-trial status. When the sweep used replicates, it also carries a groups array with the per-config mean±std(n) rollup.

Manifest Layout

Sweep state is stored beside normal tracked jobs:

.hpc-compose/
  sweeps/
    latest.json
    <sweep-id>/
      sweep.json
      t000.sbatch
      t001.sbatch
  jobs/
    <job-id>.json

With replicates > 1 the per-trial scripts are named t000r0.sbatch, t000r1.sbatch, … (config index plus replicate index) instead of the flat t000.sbatch.

Sweep-trial records have kind: sweep_trial and include sweep metadata. They do not update the normal latest.json or latest-run.json pointers, so status, watch, and logs for ordinary runs keep their existing meaning.

Objectives and Early Termination

Declare an objective block to have sweep observe parse a metric from each terminal trial, rank trials, and record the best on the manifest:

sweep:
  parameters: { lr: [0.001, 0.01, 0.1] }
  matrix: full
  objective:
    direction: minimize
    log_pattern: 'final loss=([0-9.]+)'

The trial workload prints the metric to its service log (e.g. final loss=0.034). Two parse sources are supported (set exactly one):

• log_pattern: a regex against the trial’s primary service log; capture group group (default 1) is parsed as a float.
• json_path + json_field: read a JSON field from the trial’s artifact-collected tree.

hpc-compose sweep observe -f train.yaml             # parse + rank + print best
hpc-compose sweep observe -f train.yaml --format json

Early termination stops the sweep once a threshold is met. Use --watch --stop-when to poll and auto-stop:

hpc-compose sweep observe -f train.yaml --watch --stop-when 'objective < 0.05' --poll-interval 30s

Or stop manually after inspecting sweep observe output:

hpc-compose sweep stop -f train.yaml --yes --reason 'objective threshold met'

sweep stop cancels every non-terminal trial via scancel and records the stop on the manifest. --stop-when uses a tiny grammar: objective < N, objective <= N, objective > N, or objective >= N, evaluated against the best observed value.

Bayesian/adaptive trial selection is intentionally out of scope. The objective writeback, ranking, and stop machinery here are the foundation any future optimizer would build on.

Scaling Reports

Set objective.scaling_axis to the name of a numeric sweep parameter (for example nodes or model_size) to enable a post-hoc scaling report:

sweep:
  parameters:
    nodes: [1, 2, 4, 8]
  matrix: full
  objective:
    direction: minimize
    log_pattern: 'final loss=([0-9.]+)'
    scaling_axis: nodes

scaling_axis must name a key under sweep.parameters, and every value of that parameter must parse as a number. Both are checked at validate time (including sweep submit --dry-run), so a typo or a non-numeric axis is rejected with a clear message before anything is submitted.

Run sweep observe --scaling to print the report alongside the usual ranked table:

hpc-compose sweep observe -f train.yaml --scaling
hpc-compose sweep observe -f train.yaml --scaling --format json

The report pairs each config group’s mean objective with its axis value, reports a log-log least-squares slope (ln(objective) vs ln(axis)), and computes speedup/efficiency relative to a baseline group:

scaling (minimize objective vs nodes):
  baseline nodes=1
  nodes=1 mean=0.800000 runtime=100s speedup=1.000x efficiency=100.0% (n=1)
  nodes=2 mean=0.400000 runtime=50s speedup=2.000x efficiency=100.0% (n=1)
  nodes=4 mean=0.200000 runtime=25s speedup=4.000x efficiency=100.0% (n=1)
  log-log slope (objective vs nodes): -1.0000

The report is purely read-only, post-hoc analysis over the persisted manifest and tracked local state: it reuses the same terminal-only scheduler/runtime probe as sweep observe and never opens a new connection. Runtime is taken from the maximum observed service duration of each terminal trial; trials that are non-terminal or report no runtime are skipped rather than zero-filled. The baseline is the smallest-axis group that has runtime data. The report is print/JSON-only and is never written back to the manifest, so omitting --scaling leaves observe output byte-identical.

Limitations

• Sweeps must be embedded in the same compose file. sweep.spec is not supported.
• Each trial is a separate Slurm allocation. Sweeps are not Slurm arrays.
• x-slurm.array is rejected during sweep submit.
• Trials submit sequentially. If a submission fails, later trials are not submitted and the partial manifest is kept.
• sweep status summarizes scheduler/tracking state; use sweep observe to parse and rank objectives.

Related Docs

• Right-Size With Canary Runs
• Runtime Observability
• CLI Reference
• Spec Reference

Right-Size With Canary Runs

hpc-compose germinate submits a short Slurm canary for an existing compose spec, forces runtime metrics on, waits for the canary to finish, and prints conservative resource recommendations for the original spec.

germinate is the pre-run probe: it submits a fresh short canary to estimate requests before you commit to a full run. inspect --rightsize is the post-run counterpart: it derives recommendations from the metrics a completed tracked run already produced. Use germinate when you have no run yet; use inspect --rightsize after a real run.

Canaries are short probes, not benchmark truth. They are useful for catching obvious over-requests such as asking for many GPUs when only one device is touched, or requesting far more memory than the process ever approaches during startup. They are not a substitute for full-run profiling when a workload has long warmup, data-dependent memory, lazy model loading, or late training phases.

Basic Workflow

hpc-compose germinate -f compose.yaml
hpc-compose germinate -f compose.yaml --format json
hpc-compose germinate -f compose.yaml --canary-time 00:01:00 --metrics-interval 5

The canary keeps partition, account, QoS, constraints, cache, runtime backend, and service topology from the original plan. It minimizes CPU, memory, and GPU requests in memory only, writes latest-canary.json, and leaves normal latest.json untouched.

Dry-run the canary script without submitting:

hpc-compose germinate -f compose.yaml --dry-run --script-out canary.sbatch

Output

Text output includes the canary job id, the standard right-sizing observations, and a YAML patch you can apply manually:

x-slurm:
  mem: 16G
services:
  trainer:
    x-slurm:
      cpus_per_task: 4

JSON output includes the same patch plus the full right-sizing report:

hpc-compose germinate -f compose.yaml --format json

Recommendation Rules

• CPU recommendations use observed CPU demand with conservative headroom and round up.
• Memory recommendations use the strongest available evidence from sampler rows, sstat, and sacct, then round to Slurm-friendly units.
• GPU recommendations shrink only when GPU sampler evidence shows fewer active devices.
• Walltime is observed but not down-sized from a short canary run.

Caveats

• Warmup-heavy jobs can look smaller than steady-state jobs.
• Data-dependent memory may peak after the canary exits.
• Lazy model loading can under-report memory and GPU use if no real request hits the model.
• Distributed training may need full topology even when a canary only exercises startup.
• Failed, OOM-like, time-limit, malformed-metrics, and missing-metrics cases are reported as diagnostics rather than YAML rewrites.

Start from examples/canary-right-size.yaml when you want a small, explicit spec to practice the workflow.

Related Docs

• Run Hyperparameter Sweeps
• Runtime Observability
• Runbook
• Spec Reference

Connect Jobs Across Allocations

hpc-compose cross-job rendezvous lets independent Slurm jobs coordinate through the shared cache directory. A provider job registers an address under <cache_dir>/rendezvous/<name>/latest.json; a later client job resolves that record and receives stable HPC_COMPOSE_RDZV_* environment variables.

This is same-cluster shared-storage discovery. It does not create DNS, tunnels, authentication, authorization, or a service mesh. Use it only inside a same-user or trusted shared-project cache boundary.

Provider

name: model-server

x-slurm:
  cache_dir: ${CACHE_DIR}

services:
  model:
    image: python:3.12-slim
    command: python -m http.server 8000
    readiness:
      type: tcp
      port: 8000
    x-slurm:
      rendezvous:
        register:
          name: model-server
          port: 8000
          protocol: http
          path: /
          ttl_seconds: 3600

Provider registration is declarative. If readiness is configured, the rendered script registers after the readiness check succeeds. On cleanup, it removes latest.json only when the current job still owns the latest record.

Client

name: model-client

x-slurm:
  cache_dir: ${CACHE_DIR}
  rendezvous: model-server

services:
  client:
    image: curlimages/curl:8.10.1
    command: curl -fsS "$HPC_COMPOSE_RDZV_MODEL_SERVER_URL"

Clients receive generic variables such as HPC_COMPOSE_RDZV_URL, plus name-scoped variables such as HPC_COMPOSE_RDZV_MODEL_SERVER_URL, HPC_COMPOSE_RDZV_MODEL_SERVER_HOST, and HPC_COMPOSE_RDZV_MODEL_SERVER_PORT.

Debugging CLI

hpc-compose rendezvous list --cache-dir "$CACHE_DIR"
hpc-compose rendezvous resolve model-server --cache-dir "$CACHE_DIR"
hpc-compose rendezvous register model-server --host node01 --port 8000 --job-id 12345 --cache-dir "$CACHE_DIR"
hpc-compose rendezvous prune --cache-dir "$CACHE_DIR"

register is mainly for debugging and custom workflows. Normal provider jobs should use services.<name>.x-slurm.rendezvous.register.

TTL and Staleness

Records have a TTL. Resolution ignores expired records, and prune removes expired latest and historical JSON files. If the provider job exits cleanly, cleanup removes the latest pointer only if it still points at that job, so a newer provider is not deregistered by an older job finishing later.

Requirements

• x-slurm.cache_dir must point to storage visible from the login node and compute nodes.
• Provider and client jobs must use the same cache directory.
• Names are single safe path components: ASCII letters, digits, ., _, and -.

See examples/rendezvous-model-server.yaml and examples/rendezvous-client.yaml for a runnable pair.

Related Docs

• Artifacts and Resume
• Cache Management
• Spec Reference
• Examples

Artifacts and Resume

Artifacts are collected after a run for export and provenance. Resume state is the canonical live checkpoint a later attempt loads on restart. Keep those roles separate: exported checkpoints are retrieval output, while the shared resume path is what a restarted run reads first.

Artifacts: Collection vs. Export

Artifact handling has two stages, and only the first is automatic.

1. Collection — automatic, at teardown (compute node)

When x-slurm.artifacts is enabled, the in-job teardown collects the declared paths into the tracked runtime directory:

<runtime-root>/<job-id>/artifacts/
  manifest.json
  payload/...

For resume-aware runs, the active attempt writes first under <runtime-root>/<job-id>/attempts/<attempt>/artifacts/; the top-level artifacts path is kept as the latest view.

This stage only fills the runtime payload — it never writes to export_dir.

2. Export — manual, on demand (login node)

Copying the collected payload into the configured export_dir is a separate, explicit step. Run it after the job finishes:

hpc-compose artifacts -f compose.yaml
hpc-compose artifacts -f compose.yaml --bundle checkpoints --tarball

export_dir is populated only by hpc-compose artifacts. Nothing runs it for you: down tears the job down without exporting, and pull only prints an rsync line that copies the runtime payload to your laptop (it does not touch export_dir). If downstream jobs read <export_dir>/<job-id>, run hpc-compose artifacts before down. When an export_dir is configured, hpc-compose surfaces this step in the “Next:” hints after up, status, and experiment.

export_dir is resolved relative to the compose file and expands ${SLURM_JOB_ID} from tracked metadata. Named bundles are written under <export_dir>/bundles/<bundle>/, and provenance JSON is written under <export_dir>/_hpc-compose/bundles/<bundle>.json.

The bundle name default is reserved for top-level x-slurm.artifacts.paths.

Resume-Aware Runs

When x-slurm.resume is enabled, hpc-compose:

• mounts the shared resume path into every service at /hpc-compose/resume
• injects HPC_COMPOSE_RESUME_DIR, HPC_COMPOSE_ATTEMPT, and HPC_COMPOSE_IS_RESUME
• writes attempt-specific runtime outputs under <runtime-root>/<job-id>/attempts/<attempt>/
• keeps <runtime-root>/<job-id>/{logs,metrics,artifacts,state.json} pointed at the latest attempt for compatibility

Use the shared resume directory for the canonical checkpoint a restarted run should load next. Treat exported artifacts as retrieval and provenance output after the attempt finishes, not as the primary live resume source.

Useful Commands

hpc-compose up --resume-diff-only -f compose.yaml
hpc-compose up --allow-resume-changes -f compose.yaml
hpc-compose artifacts -f compose.yaml

Related Docs

• Connect Jobs Across Allocations
• Runtime Observability
• Spec Reference
• Examples

Wire Up CI

hpc-compose ships fast, authoring-time commands (validate, lint) that are well-suited to pre-commit hooks and CI. This page covers three drop-in integrations: a pre-commit hook, a reusable GitHub Actions workflow, and a GitLab CI snippet.

All integrations require the hpc-compose binary to be installed first. The latest installer is:

curl -fsSL https://raw.githubusercontent.com/NicolasSchuler/hpc-compose/main/install.sh | sh

For pinned tags, checksum verification, and other install variants, see Installation. The CI snippets below pin a release tag for reproducible runs.

Pre-commit

The repository ships a .pre-commit-hooks.yaml defining two local hooks that run hpc-compose validate and hpc-compose lint against compose.yaml. Because hpc-compose is not distributed via pip, the hooks use language: system and require the binary to already be on PATH.

Add this repo to your .pre-commit-config.yaml:

Replace vX.Y.Z below with the latest release tag from the GitHub Releases page.

repos:
  - repo: https://github.com/NicolasSchuler/hpc-compose
    rev: vX.Y.Z  # pin to a release tag
    hooks:
      - id: hpc-compose-validate
      - id: hpc-compose-lint

By default the hooks run when a top-level compose.yaml is staged. If your project uses compose.yml or a nested spec, override both entry and files so the hook checks the file that triggered it:

      - id: hpc-compose-validate
        entry: hpc-compose validate -f compose.yml
        files: ^compose\.yml$
      - id: hpc-compose-lint
        entry: hpc-compose lint -f compose.yml --allow-warnings
        files: ^compose\.yml$

• hpc-compose-validate fails on any spec error.
• hpc-compose-lint passes with --allow-warnings (warnings are advisory). Use hpc-compose-validate plus a strict CI lint (below) to enforce both.

GitHub Actions

Reusable workflow

The simplest integration calls the maintained reusable workflow, which installs a pinned release and runs validate + lint. Replace vX.Y.Z with the latest release tag from the GitHub Releases page:

jobs:
  hpc-compose:
    uses: NicolasSchuler/hpc-compose/.github/workflows/hpc-compose-lint.yml@vX.Y.Z
    with:
      compose-file: compose.yaml
      version: vX.Y.Z
      strict: true

Set strict: true to fail on lint warnings, or strict: false (default) to allow warnings. Pin both the uses: ref and version to the same release tag.

Inline snippet

For repos that prefer an inline step. Replace vX.Y.Z with the latest release tag from the GitHub Releases page:

jobs:
  lint:
    runs-on: ubuntu-24.04
    steps:
      - uses: actions/checkout@v4
      - name: Install hpc-compose
        run: |
          set -euo pipefail
          curl -fsSL "https://raw.githubusercontent.com/NicolasSchuler/hpc-compose/vX.Y.Z/install.sh" \
            | env HPC_COMPOSE_VERSION="vX.Y.Z" sh
          echo "$HOME/.local/bin" >> "$GITHUB_PATH"
      - run: hpc-compose validate -f compose.yaml
      - run: hpc-compose lint -f compose.yaml --allow-warnings

GitLab CI

GitLab runners typically do not provide hpc-compose, so install it inside the job first. Replace vX.Y.Z with the latest release tag from the GitHub Releases page:

hpc-compose-lint:
  image: alpine:3.20
  rules:
    - changes: [compose.yaml]
  variables:
    HPC_COMPOSE_VERSION: vX.Y.Z
  before_script:
    - apk add --no-cache curl ca-certificates
    - |
      curl -fsSL "https://raw.githubusercontent.com/NicolasSchuler/hpc-compose/${HPC_COMPOSE_VERSION}/install.sh" \
        | env HPC_COMPOSE_VERSION="${HPC_COMPOSE_VERSION}" sh
    - export PATH="${HOME}/.local/bin:${PATH}"
  script:
    - hpc-compose validate -f compose.yaml
    - hpc-compose lint -f compose.yaml --allow-warnings

Strict vs. warnings

validate always fails on structural spec errors. lint emits advisory findings (HPC001–HPC007, HPC900); by default these fail the command, so add --allow-warnings for advisory-only runs. A common setup is:

• pre-commit / local: lint --allow-warnings (fast feedback, advisory).
• CI (merge gate): lint without --allow-warnings, or strict: true (enforce).

See CLI Reference for the full lint rule table and Troubleshooting for related workflows.

Related Docs

• Run a Notebook or IDE Session
• Use Secrets
• Installation
• Spec Reference

CLI Reference

This page maps the public hpc-compose CLI by workflow. Use Quickstart for the shortest install-and-run path, Runbook for real-cluster operations, and Spec Reference for YAML field behavior.

Command Index

Jump to the section that documents each command group:

Manual Pages

Every command also ships a Unix man page, generated from the same definitions as this reference:

man hpc-compose
man hpc-compose-up
man hpc-compose-checkpoints
man hpc-compose-sweep-submit

Release archives install them under share/man/man1/. From a source checkout, regenerate them with cargo run --features manpage-bin --bin gen-manpages.

Common Flags

Authoring and Setup

hpc-compose new --list-templates
hpc-compose new --describe-template minimal-batch
hpc-compose new --template minimal-batch --name my-app --output compose.yaml
hpc-compose new --template minimal-batch --name my-app --cache-dir '<shared-cache-dir>' --output compose.yaml
hpc-compose examples list
hpc-compose examples list --tag mpi --format json
hpc-compose examples search 'vllm worker'
hpc-compose examples recommend 'multi-node training' --tag gpu
hpc-compose examples recommend --format json
hpc-compose examples coverage --format markdown
hpc-compose evolve --list-lessons
hpc-compose evolve --describe-lesson progressive-complexity
hpc-compose evolve --output compose.yaml --name my-app
hpc-compose evolve --yes --until readiness --format json
hpc-compose setup
hpc-compose setup --profile-name dev --cache-dir '<shared-cache-dir>' --default-profile dev --non-interactive
hpc-compose context --format json
hpc-compose context --show-values --format json
hpc-compose completions zsh

evolve Options

evolve is authoring-only: it validates and writes candidate specs but does not prepare images, run preflight, or submit jobs. The default lesson is progressive-complexity, with steps minimal, second-service, readiness, failure-policy, and multi-node-placement.

• --list-lessons prints shipped lessons.
• --describe-lesson <LESSON> prints lesson steps and concepts.
• --lesson <LESSON> selects the lesson to run.
• --until <STEP> stops after a step id such as readiness.
• --yes accepts steps noninteractively.
• --format json is available for list/describe and for --yes runs.
• --force allows overwriting the output file.

Settings-aware commands

Use these commands and global flags when you want the project-local settings file (.hpc-compose/settings.toml) to remember compose path, env files, env vars, and binary overrides. The YAML these commands operate on is documented in Spec Reference.

Plan and Run

hpc-compose plan -f compose.yaml
hpc-compose plan --explain -f compose.yaml
hpc-compose plan --show-script -f compose.yaml
hpc-compose validate -f compose.yaml
hpc-compose lint -f compose.yaml
hpc-compose lint -f compose.yaml --allow-warnings
hpc-compose lint -f compose.yaml --fix
hpc-compose lint -f compose.yaml --fix --dry-run
hpc-compose lint -f compose.yaml --format json
hpc-compose config -f compose.yaml
hpc-compose config -f compose.yaml --variables
hpc-compose schema > hpc-compose.schema.json
hpc-compose inspect --verbose -f compose.yaml
hpc-compose inspect --dependencies -f compose.yaml
hpc-compose inspect --dependencies --dependencies-format dot -f compose.yaml
hpc-compose preflight -f compose.yaml
hpc-compose doctor cluster-report
hpc-compose doctor readiness -f compose.yaml --service api
hpc-compose doctor readiness -f compose.yaml --service api --run
hpc-compose doctor readiness -f compose.yaml --service api --run --log-file .hpc-compose/<job-id>/logs/api.log
hpc-compose doctor mpi-smoke -f compose.yaml --service trainer --script-out mpi-smoke.sbatch
hpc-compose doctor mpi-smoke -f compose.yaml --service trainer --submit
hpc-compose doctor fabric-smoke -f compose.yaml --service trainer --checks auto --script-out fabric-smoke.sbatch
hpc-compose doctor fabric-smoke -f compose.yaml --service trainer --checks mpi,nccl --submit
hpc-compose weather
hpc-compose weather --format json
hpc-compose prepare -f compose.yaml
hpc-compose render -f compose.yaml --output job.sbatch
hpc-compose up -f compose.yaml
hpc-compose up --hold-on-exit always -f compose.yaml
hpc-compose up --watch-queue --queue-warn-after 15m -f compose.yaml
hpc-compose up --detach --format json -f compose.yaml
hpc-compose up --detach --format json --print-endpoints -f compose.yaml
hpc-compose test --local -f compose.yaml
hpc-compose test --submit --time 00:01:00 -f compose.yaml
hpc-compose dev -f examples/dev-python-app.yaml
hpc-compose tmux -f examples/dev-python-app.yaml --no-attach
hpc-compose germinate -f compose.yaml
hpc-compose germinate -f compose.yaml --format json
hpc-compose germinate -f compose.yaml --dry-run --script-out canary.sbatch
hpc-compose sweep submit -f compose.yaml --dry-run
hpc-compose sweep submit -f compose.yaml --max-trials 200
hpc-compose sweep results -f compose.yaml --format csv > runs.csv
hpc-compose sweep results -f compose.yaml --include score,energy --format json
hpc-compose score --sweep sweep-1700000000-1234 -f compose.yaml --format json
hpc-compose stats --sweep sweep-1700000000-1234 -f compose.yaml --format json
hpc-compose sweep status -f compose.yaml --format json
hpc-compose sweep list -f compose.yaml
hpc-compose when -f compose.yaml --partition gpu8 --free-nodes 4
hpc-compose when -f compose.yaml --after-job 12345
hpc-compose when -f compose.yaml --between 22:00-06:00
hpc-compose when --detach --format json -f compose.yaml --partition gpu8 --free-nodes 4
hpc-compose alloc -f compose.yaml
hpc-compose run app -- python -m smoke_test
hpc-compose run --image docker://python:3.12 --resources cpu-small -- python -V
hpc-compose shell --image docker://ubuntu:24.04
hpc-compose notebook --kind jupyter --gpus 1 --volume ./project:/workspace
hpc-compose notebook --kind vscode --image ghcr.io/example/code:1 --gpus 1
hpc-compose notebook --local --kind jupyter
hpc-compose notebook --kind jupyter --format json

Lint rules

hpc-compose lint emits stable finding codes after validation and planning succeed. Warning-level findings fail the command by default; pass --allow-warnings to downgrade them to advisory so a warning-only run still succeeds.

Auto-fixable findings

hpc-compose lint --fix applies every fixable finding directly to the compose file. Today only HPC006 (implicit depends_on condition) is auto-fixable, because the rewrite is deterministic and semantics-preserving: the implicit service_started default is written out verbatim, so the rendered Slurm script is byte-identical.

hpc-compose lint -f compose.yaml --fix --dry-run   # preview the diff
hpc-compose lint -f compose.yaml --fix              # apply in place

The rewriter edits only the located depends_on block; comments, blank lines, and author formatting elsewhere are preserved byte-for-byte. A safety gate re-parses and re-plans the file after each run; if anything fails to reload the original file is restored. Path findings (HPC004, HPC005) are intentionally not auto-fixed because the correct replacement is cluster-specific.

Editor Schema

The checked-in schema is draft-07 JSON Schema and is published with the docs site at /schema/hpc-compose.schema.json. SchemaStore should associate it only with hpc-compose-specific filenames: hpc-compose.yaml, hpc-compose.yml, *.hpc-compose.yaml, and *.hpc-compose.yml. Generic compose.yaml remains a supported input file, but it is intentionally not claimed for zero-config editor association.

up Options

Useful workflow flags:

• --local runs a Pyxis/Enroot plan on the current Linux host instead of calling sbatch.
• --detach submits or launches and returns after tracking metadata is written.
• --format text|json is accepted with --detach or --dry-run.
• --watch-queue waits in line-oriented queue output until the Slurm job reaches RUNNING, then opens the normal watch view.
• --queue-warn-after <DURATION> warns once when --watch-queue stays PENDING longer than the threshold; the default is 10m, and 0 disables the warning.
• --watch-mode auto|tui|line selects the live output mode.
• --hold-on-exit never|failure|always controls whether the TUI stays open after the job reaches a terminal scheduler state.
• --allow-resume-changes acknowledges an intentional change to resume-coupled config between tracked runs.
• --resume-diff-only prints the resume-sensitive config diff without submitting.
• --script-out <PATH> keeps a copy of the rendered batch script.
• --remote[=<DEST>] delegates submission to a login node over SSH; without <DEST> it uses login_host from settings. <DEST> may be a host, an ~/.ssh/config alias, or user@host. Without an inline user@, the SSH user is taken from HPC_COMPOSE_REMOTE_USER, then settings login_user (profile over defaults), then your ~/.ssh/config. It cannot be combined with --local, --watch-queue, --script-out, or non-detached --watch-mode tui. See Submit From Your Laptop With up --remote.
• --remote-install <auto|never|force> (default auto; env HPC_COMPOSE_REMOTE_INSTALL) controls remote auto-install. up --remote probes the login node for hpc-compose and, under auto, installs the newest release into ~/.local/bin when it is missing or older than your local version. force always reinstalls before delegating; never only probes and fails with the manual install command when the binary is missing or old (use on locked-down or air-gapped login nodes).
• --force-rebuild refreshes imported and prepared artifacts before launch.
• --skip-prepare reuses an already-prepared image cache and builds nothing. On a first run (or after cache eviction) the image is not prepared yet, so preflight reports it as not-yet-prepared; run up/prepare once without --skip-prepare first. See Troubleshooting.
• --keep-failed-prep leaves the failed Enroot rootfs behind for inspection.
• Array jobs (x-slurm.array) require --detach because live watch/log fan-out is not array-aware yet.
• Scheduler dependencies from x-slurm.after_job and x-slurm.dependency are passed as sbatch --dependency=....
• stats --sweep <ID> and score --sweep <ID> require a real sweep id from sweep list; latest is not a special sentinel for those options.

Tool overrides

Commands that interact with Slurm or container runtimes accept --<tool>-bin <PATH> flags to point at non-default executables. This is useful when tools live outside PATH or when testing against fake binaries.

Note: doctor accepts the --*-bin overrides only through its deprecated top-level flag form, not the recommended doctor <subcommand> forms (doctor cluster-report, doctor readiness, doctor mpi-smoke, doctor fabric-smoke), which reject them with an “unexpected argument” error.

Settings profiles can also configure these via [defaults.binaries] or [profiles.<name>.binaries] (see Runbook).

germinate Canary Runs

germinate is the conservative right-sizing workflow:

hpc-compose germinate -f compose.yaml
hpc-compose germinate -f compose.yaml --canary-time 00:01:00 --metrics-interval 5
hpc-compose germinate -f compose.yaml --pending-timeout 30m --format json

Useful options:

• --canary-time <TIME> defaults to 00:01:00.
• --metrics-interval <SECONDS> defaults to 5 and is forced on in the canary plan.
• --pending-timeout <DURATION> defaults to 30m.
• --min-cpus <N>, --min-mem <MEM>, and --min-gpus <N> set canary floors.
• --dry-run renders the canary script without calling sbatch.
• --skip-prepare, --force-rebuild, --keep-failed-prep, --no-preflight, and --script-out match the normal preparation flags.

The command rejects x-slurm.array and never rewrites your compose file automatically. See Right-Size With Canary Runs.

sweep Hyperparameter Sweeps

sweep expands the top-level sweep block in a compose file. Each generated trial is rendered and submitted as an independent tracked Slurm job; sweep status and sweep list read the persisted manifest under .hpc-compose/sweeps/.

When the sweep sets replicates: N, each parameter config fans out into N seeded trials (t000r0, t000r1, …). sweep status, sweep observe, and sweep results then add a per-config mean±std(n) rollup (text and a groups array in JSON), sweep results --format csv gains config_key and replicate columns, and best_trial ranks on the per-config group mean rather than the single best replicate.

hpc-compose sweep submit -f train.yaml --dry-run
hpc-compose sweep submit -f train.yaml --max-trials 200
hpc-compose sweep submit -f train.yaml --format json
hpc-compose sweep status -f train.yaml
hpc-compose sweep status -f train.yaml --sweep-id sweep-123 --format json
hpc-compose sweep list -f train.yaml --format json

sweep submit options:

sweep status options:

sweep list options:

sweep observe options:

sweep stop options:

sweep stop cancels every still-running or pending trial of a sweep with scancel and records the stop on the manifest. Use it after sweep observe to realize early termination once an objective threshold is met.

See Hyperparameter Sweeps for the sweep spec shape, interpolation rules, status categories, and current limitations.

when Conditional Submission

when is a foreground monitor for constrained partitions and off-hour workflows. It runs the normal pre-submit work first, then polls until every supplied condition is true:

hpc-compose when -f compose.yaml --partition gpu8 --free-nodes 4
hpc-compose when -f compose.yaml --after-job 12345 --after-job-condition afterok
hpc-compose when -f compose.yaml --between 22:00-06:00

All conditions must hold (logical AND). --free-nodes counts only idle rows from sinfo -h -p <partition> -o "%T|%D" and requires --partition to match x-slurm.partition. --after-job polls squeue first and then sacct; afterok and afternotok fail immediately when the prior job reaches a terminal state that can never satisfy the requested condition. --between uses local login-node wall-clock time and supports wraparound windows such as 22:00-06:00.

Useful options:

• --poll-interval <DURATION> defaults to 60s; the minimum is 5s.
• --timeout <DURATION> gives up if conditions are not met; 0s performs one check.
• --detach returns after submission and tracking metadata are written.
• --format json is accepted with --detach and returns the condition summaries plus normal submission metadata.
• --skip-prepare, --force-rebuild, --keep-failed-prep, --no-preflight, and --script-out match the corresponding up preparation flags.

Example JSON automation:

hpc-compose when --detach --format json -f compose.yaml --partition gpu8 --free-nodes 4

There is no x-when YAML field. Conditional submission is intentionally a CLI workflow layered over the normal compose spec.

up --local

up --local launches a Pyxis/Enroot plan on the current host instead of calling sbatch. It is useful for local authoring and script inspection, not for distributed Slurm execution.

hpc-compose up --local --dry-run -f compose.yaml

Current constraints:

• Linux hosts only
• runtime.backend: pyxis only
• single-host specs only
• no distributed or partitioned placement
• no services.<name>.x-slurm.extra_srun_args
• no services.<name>.x-slurm.mpi
• no x-slurm.array
• no scheduler dependencies from x-slurm.after_job or x-slurm.dependency
• reservation-related x-slurm.submit_args are ignored
• x-slurm.error is ignored, and local batch stderr is written into the tracked local batch log

up --local follows the tracked local launch immediately, just like up does for a submitted job. Add --detach when you want to launch and return.

In local mode the batch script also exports HPC_COMPOSE_BACKEND_OVERRIDE=local, HPC_COMPOSE_LOCAL_ENROOT_BIN pointing to the resolved enroot binary, and HPC_COMPOSE_LOCAL_BIN_DIR containing a generated srun shim. These variables are internal to hpc-compose and not intended for direct use in compose specs.

Development Workflow

test, dev, and tmux are intentionally small workflows layered over the same render/prepare/tracking machinery as up. See Development Workflow for the smoke-test guide, hot-reload behavior, and local-mode constraints.

test is for finite smoke specs:

hpc-compose test --local -f compose.yaml
hpc-compose test --submit --time 00:01:00 --timeout 180s -f compose.yaml
hpc-compose test --submit --format json -f compose.yaml

Success means all tracked services appear in runtime state, launched at least once, passed readiness when readiness is configured, and completed successfully. Long-running application specs should use a smoke-test variant of the command or service entrypoint that exits after proving the workflow.

Useful test options:

dev is local-only and watches host directories from service volumes:

hpc-compose dev -f examples/dev-python-app.yaml
hpc-compose dev -f compose.yaml --watch-paths ./src --debounce-ms 500

Directory bind mounts are mapped back to affected services. File mounts, missing paths, container-only paths, cache paths, and non-directory paths are ignored. --watch-paths adds an explicit directory and restarts all services when it changes. By default, leaving dev stops the local supervisor; use --keep-running when you want the tracked local job to continue.

Useful dev options:

tmux opens a log dashboard for local runs:

hpc-compose tmux -f compose.yaml
hpc-compose tmux -f compose.yaml --job-id local-123
hpc-compose tmux -f compose.yaml --session demo --no-attach

When --job-id is omitted, tmux launches a new local run first. Each pane runs tail -F against one tracked service log and uses the service name as the pane title.

Useful tmux options:

run and shell

run has two forms:

hpc-compose run [-f compose.yaml] SERVICE -- CMD [ARGS...]
hpc-compose run --image IMAGE [--resources NAME] [--time T] [--mem M] [--cpus-per-task N] [--gpus N] [--partition P] [--env K=V] [--dataset PATH] [--output DIR] [--local] -- CMD [ARGS...]

Service mode reuses the named service’s image, environment, mounts, working directory, and prepare rules, clears depends_on, and submits a fresh tracked run job. When launched inside hpc-compose alloc, service mode detects HPC_COMPOSE_ALLOCATION=1 and SLURM_JOB_ID, prints the active allocation id, runs the one-service launcher inside the allocation with srun, and records the latest run metadata against the allocation job id. Image mode creates an ephemeral one-service plan from CLI flags, then follows the normal render/prepare/submit path. --resources refers to [resource_profiles.<name>] in settings; it is not the global --profile selector.

Image mode also accepts two batch-inference flags (both image-mode-only; using either without --image is an error):

• --dataset <PATH> binds an existing shared-filesystem path read-only into the container and exposes its in-container location as HPC_COMPOSE_DATASET_DIR. The path is filesystem-based only; remote or registry schemes such as hf:// are rejected, and a non-existent path fails before any submission. Copy datasets onto the shared filesystem first.
• --output <DIR> turns on artifact export: the in-container path exposed as HPC_COMPOSE_OUTPUT_DIR is collected after the job and exported into <DIR> (recorded as the run’s artifact_export_dir). Have the in-job command write its results under $HPC_COMPOSE_OUTPUT_DIR.

hpc-compose run --image docker://python:3.12 --dataset /scratch/data --output ./results -- python infer.py

alloc requests an interactive allocation through salloc:

hpc-compose alloc -f compose.yaml
hpc-compose alloc -f compose.yaml -- bash -lc 'hpc-compose run app -- python -m pytest'

It runs preflight and image preparation by default, accepts the matching up preparation flags (--no-preflight, --skip-prepare, --force-rebuild, and --keep-failed-prep), rejects x-slurm.array, and exports allocation metadata such as HPC_COMPOSE_COMPOSE_FILE, HPC_COMPOSE_CACHE_DIR, HPC_COMPOSE_NODELIST_FILE, and HPC_COMPOSE_PRIMARY_NODE.

shell is intentionally thinner:

hpc-compose shell --image IMAGE [--resources NAME] [--time T] [--mem M] [--cpus-per-task N] [--gpus N] [--partition P] [--env K=V]

It calls srun --pty directly with Pyxis --container-image and defaults to bash -l. It does not render an sbatch script or create tracked job metadata.

notebook launches a tracked interactive server:

hpc-compose notebook [--kind jupyter|vscode] [--image IMAGE] [--port N] [--token TOKEN]
                     [--volume HOST:CONTAINER]... [--working-dir PATH] [--tunnel-name NAME]
                     [--ready-timeout DURATION] [--follow] [--dry-run] [--local] [-- ARGS...]
                     [--resources NAME] [--time T] [--mem M] [--cpus-per-task N] [--gpus N]
                     [--partition P] [--env K=V]

It synthesizes a one-service compose job from the preset, runs the normal preflight/prepare/render path, submits (or launches locally with --local), waits for a log readiness signal, then prints the connection URL — a localhost Jupyter URL plus an SSH tunnel hint for Jupyter on Slurm, or the scraped vscode.dev link for VS Code. The session is a tracked job of kind notebook (see Notebook Sessions); stop it with hpc-compose cancel. --kind vscode requires --image because no universal default code image is shipped.

Accessible and Automation-Friendly Output

Use plain or structured output when terminal styling, progress labels, or alternate-screen interfaces make automation or assistive tooling harder:

hpc-compose --color never plan -f compose.yaml
hpc-compose --quiet validate -f compose.yaml
hpc-compose watch -f compose.yaml --watch-mode line
hpc-compose logs -f compose.yaml --service app --follow
hpc-compose logs -f compose.yaml --grep 'error|oom' --since 30m
hpc-compose status -f compose.yaml --format json

context and config --variables intentionally scope interpolation variables to names referenced by the compose file. Values whose names look secret-bearing are shown as <redacted> by default; add --show-values only in trusted local diagnostics. A name triggers redaction when, after upper-casing, it contains any of these case-insensitive substrings: SECRET, TOKEN, PASSWORD, PASSWD, API_KEY, ACCESS_KEY, PRIVATE_KEY, CREDENTIAL, AUTH, COOKIE, SESSION, BEARER. Because the test is a substring match, names such as SESSION_DIR or AUTH_MODE also match.

Tracked Runtime

Add --remote[=<HOST>] to status, ps, stats, score, logs, or pull to run that command on the login node’s staged checkout from a prior up --remote, over SSH, streaming output back. With no value it uses the configured login_host; pass user@host to override.

hpc-compose debug -f compose.yaml
hpc-compose debug -f compose.yaml --preflight
hpc-compose jobs list
hpc-compose status -f compose.yaml --format json
hpc-compose status -f compose.yaml --array
hpc-compose status -f compose.yaml --job-id 12345_7 --array
hpc-compose ps -f compose.yaml
hpc-compose watch -f compose.yaml --watch-mode line
hpc-compose watch -f compose.yaml --hold-on-exit always
hpc-compose replay -f compose.yaml
hpc-compose replay -f compose.yaml --speed 10
hpc-compose replay -f compose.yaml --job-id 12345 --service app
hpc-compose replay -f compose.yaml --format json
hpc-compose checkpoints -f compose.yaml
hpc-compose checkpoints --job-id 12345 --format json
hpc-compose logs -f compose.yaml --service app --follow
hpc-compose logs -f compose.yaml --grep 'error|oom' --since 30m
hpc-compose inspect -f compose.yaml --rightsize
hpc-compose stats -f compose.yaml --format jsonl
hpc-compose stats -f compose.yaml --accounting --format csv
hpc-compose score 12345
hpc-compose diff 12345 12346 -f compose.yaml
hpc-compose diff --jobs 12345,12346,12347 --matrix-format json
hpc-compose diff --across sweep-1700000000-1234 --matrix-format csv
hpc-compose artifacts -f compose.yaml --bundle checkpoints --tarball
hpc-compose down -f compose.yaml --yes
hpc-compose cancel -f compose.yaml --yes
hpc-compose clean -f compose.yaml --age 7 --dry-run
hpc-compose rendezvous list
hpc-compose rendezvous resolve model-server
hpc-compose rendezvous register model-server --host node01 --port 8000 --job-id 12345
hpc-compose rendezvous prune

Cache Maintenance

hpc-compose cache list
hpc-compose cache inspect -f compose.yaml --service app
hpc-compose cache prune --age 7 --cache-dir '<shared-cache-dir>' --yes
hpc-compose cache prune --all-unused -f compose.yaml --yes

Related Docs

• Spec Reference
• Glossary
• Full Example Specs
• Roadmap and Non-Goals
• Runbook

Spec Reference

This page describes the Compose subset that hpc-compose accepts today. Unknown or unsupported fields are rejected unless this page explicitly says otherwise.

How To Use This Reference

This page is intentionally complete. If you are new, start with Quickstart, Examples, and Runtime Backends, then use the table below to jump into the field group you need.

Top-level shape

name: demo
version: "1"

runtime:
  backend: pyxis

x-slurm:
  time: "00:30:00"

services:
  app:
    image: python:3.11-slim
    command: python -m main

Top-level fields

extends

extends is an authoring feature for sharing base specs and service templates without copying large cluster-specific blocks. It is resolved before interpolation, validation, planning, rendering, tracked metadata, and hpc-compose config; the effective config no longer contains any extends keys.

Top-level extends points at a base YAML file:

extends: cluster-base.yaml

x-slurm:
  time: "02:00:00"

services:
  trainer:
    command: python train.py

Service-level extends supports three forms:

services:
  api:
    extends: base-service

  worker:
    extends: service-templates.yaml

  trainer:
    extends:
      file: ml-templates.yaml
      service: gpu-worker

Rules:

• Top-level extends must be a file path string.
• A service string that looks like a YAML file path, such as base.yaml, ../base.yml, or a path with a separator, uses the same service name from that file. Other strings refer to a service in the same file.
• A service mapping can select { file, service }; omit file to select a service from the same file.
• Extends references are recursive and cycles are rejected.
• Maps merge recursively. Sequences append base-first. Child scalars replace base scalars.
• Service volumes merge by container target, so a child mount for /data replaces the base mount for /data while unrelated base mounts are kept.
• Relative host paths in the final plan still resolve against the leaf compose file passed with -f.
• There is no delete or unset syntax in this version.

sweep

sweep defines trial variables for hpc-compose sweep submit. It is a top-level metadata block; every generated trial is still planned, rendered, submitted, and tracked as a normal one-allocation job.

Full Cartesian product:

sweep:
  parameters:
    lr: [0.001, 0.01, 0.1]
    batch_size: [32, 64]
  matrix: full

Random sample without replacement:

sweep:
  parameters:
    lr: [0.001, 0.01, 0.1]
    batch_size: [32, 64]
  matrix:
    random: 5
    seed: "optional-stable-seed"

Rules:

• parameters must contain at least one key, and every value list must contain at least one scalar.
• Parameter keys must be valid interpolation variable names: [A-Za-z_][A-Za-z0-9_]*.
• Parameter keys must not use the reserved HPC_COMPOSE_SWEEP_ prefix.
• Parameter values may be strings, numbers, or booleans. They are passed to interpolation as strings.
• matrix: full expands the Cartesian product deterministically over sorted parameter names.
• matrix.random must be at least 1 and cannot exceed the total number of combinations.
• matrix.seed is optional. If omitted, sweep submit derives a seed from the new sweep id and persists it.
• replicates (optional, default 1) submits N seeded replicate trials per parameter config. Each replicate is a separate allocation with a deterministic per-replicate seed; sweep status/observe roll up mean±std(n) per config. The --max-trials guard counts combinations × replicates. replicates: 0 is rejected; replicates: 1 is byte-identical to a non-replicated sweep (legacy t000 trial ids). See Hyperparameter Sweeps.
• objective (optional) declares how sweep observe parses and ranks each terminal trial. Set direction (minimize/maximize) and exactly one parse source: log_pattern (a regex with optional capture group, default 1) or json_path + json_field. Optionally set objective.scaling_axis to the name of a numeric sweep parameter (e.g. nodes) to enable the read-only sweep observe --scaling report; the named parameter must exist under sweep.parameters and all its values must parse as numbers (both checked at validate time). See Scaling Reports.
• sweep.spec is not supported; embed the sweep in the same compose file.

For each trial, sweep variables override existing interpolation variables from .env, environment, settings, or --env. These reserved variables are also available:

Normal commands do not expand the sweep matrix. If the runnable spec contains ${lr} with no default, ordinary plan, up, and render still fail unless lr is provided. Use defaults such as ${lr:-0.001} when the base spec should remain runnable, or use hpc-compose sweep submit --dry-run to validate sweep-only variables.

hpc-compose sweep submit rejects x-slurm.array, because every sweep trial is already its own allocation. See Hyperparameter Sweeps for manifests, status aggregation, objective ranking via sweep observe, and early termination via sweep stop.

secrets

secrets maps secret names to local file: or env: sources. Each value is resolved into the interpolation map tagged as a secret, so ${name} works in environment: and is redacted in config/context/inspect output regardless of its name.

secrets:
  hf_token:
    file: ./secrets/hf.txt
  db_password:
    env: DB_PASSWORD
services:
  app:
    image: redis:7
    environment:
      HF_TOKEN: ${hf_token}

See Secrets for the full redaction model, resolution order, and what is deferred (Vault/KMS, /run/secrets file mounts).

x-env

x-env is structured host-side software setup. It is available at the top level and under services.<name>.

x-env:
  modules:
    - cuda/12.4
    - openmpi/5
  spack:
    view: /shared/spack/views/ml
  env:
    HDF5_USE_FILE_LOCKING: "FALSE"

services:
  app:
    image: python:3.11-slim
    x-env:
      modules:
        purge: false
        load:
          - netcdf/4.9
      env:
        OMP_NUM_THREADS: "8"

Supported forms:

• modules: [name, ...]
• modules: { purge: bool, load: [name, ...] }
• spack: { view: /path/to/view }
• env: { KEY: VALUE }

Rules:

• Top-level x-env renders before x-slurm.setup.
• Service-level x-env renders immediately before that service’s srun.
• env entries are exported on the host and forwarded into Pyxis containers.
• Service-level x-env.env overrides top-level x-env.env when the same variable is set.
• Top-level modules: [...] and service-level modules: [...] are shorthand for the matching x-env.modules.load list. The shorthand is list-only and cannot be combined with x-env.modules at the same scope.
• spack.view prepends bin, lib, lib64, and Python site-package paths only when those directories exist.
• Modules and Spack views are host-side setup. Container filesystem visibility still requires explicit volumes, x-slurm.mpi.host_mpi.bind_paths, or other site-specific binds.

Settings and lint commands

CLI behavior for the settings-aware commands (--profile, --settings-file, setup, context, validate --strict-env, lint, schema) and the full lint-rule table (HPC001-HPC900, including auto-fix) now lives in CLI Reference: see Settings-aware commands and Lint rules. This page describes only the YAML these commands operate on.

x-slurm

These fields live under the top-level x-slurm block.

x-slurm.parallelism

parallelism records the tensor (tensor) and pipeline (pipeline) sizes a job intends to use. Both fields are required and must be at least 1. It is purely descriptive: it lowers onto the existing single-srun-per-service placement and emits no extra #SBATCH or srun flags.

When gpus_per_node is set at the same scope, validation cross-checks that tensor * pipeline == nodes * gpus_per_node (where nodes defaults to 1 when omitted). A mismatch fails validate/config with a scoped diagnostic; the check is skipped entirely when gpus_per_node is not set.

x-slurm:
  nodes: 2
  gpus_per_node: 4
  parallelism:
    tensor: 4
    pipeline: 2 # 4 * 2 == 2 * 4

Resource profiles

Resource profiles are reusable settings defaults, distinct from the global --profile setting selector. Define them in .hpc-compose/settings.toml:

[resource_profiles.gpu-small]
partition = "gpu"
time = "01:00:00"
gpus = 1
cpus_per_task = 8
mem = "32G"

Reference one from the spec:

x-slurm:
  resources: gpu-small
  mem: 64G

The profile fills only omitted resource fields. In the example above, partition, time, gpus, and cpus_per_task come from the profile, while the explicit mem: 64G wins. Profiles intentionally exclude behavior such as job_name, cache_dir, arrays, dependencies, submit_args, setup hooks, scratch/staging, artifacts, resume, notify, and metrics.

Allowed profile fields are: partition, account, qos, time, nodes, ntasks, ntasks_per_node, cpus_per_task, mem, gres, gpus, gpus_per_node, gpus_per_task, cpus_per_gpu, mem_per_gpu, gpu_bind, cpu_bind, mem_bind, distribution, hint, and constraint.

x-slurm.array

x-slurm:
  array: 0-99%10
  output: logs/%A_%a.out
services:
  worker:
    image: python:3.12-slim
    command: python worker.py

array accepts Slurm list, range, step, and concurrency forms such as 0, 1-10, 1-10:2, 0,3,8-12, and 0-99%10. Values with spaces, null bytes, malformed ranges, negative numbers, zero step, or zero concurrency are rejected.

Array jobs currently require hpc-compose up --detach; live watch/log fan-out for per-task array elements is future work. --local rejects array specs. Slurm provides SLURM_ARRAY_JOB_ID, SLURM_ARRAY_TASK_ID, SLURM_ARRAY_TASK_COUNT, SLURM_ARRAY_TASK_MAX, SLURM_ARRAY_TASK_MIN, and SLURM_ARRAY_TASK_STEP; for Pyxis jobs, hpc-compose forwards these names into the container when x-slurm.array is set. Prefer output patterns such as %A_%a so task logs do not overwrite each other.