CLI Reference

This page maps the public hpc-compose CLI by workflow. Use Quickstart for the shortest install-and-run path, Runbook for real-cluster operations, and Spec Reference for YAML field behavior.

Command Index

Jump to the section that documents each command group:

Manual Pages

Every command also ships a Unix man page, generated from the same definitions as this reference:

man hpc-compose
man hpc-compose-up
man hpc-compose-checkpoints
man hpc-compose-sweep-submit

Release archives install them under share/man/man1/. From a source checkout, regenerate them with cargo run --features manpage-bin --bin gen-manpages.

Common Flags

Authoring and Setup

hpc-compose new --list-templates
hpc-compose new --describe-template minimal-batch
hpc-compose new --template minimal-batch --name my-app --output compose.yaml
hpc-compose new --template minimal-batch --name my-app --cache-dir '<shared-cache-dir>' --output compose.yaml
hpc-compose examples list
hpc-compose examples list --tag mpi --format json
hpc-compose examples search 'vllm worker'
hpc-compose examples recommend 'multi-node training' --tag gpu
hpc-compose examples recommend --format json
hpc-compose examples coverage --format markdown
hpc-compose evolve --list-lessons
hpc-compose evolve --describe-lesson progressive-complexity
hpc-compose evolve --output compose.yaml --name my-app
hpc-compose evolve --yes --until readiness --format json
hpc-compose setup
hpc-compose setup --profile-name dev --cache-dir '<shared-cache-dir>' --default-profile dev --non-interactive
hpc-compose context --format json
hpc-compose context --show-values --format json
hpc-compose completions zsh

evolve Options

evolve is authoring-only: it validates and writes candidate specs but does not prepare images, run preflight, or submit jobs. The default lesson is progressive-complexity, with steps minimal, second-service, readiness, failure-policy, and multi-node-placement.

• --list-lessons prints shipped lessons.
• --describe-lesson <LESSON> prints lesson steps and concepts.
• --lesson <LESSON> selects the lesson to run.
• --until <STEP> stops after a step id such as readiness.
• --yes accepts steps noninteractively.
• --format json is available for list/describe and for --yes runs.
• --force allows overwriting the output file.

Settings-aware commands

Use these commands and global flags when you want the project-local settings file (.hpc-compose/settings.toml) to remember compose path, env files, env vars, and binary overrides. The YAML these commands operate on is documented in Spec Reference.

Plan and Run

hpc-compose plan -f compose.yaml
hpc-compose plan --explain -f compose.yaml
hpc-compose plan --show-script -f compose.yaml
hpc-compose validate -f compose.yaml
hpc-compose lint -f compose.yaml
hpc-compose lint -f compose.yaml --allow-warnings
hpc-compose lint -f compose.yaml --fix
hpc-compose lint -f compose.yaml --fix --dry-run
hpc-compose lint -f compose.yaml --format json
hpc-compose config -f compose.yaml
hpc-compose config -f compose.yaml --variables
hpc-compose schema > hpc-compose.schema.json
hpc-compose inspect --verbose -f compose.yaml
hpc-compose inspect --dependencies -f compose.yaml
hpc-compose inspect --dependencies --dependencies-format dot -f compose.yaml
hpc-compose preflight -f compose.yaml
hpc-compose doctor cluster-report
hpc-compose doctor readiness -f compose.yaml --service api
hpc-compose doctor readiness -f compose.yaml --service api --run
hpc-compose doctor readiness -f compose.yaml --service api --run --log-file .hpc-compose/<job-id>/logs/api.log
hpc-compose doctor mpi-smoke -f compose.yaml --service trainer --script-out mpi-smoke.sbatch
hpc-compose doctor mpi-smoke -f compose.yaml --service trainer --submit
hpc-compose doctor fabric-smoke -f compose.yaml --service trainer --checks auto --script-out fabric-smoke.sbatch
hpc-compose doctor fabric-smoke -f compose.yaml --service trainer --checks mpi,nccl --submit
hpc-compose weather
hpc-compose weather --format json
hpc-compose prepare -f compose.yaml
hpc-compose render -f compose.yaml --output job.sbatch
hpc-compose up -f compose.yaml
hpc-compose up --hold-on-exit always -f compose.yaml
hpc-compose up --watch-queue --queue-warn-after 15m -f compose.yaml
hpc-compose up --detach --format json -f compose.yaml
hpc-compose up --detach --format json --print-endpoints -f compose.yaml
hpc-compose test --local -f compose.yaml
hpc-compose test --submit --time 00:01:00 -f compose.yaml
hpc-compose dev -f examples/dev-python-app.yaml
hpc-compose tmux -f examples/dev-python-app.yaml --no-attach
hpc-compose germinate -f compose.yaml
hpc-compose germinate -f compose.yaml --format json
hpc-compose germinate -f compose.yaml --dry-run --script-out canary.sbatch
hpc-compose sweep submit -f compose.yaml --dry-run
hpc-compose sweep submit -f compose.yaml --max-trials 200
hpc-compose sweep results -f compose.yaml --format csv > runs.csv
hpc-compose sweep results -f compose.yaml --include score,energy --format json
hpc-compose score --sweep sweep-1700000000-1234 -f compose.yaml --format json
hpc-compose stats --sweep sweep-1700000000-1234 -f compose.yaml --format json
hpc-compose sweep status -f compose.yaml --format json
hpc-compose sweep list -f compose.yaml
hpc-compose when -f compose.yaml --partition gpu8 --free-nodes 4
hpc-compose when -f compose.yaml --after-job 12345
hpc-compose when -f compose.yaml --between 22:00-06:00
hpc-compose when --detach --format json -f compose.yaml --partition gpu8 --free-nodes 4
hpc-compose alloc -f compose.yaml
hpc-compose run app -- python -m smoke_test
hpc-compose run --image docker://python:3.12 --resources cpu-small -- python -V
hpc-compose shell --image docker://ubuntu:24.04
hpc-compose notebook --kind jupyter --gpus 1 --volume ./project:/workspace
hpc-compose notebook --kind vscode --image ghcr.io/example/code:1 --gpus 1
hpc-compose notebook --local --kind jupyter
hpc-compose notebook --kind jupyter --format json

Lint rules

hpc-compose lint emits stable finding codes after validation and planning succeed. Warning-level findings fail the command by default; pass --allow-warnings to downgrade them to advisory so a warning-only run still succeeds.

Auto-fixable findings

hpc-compose lint --fix applies every fixable finding directly to the compose file. Today only HPC006 (implicit depends_on condition) is auto-fixable, because the rewrite is deterministic and semantics-preserving: the implicit service_started default is written out verbatim, so the rendered Slurm script is byte-identical.

hpc-compose lint -f compose.yaml --fix --dry-run   # preview the diff
hpc-compose lint -f compose.yaml --fix              # apply in place

The rewriter edits only the located depends_on block; comments, blank lines, and author formatting elsewhere are preserved byte-for-byte. A safety gate re-parses and re-plans the file after each run; if anything fails to reload the original file is restored. Path findings (HPC004, HPC005) are intentionally not auto-fixed because the correct replacement is cluster-specific.

Editor Schema

The checked-in schema is draft-07 JSON Schema and is published with the docs site at /schema/hpc-compose.schema.json. SchemaStore should associate it only with hpc-compose-specific filenames: hpc-compose.yaml, hpc-compose.yml, *.hpc-compose.yaml, and *.hpc-compose.yml. Generic compose.yaml remains a supported input file, but it is intentionally not claimed for zero-config editor association.

up Options

Useful workflow flags:

• --local runs a Pyxis/Enroot plan on the current Linux host instead of calling sbatch.
• --detach submits or launches and returns after tracking metadata is written.
• --format text|json is accepted with --detach or --dry-run.
• --watch-queue waits in line-oriented queue output until the Slurm job reaches RUNNING, then opens the normal watch view.
• --queue-warn-after <DURATION> warns once when --watch-queue stays PENDING longer than the threshold; the default is 10m, and 0 disables the warning.
• --watch-mode auto|tui|line selects the live output mode.
• --hold-on-exit never|failure|always controls whether the TUI stays open after the job reaches a terminal scheduler state.
• --allow-resume-changes acknowledges an intentional change to resume-coupled config between tracked runs.
• --resume-diff-only prints the resume-sensitive config diff without submitting.
• --script-out <PATH> keeps a copy of the rendered batch script.
• --remote[=<DEST>] delegates submission to a login node over SSH; without <DEST> it uses login_host from settings. <DEST> may be a host, an ~/.ssh/config alias, or user@host. Without an inline user@, the SSH user is taken from HPC_COMPOSE_REMOTE_USER, then settings login_user (profile over defaults), then your ~/.ssh/config. It cannot be combined with --local, --watch-queue, --script-out, or non-detached --watch-mode tui. See Submit From Your Laptop With up --remote.
• --remote-install <auto|never|force> (default auto; env HPC_COMPOSE_REMOTE_INSTALL) controls remote auto-install. up --remote probes the login node for hpc-compose and, under auto, installs the newest release into ~/.local/bin when it is missing or older than your local version. force always reinstalls before delegating; never only probes and fails with the manual install command when the binary is missing or old (use on locked-down or air-gapped login nodes).
• --force-rebuild refreshes imported and prepared artifacts before launch.
• --skip-prepare reuses an already-prepared image cache and builds nothing. On a first run (or after cache eviction) the image is not prepared yet, so preflight reports it as not-yet-prepared; run up/prepare once without --skip-prepare first. See Troubleshooting.
• --keep-failed-prep leaves the failed Enroot rootfs behind for inspection.
• Array jobs (x-slurm.array) require --detach because live watch/log fan-out is not array-aware yet.
• Scheduler dependencies from x-slurm.after_job and x-slurm.dependency are passed as sbatch --dependency=....
• stats --sweep <ID> and score --sweep <ID> require a real sweep id from sweep list; latest is not a special sentinel for those options.

Tool overrides

Commands that interact with Slurm or container runtimes accept --<tool>-bin <PATH> flags to point at non-default executables. This is useful when tools live outside PATH or when testing against fake binaries.

Note: doctor accepts the --*-bin overrides only through its deprecated top-level flag form, not the recommended doctor <subcommand> forms (doctor cluster-report, doctor readiness, doctor mpi-smoke, doctor fabric-smoke), which reject them with an “unexpected argument” error.

Settings profiles can also configure these via [defaults.binaries] or [profiles.<name>.binaries] (see Runbook).

germinate Canary Runs

germinate is the conservative right-sizing workflow:

hpc-compose germinate -f compose.yaml
hpc-compose germinate -f compose.yaml --canary-time 00:01:00 --metrics-interval 5
hpc-compose germinate -f compose.yaml --pending-timeout 30m --format json

Useful options:

• --canary-time <TIME> defaults to 00:01:00.
• --metrics-interval <SECONDS> defaults to 5 and is forced on in the canary plan.
• --pending-timeout <DURATION> defaults to 30m.
• --min-cpus <N>, --min-mem <MEM>, and --min-gpus <N> set canary floors.
• --dry-run renders the canary script without calling sbatch.
• --skip-prepare, --force-rebuild, --keep-failed-prep, --no-preflight, and --script-out match the normal preparation flags.

The command rejects x-slurm.array and never rewrites your compose file automatically. See Right-Size With Canary Runs.

sweep Hyperparameter Sweeps

sweep expands the top-level sweep block in a compose file. Each generated trial is rendered and submitted as an independent tracked Slurm job; sweep status and sweep list read the persisted manifest under .hpc-compose/sweeps/.

When the sweep sets replicates: N, each parameter config fans out into N seeded trials (t000r0, t000r1, …). sweep status, sweep observe, and sweep results then add a per-config mean±std(n) rollup (text and a groups array in JSON), sweep results --format csv gains config_key and replicate columns, and best_trial ranks on the per-config group mean rather than the single best replicate.

hpc-compose sweep submit -f train.yaml --dry-run
hpc-compose sweep submit -f train.yaml --max-trials 200
hpc-compose sweep submit -f train.yaml --format json
hpc-compose sweep status -f train.yaml
hpc-compose sweep status -f train.yaml --sweep-id sweep-123 --format json
hpc-compose sweep list -f train.yaml --format json

sweep submit options:

sweep status options:

sweep list options:

sweep observe options:

sweep stop options:

sweep stop cancels every still-running or pending trial of a sweep with scancel and records the stop on the manifest. Use it after sweep observe to realize early termination once an objective threshold is met.

See Hyperparameter Sweeps for the sweep spec shape, interpolation rules, status categories, and current limitations.

when Conditional Submission

when is a foreground monitor for constrained partitions and off-hour workflows. It runs the normal pre-submit work first, then polls until every supplied condition is true:

hpc-compose when -f compose.yaml --partition gpu8 --free-nodes 4
hpc-compose when -f compose.yaml --after-job 12345 --after-job-condition afterok
hpc-compose when -f compose.yaml --between 22:00-06:00

All conditions must hold (logical AND). --free-nodes counts only idle rows from sinfo -h -p <partition> -o "%T|%D" and requires --partition to match x-slurm.partition. --after-job polls squeue first and then sacct; afterok and afternotok fail immediately when the prior job reaches a terminal state that can never satisfy the requested condition. --between uses local login-node wall-clock time and supports wraparound windows such as 22:00-06:00.

Useful options:

• --poll-interval <DURATION> defaults to 60s; the minimum is 5s.
• --timeout <DURATION> gives up if conditions are not met; 0s performs one check.
• --detach returns after submission and tracking metadata are written.
• --format json is accepted with --detach and returns the condition summaries plus normal submission metadata.
• --skip-prepare, --force-rebuild, --keep-failed-prep, --no-preflight, and --script-out match the corresponding up preparation flags.

Example JSON automation:

hpc-compose when --detach --format json -f compose.yaml --partition gpu8 --free-nodes 4

There is no x-when YAML field. Conditional submission is intentionally a CLI workflow layered over the normal compose spec.

up --local

up --local launches a Pyxis/Enroot plan on the current host instead of calling sbatch. It is useful for local authoring and script inspection, not for distributed Slurm execution.

hpc-compose up --local --dry-run -f compose.yaml

Current constraints:

• Linux hosts only
• runtime.backend: pyxis only
• single-host specs only
• no distributed or partitioned placement
• no services.<name>.x-slurm.extra_srun_args
• no services.<name>.x-slurm.mpi
• no x-slurm.array
• no scheduler dependencies from x-slurm.after_job or x-slurm.dependency
• reservation-related x-slurm.submit_args are ignored
• x-slurm.error is ignored, and local batch stderr is written into the tracked local batch log

up --local follows the tracked local launch immediately, just like up does for a submitted job. Add --detach when you want to launch and return.

In local mode the batch script also exports HPC_COMPOSE_BACKEND_OVERRIDE=local, HPC_COMPOSE_LOCAL_ENROOT_BIN pointing to the resolved enroot binary, and HPC_COMPOSE_LOCAL_BIN_DIR containing a generated srun shim. These variables are internal to hpc-compose and not intended for direct use in compose specs.

Development Workflow

test, dev, and tmux are intentionally small workflows layered over the same render/prepare/tracking machinery as up. See Development Workflow for the smoke-test guide, hot-reload behavior, and local-mode constraints.

test is for finite smoke specs:

hpc-compose test --local -f compose.yaml
hpc-compose test --submit --time 00:01:00 --timeout 180s -f compose.yaml
hpc-compose test --submit --format json -f compose.yaml

Success means all tracked services appear in runtime state, launched at least once, passed readiness when readiness is configured, and completed successfully. Long-running application specs should use a smoke-test variant of the command or service entrypoint that exits after proving the workflow.

Useful test options:

dev is local-only and watches host directories from service volumes:

hpc-compose dev -f examples/dev-python-app.yaml
hpc-compose dev -f compose.yaml --watch-paths ./src --debounce-ms 500

Directory bind mounts are mapped back to affected services. File mounts, missing paths, container-only paths, cache paths, and non-directory paths are ignored. --watch-paths adds an explicit directory and restarts all services when it changes. By default, leaving dev stops the local supervisor; use --keep-running when you want the tracked local job to continue.

Useful dev options:

tmux opens a log dashboard for local runs:

hpc-compose tmux -f compose.yaml
hpc-compose tmux -f compose.yaml --job-id local-123
hpc-compose tmux -f compose.yaml --session demo --no-attach

When --job-id is omitted, tmux launches a new local run first. Each pane runs tail -F against one tracked service log and uses the service name as the pane title.

Useful tmux options:

run and shell

run has two forms:

hpc-compose run [-f compose.yaml] SERVICE -- CMD [ARGS...]
hpc-compose run --image IMAGE [--resources NAME] [--time T] [--mem M] [--cpus-per-task N] [--gpus N] [--partition P] [--env K=V] [--dataset PATH] [--output DIR] [--local] -- CMD [ARGS...]

Service mode reuses the named service’s image, environment, mounts, working directory, and prepare rules, clears depends_on, and submits a fresh tracked run job. When launched inside hpc-compose alloc, service mode detects HPC_COMPOSE_ALLOCATION=1 and SLURM_JOB_ID, prints the active allocation id, runs the one-service launcher inside the allocation with srun, and records the latest run metadata against the allocation job id. Image mode creates an ephemeral one-service plan from CLI flags, then follows the normal render/prepare/submit path. --resources refers to [resource_profiles.<name>] in settings; it is not the global --profile selector.

Image mode also accepts two batch-inference flags (both image-mode-only; using either without --image is an error):

• --dataset <PATH> binds an existing shared-filesystem path read-only into the container and exposes its in-container location as HPC_COMPOSE_DATASET_DIR. The path is filesystem-based only; remote or registry schemes such as hf:// are rejected, and a non-existent path fails before any submission. Copy datasets onto the shared filesystem first.
• --output <DIR> turns on artifact export: the in-container path exposed as HPC_COMPOSE_OUTPUT_DIR is collected after the job and exported into <DIR> (recorded as the run’s artifact_export_dir). Have the in-job command write its results under $HPC_COMPOSE_OUTPUT_DIR.

hpc-compose run --image docker://python:3.12 --dataset /scratch/data --output ./results -- python infer.py

alloc requests an interactive allocation through salloc:

hpc-compose alloc -f compose.yaml
hpc-compose alloc -f compose.yaml -- bash -lc 'hpc-compose run app -- python -m pytest'

It runs preflight and image preparation by default, accepts the matching up preparation flags (--no-preflight, --skip-prepare, --force-rebuild, and --keep-failed-prep), rejects x-slurm.array, and exports allocation metadata such as HPC_COMPOSE_COMPOSE_FILE, HPC_COMPOSE_CACHE_DIR, HPC_COMPOSE_NODELIST_FILE, and HPC_COMPOSE_PRIMARY_NODE.

shell is intentionally thinner:

hpc-compose shell --image IMAGE [--resources NAME] [--time T] [--mem M] [--cpus-per-task N] [--gpus N] [--partition P] [--env K=V]

It calls srun --pty directly with Pyxis --container-image and defaults to bash -l. It does not render an sbatch script or create tracked job metadata.

notebook launches a tracked interactive server:

hpc-compose notebook [--kind jupyter|vscode] [--image IMAGE] [--port N] [--token TOKEN]
                     [--volume HOST:CONTAINER]... [--working-dir PATH] [--tunnel-name NAME]
                     [--ready-timeout DURATION] [--follow] [--dry-run] [--local] [-- ARGS...]
                     [--resources NAME] [--time T] [--mem M] [--cpus-per-task N] [--gpus N]
                     [--partition P] [--env K=V]

It synthesizes a one-service compose job from the preset, runs the normal preflight/prepare/render path, submits (or launches locally with --local), waits for a log readiness signal, then prints the connection URL — a localhost Jupyter URL plus an SSH tunnel hint for Jupyter on Slurm, or the scraped vscode.dev link for VS Code. The session is a tracked job of kind notebook (see Notebook Sessions); stop it with hpc-compose cancel. --kind vscode requires --image because no universal default code image is shipped.

Accessible and Automation-Friendly Output

Use plain or structured output when terminal styling, progress labels, or alternate-screen interfaces make automation or assistive tooling harder:

hpc-compose --color never plan -f compose.yaml
hpc-compose --quiet validate -f compose.yaml
hpc-compose watch -f compose.yaml --watch-mode line
hpc-compose logs -f compose.yaml --service app --follow
hpc-compose logs -f compose.yaml --grep 'error|oom' --since 30m
hpc-compose status -f compose.yaml --format json

context and config --variables intentionally scope interpolation variables to names referenced by the compose file. Values whose names look secret-bearing are shown as <redacted> by default; add --show-values only in trusted local diagnostics. A name triggers redaction when, after upper-casing, it contains any of these case-insensitive substrings: SECRET, TOKEN, PASSWORD, PASSWD, API_KEY, ACCESS_KEY, PRIVATE_KEY, CREDENTIAL, AUTH, COOKIE, SESSION, BEARER. Because the test is a substring match, names such as SESSION_DIR or AUTH_MODE also match.

Tracked Runtime

Add --remote[=<HOST>] to status, ps, stats, score, logs, or pull to run that command on the login node’s staged checkout from a prior up --remote, over SSH, streaming output back. With no value it uses the configured login_host; pass user@host to override.

hpc-compose debug -f compose.yaml
hpc-compose debug -f compose.yaml --preflight
hpc-compose jobs list
hpc-compose status -f compose.yaml --format json
hpc-compose status -f compose.yaml --array
hpc-compose status -f compose.yaml --job-id 12345_7 --array
hpc-compose ps -f compose.yaml
hpc-compose watch -f compose.yaml --watch-mode line
hpc-compose watch -f compose.yaml --hold-on-exit always
hpc-compose replay -f compose.yaml
hpc-compose replay -f compose.yaml --speed 10
hpc-compose replay -f compose.yaml --job-id 12345 --service app
hpc-compose replay -f compose.yaml --format json
hpc-compose checkpoints -f compose.yaml
hpc-compose checkpoints --job-id 12345 --format json
hpc-compose logs -f compose.yaml --service app --follow
hpc-compose logs -f compose.yaml --grep 'error|oom' --since 30m
hpc-compose inspect -f compose.yaml --rightsize
hpc-compose stats -f compose.yaml --format jsonl
hpc-compose stats -f compose.yaml --accounting --format csv
hpc-compose score 12345
hpc-compose diff 12345 12346 -f compose.yaml
hpc-compose diff --jobs 12345,12346,12347 --matrix-format json
hpc-compose diff --across sweep-1700000000-1234 --matrix-format csv
hpc-compose artifacts -f compose.yaml --bundle checkpoints --tarball
hpc-compose down -f compose.yaml --yes
hpc-compose cancel -f compose.yaml --yes
hpc-compose clean -f compose.yaml --age 7 --dry-run
hpc-compose rendezvous list
hpc-compose rendezvous resolve model-server
hpc-compose rendezvous register model-server --host node01 --port 8000 --job-id 12345
hpc-compose rendezvous prune

Cache Maintenance

hpc-compose cache list
hpc-compose cache inspect -f compose.yaml --service app
hpc-compose cache prune --age 7 --cache-dir '<shared-cache-dir>' --yes
hpc-compose cache prune --all-unused -f compose.yaml --yes

Related Docs

• Spec Reference
• Glossary
• Full Example Specs
• Roadmap and Non-Goals
• Runbook