Files and Directories

hpc-compose writes to three independent on-disk roots, and keeping them separate is deliberate. Compose-level metadata lives next to the compose file so tracked records travel with your project; per-job runtime state lives under a per-job runtime root resolved at submit time; and the cache is a content-addressed store shared across jobs and visible from both the login node and the compute nodes. src/tracked_paths.rs is the single source of truth for every leaf name documented here, so the layout below matches what tooling reads and writes exactly.

The three roots at a glance

The metadata directory and the default per-job runtime root share the same .hpc-compose/ directory name, but they are addressed independently: the metadata root is anchored to the compose file’s directory, while the runtime root is anchored to the submit directory (and is overridable). They coincide only when you submit from the directory that holds the compose file and leave x-slurm.runtime_root unset.

Metadata directory

The metadata directory sits next to the compose file (metadata_root_for joins .hpc-compose onto the compose file’s parent). It holds the durable record of every submission plus the latest-pointers that let follow-up commands reconnect without resubmitting.

<compose-file-dir>/.hpc-compose/
├── latest.json              # most recent `up` (main) submission record
├── latest-run.json          # most recent `run` submission record
├── latest-canary.json       # most recent `germinate` canary record
├── latest-notebook.json     # most recent `notebook` server record
├── jobs/
│   └── <job-id>.json        # one tracked SubmissionRecord per submitted job
└── sweeps/
    ├── latest.json          # most recent sweep manifest pointer
    └── <sweep-id>/
        └── sweep.json       # per-sweep manifest

A SubmissionRecord carries the paths the runtime root resolves to, including runtime_root (the resolved x-slurm.runtime_root override, present only when set), batch_log, batch_log_managed, and service_logs (the authoritative service-name to log-path map; see Log lifecycle). The current SubmissionRecord schema version is 3. Records written by schema 3 persist the runtime_root override when one was set; older records that lack the field fall back to the default <submit-dir>/.hpc-compose layout when read.

Per-job runtime root

Each job gets its own runtime root: <runtime-root>/<job-id>/, where <runtime-root> defaults to <submit-dir>/.hpc-compose (runtime_root_for) and is overridable with x-slurm.runtime_root (resolve_runtime_root). The renderer resolves this to an absolute path at submit time and bakes it into the rendered JOB_ROOT, so a running job never depends on $SLURM_SUBMIT_DIR being set or shared-visible at compute-node runtime. A relative x-slurm.runtime_root resolves against the submit directory; an absolute one is used as-is.

<runtime-root>/
├── logs/
│   ├── hpc-compose-%j.out        # default batch log (job-id, Slurm-expanded)
│   └── <service-token>.log       # one log per service (see Log lifecycle)
└── <job-id>/
    ├── state.json                # job state snapshot (latest view)
    ├── logs/
    │   └── <service-token>.log   # per-service logs, latest attempt
    ├── metrics/
    │   ├── meta.json
    │   ├── gpu.jsonl
    │   ├── gpu_processes.jsonl
    │   ├── slurm.jsonl
    │   ├── diagnostics/
    │   └── gpu-node-samples/
    ├── artifacts/
    │   ├── manifest.json
    │   └── payload/
    ├── allocation/
    │   ├── primary_node
    │   ├── nodes.txt
    │   ├── service-nodelists/
    │   ├── mpi-hostfiles/
    │   └── distributed-hostfiles/
    ├── service-exits/
    ├── hooks/
    └── attempts/                 # resume-aware runs only
        └── <n>/                  # logs/, metrics/, artifacts/, state.json per attempt

The batch script keeps the root-level logs/, metrics/, artifacts/, and state.json as the “latest” view (it updates them to point at the most recent attempt) so status and export commands read the latest attempt without reconstructing shell logic.

Default batch log location

When you do not set x-slurm.output, real submissions get a baked --output directive at <runtime-root>/logs/hpc-compose-%j.out. Note that this parent is job-id-free (<runtime-root>/logs/, not under <runtime-root>/<job-id>/), because Slurm opens --output before the script body runs, so the CLI pre-creates that directory host-side before sbatch. The default basename deliberately avoids %x so a raw job name can never become a path component; %j is expanded by Slurm. Setting x-slurm.output replaces this default entirely. Dry-run previews (inspect, render) keep the portable Slurm default instead of a baked absolute path so committed example renders stay machine-independent.

Cache directory

The cache directory defaults to $HOME/.cache/hpc-compose/ and is set with x-slurm.cache_dir (resolved with the precedence documented in Spec Reference). It must be visible from both the login node and the compute nodes. Image artifacts are content-addressed: the filename embeds a short hash of the cache key, so identical inputs reuse the same artifact across jobs and machines.

<cache_dir>/
├── base/
│   ├── <hash>-<label>.sqsh        # imported base image
│   ├── <hash>-<label>.sqsh.json   # manifest sidecar
│   └── <hash>-<label>.sqsh.json.lock  # advisory-lock sidecar
├── prepared/
│   ├── <hash>-<name>.sqsh         # prepared runtime image
│   └── <hash>-<name>.sqsh.json    # manifest sidecar
├── enroot/                        # login-node shared enroot store
│   ├── cache/
│   ├── data/
│   └── tmp/
├── runtime/
│   └── <job-id>/                  # per-job compute-node enroot runtime cache
│       ├── cache/                 # ENROOT_CACHE_PATH
│       ├── data/                  # ENROOT_DATA_PATH
│       └── tmp/                   # ENROOT_TEMP_PATH
└── rendezvous/
    └── <name>/
        ├── latest.json            # current provider for this rendezvous name
        └── <token>.json           # historical per-registration records

Manifest .lock sidecars carry no data and only serialize writers; the manifest JSON next to each artifact is the persisted record. See Connect Jobs Across Allocations for how rendezvous records are produced and resolved.

Repo staging vs cluster workspace provisioning

The three roots above are written by hpc-compose itself. They are not the same as the cluster workspaces and site storage directories your job reads and writes — those you provision yourself.

When you submit from a laptop with hpc-compose up --remote, the project is first staged to a per-project directory on the login node:

~/.hpc-compose-remote/<project>/      # rsync'd copy of your settings base on the login node

The staged root is the settings base: the directory that contains .hpc-compose/settings.toml. Keep that file at the repo root so your whole source tree is staged. If your compose file sits in a subdirectory with no repo-root settings file, only that subdirectory is staged and the rest of your tree is hidden from the job (hpc-compose warns when it stages only a subdir). The stage includes project settings (.hpc-compose/settings.toml, .hpc-compose/cluster.toml) but excludes tracked job/runtime state. See Submit From Your Laptop With up --remote.

Staging copies your repo. It does not allocate cluster workspaces (for example ws_allocate) or create site storage directories. You must create cache, dataset, checkpoint, and other site storage paths yourself before the run — a missing host bind-mount or storage directory blocks preflight.

Preflight remediation reflects this boundary. For a relative or in-repo missing path it tells you to create the directory; for an absolute missing path it notes that the path may be a cluster workspace or site storage location and should be provisioned with your site’s allocation command (for example ws_allocate) or an x-slurm.setup step, because hpc-compose stages your repo but does not allocate workspaces or create site storage directories.

Bootstrapping required directories

x-slurm.setup is the declarative bootstrap phase: its commands run on the allocated node before any service starts, so it is the right place to create the cache/data/results sub-directories your bind mounts expect. Allocate (or look up) the workspace first, then create the layout declaratively:

x-slurm:
  setup:
    # $WORKSPACE is resolved on the node (e.g. exported by an earlier step or your shell rc);
    # ws_allocate / ws_find belong in your session, not here, because they allocate quota.
    - mkdir -p "$WORKSPACE"/{cache,data,results,runtime}

For in-repo directories (relative bind-mount sources such as ./results), commit them with a .gitkeep so they exist and are staged, rather than relying on them being created at runtime. Use absolute cluster paths for large/scratch data that should not be staged, and relative in-repo paths for small inputs that travel with the project.

Excluding files from staging (.hpcignore)

A repo-root .hpcignore adds extra excludes on top of .gitignore when the source tree is snapshotted (for up, prepare, and up --remote). It uses gitignore-style patterns, so anchoring matters:

• An unanchored directory pattern like data/ matches that name at any depth — including a Python package subtree such as src/mypackage/data/. Excluding package source there causes ModuleNotFoundError at runtime.
• Anchor artifact patterns to the repo root with a leading slash — /data/, /runs/, /results/ — so they only match the top-level artifact directories and never a nested package.

hpc-compose warns when .hpcignore excludes any .py file (the usual symptom of this mistake). To see exactly what an .hpcignore removes from the snapshot, set HPC_COMPOSE_DEBUG_STAGING=1, which lists every excluded path during staging.

Environment variables that affect paths

hpc-compose both reads some path-affecting variables from your environment and sets others into the running job. The table below consolidates the relevant ones.

During login-node prepare the same enroot variables are pointed at the shared <cache_dir>/enroot/{cache,data,tmp} store rather than the per-job runtime/<job-id> store. The persistent layer cache (ENROOT_CACHE_PATH) always stays under cache_dir, but the temporary extraction scratch (ENROOT_TEMP_PATH) — and, when that scratch is redirected, the transient prepare rootfs (ENROOT_DATA_PATH, where enroot create unsquashes the image before the prepared .sqsh is exported) — can be moved to fast node-local storage together. By default the scratch stays at <cache_dir>/enroot/tmp; opt in by setting x-slurm.enroot_temp_dir in the spec (interpolation-aware, e.g. /tmp/${USER}-hpc-compose-enroot), cache.enroot_temp_dir in .hpc-compose/settings.toml (project-wide default, mirroring cache.dir), or the HPC_COMPOSE_ENROOT_TEMP_DIR environment variable. Precedence is HPC_COMPOSE_ENROOT_TEMP_DIR > x-slurm.enroot_temp_dir > cache.enroot_temp_dir > the <cache_dir>/enroot/tmp default. When the scratch is left at its default the prepare rootfs stays on the shared cache (<cache_dir>/enroot/data); redirecting the scratch moves both the extraction scratch and the transient rootfs to an hpc-compose-owned per-process subdir under the node-local path. This matters on shared NFS/Lustre/GPFS home/work storage, where the extract-then-mksquashfs import and the unsquashfs create step are slow and can fail with Stale file handle (ESTALE); pointing the scratch at node-local /tmp keeps the final .sqsh and layer cache on the shared cache while extraction and rootfs creation happen locally. The override applies to prepare-time import only, not the compute-node runtime. hpc-compose preflight surfaces the resolved enroot temp path, and hpc-compose context shows the settings-level value. The full set of HPC_COMPOSE_* runtime variables injected into services (distributed, rendezvous, MPI, scratch, and hook variables) is described in Monitor a Run and the feature guides.

Cleanup scope

Different commands reap different subsets of these roots. The table is precise about what each one deletes and what it leaves intact.

Two things to keep in mind: tracked metadata records live next to the compose file while the managed default batch log lives under <runtime-root>/logs/, so cleanup uses the persisted record to remove only the log hpc-compose owns; and the per-job enroot dir is namespaced by job id, so reaping it can never touch the shared cache root or another job’s runtime cache.

Log lifecycle

The default batch log (sbatch stdout/stderr) is <runtime-root>/logs/hpc-compose-%j.out unless you set x-slurm.output (see Default batch log location).

Service logs are written one-per-service under <job-id>/logs/. The filename is produced by a reversible token encoding of the service name: each non-alphanumeric byte becomes an _x{hh}_ hex sequence. For example, db.primary (the . is byte 0x2e) becomes db_x2e_primary.log. Do not parse these filenames by hand; the authoritative service-name to log-path map is SubmissionRecord.service_logs, which logs, watch, and replay read.

For resume-aware runs, each attempt’s logs and state are preserved under attempts/<n>/, while the root-level logs//state.json track the latest attempt.

Automatic size-based log rotation is not yet implemented. There is no x-slurm.logs key; cap log volume from inside your service command (for example by limiting verbosity or rotating within your own process) if a long-running service can produce unbounded output.

Related Docs

• Spec Reference
• Architecture for Contributors
• Monitor a Run
• Manage the Cache and Clean Up
• Operate a Real Cluster Run