unslothai/unsloth · error · RuntimeError

{spec.name} source changed while preparing its runtime

Error message

{spec.name} source changed while preparing its runtime

What it means

During runtime preparation each file is copied (shutil.copy2) into staging and then re-hashed; if the staged copy's SHA-256 differs from the manifest's expected digest the process aborts. This is a TOCTOU guard: the source tree changed on disk between the manifest validation (1596) and the per-file copy — i.e. concurrent modification during preparation.

Source

Thrown at studio/backend/utils/third_party_source.py:760

def _install_runtime(runtime: Path, checkout: Path, spec: PinnedSource) -> None:
    workspace = Path(tempfile.mkdtemp(prefix = ".runtime-", dir = runtime.parent))
    staging = workspace / "runtime"
    staging.mkdir()
    try:
        if spec.source_tree_digest is not None:
            source_manifest = _sealed_source_manifest(checkout, spec)
            if source_manifest is None:
                raise RuntimeError(f"The cached {spec.name} source failed integrity validation")
        else:
            source_manifest = _checkout_manifest(checkout, spec)
        for relative, expected_digest in source_manifest.items():
            source_file = checkout / relative
            destination_file = staging / relative
            destination_file.parent.mkdir(parents = True, exist_ok = True)
            shutil.copy2(source_file, destination_file)
            if hashlib.sha256(destination_file.read_bytes()).hexdigest() != expected_digest:
                raise RuntimeError(f"{spec.name} source changed while preparing its runtime")
        for relative, content in _generated_file_contents(spec).items():
            destination_file = staging / relative
            destination_file.parent.mkdir(parents = True, exist_ok = True)
            destination_file.write_bytes(content)
        if not _valid_runtime(staging, spec, checkout):
            raise RuntimeError(f"The prepared {spec.name} runtime failed integrity validation")
        _replace_owned_directory(staging, runtime)
    finally:
        _remove_owned_path(workspace)


def ensure_pinned_source(
    spec: PinnedSource, *, legacy_sources: tuple[Path | str, ...] = ()
) -> Path:
    revision = spec.revision.lower()
    if _REVISION_PATTERN.fullmatch(revision) is None or revision != spec.revision:
        raise RuntimeError(f"{spec.name} source revision must be a lowercase full Git commit")
    for digest in (spec.source_tree_digest, spec.runtime_tree_digest):

View on GitHub (pinned to 203007d190)

Solutions

1. Ensure only one process bootstraps a given spec at a time (startup lock / file lock around ensure_pinned_source)
2. Purge the spec cache slice and retry once to rule out a stale-mutated tree (also see 1596)
3. Stop tools from writing into cache_root()/third-party-sources (permissions: make it read-only after install)
4. Investigate which process holds write handles on the checkout (lsof) if it recurs

Example fix

# before: concurrent bootstrap racing the copy step
asyncio.gather(*(bootstrap(s) for s in specs))  # no serialization

# after: serialize bootstrap per spec
import filelock
lock = filelock.FileLock(cache_root() / f"third-party-sources/{spec.name}.lock")
with lock:
    runtime = ensure_pinned_source(spec)

with filelock.FileLock(cache_root() / f"third-party-sources/{spec.name}.lock"):
    try:
        runtime = ensure_pinned_source(spec)
    except RuntimeError as e:
        if "changed while preparing" in str(e):
            shutil.rmtree(cache_root() / "third-party-sources" / spec.name, ignore_errors=True)
            runtime = ensure_pinned_source(spec)
        else:
            raise

When it happens

Trigger: Between _sealed_source_manifest(checkout) and the copy of some file, an external process rewrites checkout/<relative>; the copied bytes then hash differently than expected. Requires an active writer racing the preparation step.

Common situations: Two ensure_pinned_source/processes (or a build job) operating on the same cache concurrently; an IDE formatter or codegen task touching the cached checkout; AV/sync tools rewriting files in place while the server bootstraps.

Related errors

• The diffusion engine changed while this load was starting. R
• The stable-diffusion.cpp server binary was replaced by an in
• The stable-diffusion.cpp binary was replaced by an install f
• The dictation model changed while this recording was being p
• Linked folder changed after it was selected

AI-assisted analysis of unslothai/unsloth@203007d190 (2026-08-15). Data as JSON: /api/errors/61eb15ab1f1a504f. Report an issue: GitHub.