{"record":{"id":"61eb15ab1f1a504f","repo":"unslothai/unsloth","slug":"spec-name-source-changed-while-preparing-its-run","errorCode":null,"errorMessage":"{spec.name} source changed while preparing its runtime","messagePattern":"(.+?) source changed while preparing its runtime","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"studio/backend/utils/third_party_source.py","lineNumber":760,"sourceCode":"\ndef _install_runtime(runtime: Path, checkout: Path, spec: PinnedSource) -> None:\n    workspace = Path(tempfile.mkdtemp(prefix = \".runtime-\", dir = runtime.parent))\n    staging = workspace / \"runtime\"\n    staging.mkdir()\n    try:\n        if spec.source_tree_digest is not None:\n            source_manifest = _sealed_source_manifest(checkout, spec)\n            if source_manifest is None:\n                raise RuntimeError(f\"The cached {spec.name} source failed integrity validation\")\n        else:\n            source_manifest = _checkout_manifest(checkout, spec)\n        for relative, expected_digest in source_manifest.items():\n            source_file = checkout / relative\n            destination_file = staging / relative\n            destination_file.parent.mkdir(parents = True, exist_ok = True)\n            shutil.copy2(source_file, destination_file)\n            if hashlib.sha256(destination_file.read_bytes()).hexdigest() != expected_digest:\n                raise RuntimeError(f\"{spec.name} source changed while preparing its runtime\")\n        for relative, content in _generated_file_contents(spec).items():\n            destination_file = staging / relative\n            destination_file.parent.mkdir(parents = True, exist_ok = True)\n            destination_file.write_bytes(content)\n        if not _valid_runtime(staging, spec, checkout):\n            raise RuntimeError(f\"The prepared {spec.name} runtime failed integrity validation\")\n        _replace_owned_directory(staging, runtime)\n    finally:\n        _remove_owned_path(workspace)\n\n\ndef ensure_pinned_source(\n    spec: PinnedSource, *, legacy_sources: tuple[Path | str, ...] = ()\n) -> Path:\n    revision = spec.revision.lower()\n    if _REVISION_PATTERN.fullmatch(revision) is None or revision != spec.revision:\n        raise RuntimeError(f\"{spec.name} source revision must be a lowercase full Git commit\")\n    for digest in (spec.source_tree_digest, spec.runtime_tree_digest):","sourceCodeStart":742,"sourceCodeEnd":778,"githubUrl":"https://github.com/unslothai/unsloth/blob/203007d19051dcd2ae33876786d117c99f6b0368/studio/backend/utils/third_party_source.py#L742-L778","documentation":"During runtime preparation each file is copied (shutil.copy2) into staging and then re-hashed; if the staged copy's SHA-256 differs from the manifest's expected digest the process aborts. This is a TOCTOU guard: the source tree changed on disk between the manifest validation (1596) and the per-file copy — i.e. concurrent modification during preparation.","triggerScenarios":"Between _sealed_source_manifest(checkout) and the copy of some file, an external process rewrites checkout/<relative>; the copied bytes then hash differently than expected. Requires an active writer racing the preparation step.","commonSituations":"Two ensure_pinned_source/processes (or a build job) operating on the same cache concurrently; an IDE formatter or codegen task touching the cached checkout; AV/sync tools rewriting files in place while the server bootstraps.","solutions":["Ensure only one process bootstraps a given spec at a time (startup lock / file lock around ensure_pinned_source)","Purge the spec cache slice and retry once to rule out a stale-mutated tree (also see 1596)","Stop tools from writing into cache_root()/third-party-sources (permissions: make it read-only after install)","Investigate which process holds write handles on the checkout (lsof) if it recurs"],"exampleFix":"# before: concurrent bootstrap racing the copy step\nasyncio.gather(*(bootstrap(s) for s in specs))  # no serialization\n\n# after: serialize bootstrap per spec\nimport filelock\nlock = filelock.FileLock(cache_root() / f\"third-party-sources/{spec.name}.lock\")\nwith lock:\n    runtime = ensure_pinned_source(spec)","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"with filelock.FileLock(cache_root() / f\"third-party-sources/{spec.name}.lock\"):\n    try:\n        runtime = ensure_pinned_source(spec)\n    except RuntimeError as e:\n        if \"changed while preparing\" in str(e):\n            shutil.rmtree(cache_root() / \"third-party-sources\" / spec.name, ignore_errors=True)\n            runtime = ensure_pinned_source(spec)\n        else:\n            raise","preventionTips":["Serialize ensure_pinned_source per spec with a file lock so concurrent bootstraps cannot race","Never point builds, formatters, or codegen at the third-party-sources cache tree"],"tags":["race-condition","cache","integrity","toctou","concurrency"],"backgroundTag":null,"analyzedSha":"203007d19051dcd2ae33876786d117c99f6b0368","analyzedAt":"2026-08-15T02:48:39.846Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}