xai-org/x-algorithm · critical · RuntimeError

backbone hash mismatch: file={actual} manifest={manifest.get

Error message

backbone hash mismatch: file={actual} manifest={manifest.get('file_sha256')} pinned={PINNED_BACKBONE_SHA256} — refusing to start

What it means

The GPU scorer pins model weights by sha256. _verify_pinned_weights hashes backbone.npz and cross-checks MANIFEST.json's file_sha256 against the compiled-in PINNED_BACKBONE_SHA256; any disagreement aborts startup so the scoring service can never run on tampered or stale weights.

Source

Thrown at bdsm/runtime/gpu_scorer.py:59

    p.add_argument("--kafka-password", required=True)
    p.add_argument("--input-topic", default="abuse_ready_batches")
    p.add_argument("--output-topic", default="abuse_scored_results")
    p.add_argument("--kafka-group", default="bdsm-gpu-scorer")
    p.add_argument("--backbone-dir", required=True)
    p.add_argument("--head-checkpoint", required=True)
    p.add_argument("--model-version", default=PINNED_RUN_NAME)
    p.add_argument("--health-port", type=int, default=8081)
    return p.parse_args()


def _verify_pinned_weights(args) -> tuple[dict, dict, list[str]]:
    manifest_path = os.path.join(args.backbone_dir, "MANIFEST.json")
    with open(manifest_path) as f:
        manifest = json.load(f)
    npz_path = os.path.join(args.backbone_dir, "backbone.npz")
    actual = _file_sha256(npz_path)
    if actual != PINNED_BACKBONE_SHA256 or manifest.get("file_sha256") != PINNED_BACKBONE_SHA256:
        raise RuntimeError(
            f"backbone hash mismatch: file={actual} manifest={manifest.get('file_sha256')} "
            f"pinned={PINNED_BACKBONE_SHA256} — refusing to start"
        )

    with open(os.path.join(args.head_checkpoint, "config.json")) as f:
        head_cfg = json.load(f)
    if head_cfg.get("head_registry_hash") != PINNED_HEAD_REGISTRY_HASH:
        raise RuntimeError(
            f"head registry hash {head_cfg.get('head_registry_hash')} != "
            f"{PINNED_HEAD_REGISTRY_HASH} — refusing to start"
        )
    if head_cfg.get("backbone_sha256") != PINNED_BACKBONE_SHA256:
        raise RuntimeError(
            "head checkpoint was trained against a different backbone — refusing to start"
        )
    head_names = list(head_cfg["head_names"])
    unknown = [n for n in head_names if n not in HEAD_ORDER]
    if unknown:

View on GitHub (pinned to 24c60942c5)

Solutions

1. Compare sha256sum of backbone_dir/backbone.npz against PINNED_BACKBONE_SHA256 in the deployed code
2. If code was updated, redeploy the matching artifact bundle (or roll back code to the bundle's pin)
3. If the artifact is legitimately new, update PINNED_BACKBONE_SHA256 and MANIFEST.json together and rebuild
4. Check for truncated/corrupted file transfer and re-sync the backbone directory

Example fix

# verify before start:
sha256sum /models/backbone/backbone.npz  # must equal PINNED_BACKBONE_SHA256
# if not, redeploy the matching artifact bundle

import hashlib
h = hashlib.sha256(open(f"{args.backbone_dir}/backbone.npz", 'rb').read()).hexdigest()
assert h == PINNED_BACKBONE_SHA256, f"backbone drift: {h}"

try:
    _verify_pinned_weights(args)
except RuntimeError as e:
    if 'hash mismatch' in str(e):
        redeploy_artifacts(); sys.exit(2)
    raise

When it happens

Trigger: Starting the scorer with a backbone.npz that was re-saved, re-quantized, or truncated; MANIFEST.json regenerated with a different hash; code updated with a new PINNED_BACKBONE_SHA256 but the deployed backbone_dir not refreshed (or vice versa); corrupted download.

Common situations: Deployment skew between the pinned-constants build and the artifact bundle; someone hand-edited or re-packed the npz; partial file sync to the GPU host; supply-chain verification catching an unexpected artifact.

Related errors

• head registry hash {head_cfg.get('head_registry_hash')} != {
• head checkpoint was trained against a different backbone — r
• head names {unknown} are not in HEAD_ORDER — refusing to sta
• {key}: sha256 mismatch
• Did not find any files matching {file_pattern}

AI-assisted analysis of xai-org/x-algorithm@24c60942c5 (2026-08-28). Data as JSON: /api/errors/e4d49530befa9cdd. Report an issue: GitHub.