xai-org/x-algorithm · critical · RuntimeError

head registry hash {head_cfg.get('head_registry_hash')} != {

Error message

head registry hash {head_cfg.get('head_registry_hash')} != {PINNED_HEAD_REGISTRY_HASH} — refusing to start

What it means

The head checkpoint's config.json records head_registry_hash, the identity of the head registry it was trained with. The scorer requires it to equal PINNED_HEAD_REGISTRY_HASH so head-name-to-index mappings cannot drift between training and serving.

Source

Thrown at bdsm/runtime/gpu_scorer.py:67

    return p.parse_args()


def _verify_pinned_weights(args) -> tuple[dict, dict, list[str]]:
    manifest_path = os.path.join(args.backbone_dir, "MANIFEST.json")
    with open(manifest_path) as f:
        manifest = json.load(f)
    npz_path = os.path.join(args.backbone_dir, "backbone.npz")
    actual = _file_sha256(npz_path)
    if actual != PINNED_BACKBONE_SHA256 or manifest.get("file_sha256") != PINNED_BACKBONE_SHA256:
        raise RuntimeError(
            f"backbone hash mismatch: file={actual} manifest={manifest.get('file_sha256')} "
            f"pinned={PINNED_BACKBONE_SHA256} — refusing to start"
        )

    with open(os.path.join(args.head_checkpoint, "config.json")) as f:
        head_cfg = json.load(f)
    if head_cfg.get("head_registry_hash") != PINNED_HEAD_REGISTRY_HASH:
        raise RuntimeError(
            f"head registry hash {head_cfg.get('head_registry_hash')} != "
            f"{PINNED_HEAD_REGISTRY_HASH} — refusing to start"
        )
    if head_cfg.get("backbone_sha256") != PINNED_BACKBONE_SHA256:
        raise RuntimeError(
            "head checkpoint was trained against a different backbone — refusing to start"
        )
    head_names = list(head_cfg["head_names"])
    unknown = [n for n in head_names if n not in HEAD_ORDER]
    if unknown:
        raise RuntimeError(f"head names {unknown} are not in HEAD_ORDER — refusing to start")
    log.info(
        f"weights pinned OK: backbone sha {actual[:8]}…, "
        f"head registry {PINNED_HEAD_REGISTRY_HASH}, heads {head_names}"
    )
    return manifest, head_cfg, head_names

View on GitHub (pinned to 24c60942c5)

Solutions

Read head_registry_hash from head_checkpoint/config.json and compare with PINNED_HEAD_REGISTRY_HASH in the deployed code
Deploy binary and head checkpoint from the same release bundle
If the registry legitimately changed, retrain/re-export checkpoints and update the pin together
Audit for mixed-version artifacts on the host (stale dirs, wrong --head_checkpoint flag)

Example fix

# ensure the checkpoint matches the binary's pin:
jq .head_registry_hash /ckpt/config.json  # must equal PINNED_HEAD_REGISTRY_HASH

Defensive patterns

Strategy: validation

Validate before calling

cfg = json.load(open(f"{args.head_checkpoint}/config.json"))
assert cfg.get('head_registry_hash') == PINNED_HEAD_REGISTRY_HASH

Try / catch

try:
    _verify_pinned_weights(args)
except RuntimeError as e:
    if 'head registry hash' in str(e):
        rollback_to_matching_release()
    raise

Prevention

Version head registry, checkpoints, and binary together in one release
Add CI that cross-checks release artifact hashes before publish
Never mix checkpoint dirs across releases

When it happens

Trigger: Starting the scorer with a head checkpoint trained against an older/newer head registry while the binary pins a different PINNED_HEAD_REGISTRY_HASH; mixing artifacts from two releases; head registry changed (heads added/renamed) without retraining/re-exporting the checkpoint.

Common situations: Deploying a new head checkpoint built from a different branch; registry schema change making old checkpoints incompatible; partial rollout where only one of binary or checkpoint was updated.

Related errors

AI-assisted analysis of xai-org/x-algorithm@24c60942c5 (2026-08-28). Data as JSON: /api/errors/16f451363e90b4ce. Report an issue: GitHub.