xai-org/x-algorithm · critical · RuntimeError

head checkpoint was trained against a different backbone — r

Error message

head checkpoint was trained against a different backbone — refusing to start

What it means

config.json in the head checkpoint stores backbone_sha256, the backbone the head was trained on. If it differs from PINNED_BACKBONE_SHA256, the head's learned weights correspond to different features than the backbone will produce, so the scorer refuses to start.

Source

Thrown at bdsm/runtime/gpu_scorer.py:72

    with open(manifest_path) as f:
        manifest = json.load(f)
    npz_path = os.path.join(args.backbone_dir, "backbone.npz")
    actual = _file_sha256(npz_path)
    if actual != PINNED_BACKBONE_SHA256 or manifest.get("file_sha256") != PINNED_BACKBONE_SHA256:
        raise RuntimeError(
            f"backbone hash mismatch: file={actual} manifest={manifest.get('file_sha256')} "
            f"pinned={PINNED_BACKBONE_SHA256} — refusing to start"
        )

    with open(os.path.join(args.head_checkpoint, "config.json")) as f:
        head_cfg = json.load(f)
    if head_cfg.get("head_registry_hash") != PINNED_HEAD_REGISTRY_HASH:
        raise RuntimeError(
            f"head registry hash {head_cfg.get('head_registry_hash')} != "
            f"{PINNED_HEAD_REGISTRY_HASH} — refusing to start"
        )
    if head_cfg.get("backbone_sha256") != PINNED_BACKBONE_SHA256:
        raise RuntimeError(
            "head checkpoint was trained against a different backbone — refusing to start"
        )
    head_names = list(head_cfg["head_names"])
    unknown = [n for n in head_names if n not in HEAD_ORDER]
    if unknown:
        raise RuntimeError(f"head names {unknown} are not in HEAD_ORDER — refusing to start")
    log.info(
        f"weights pinned OK: backbone sha {actual[:8]}…, "
        f"head registry {PINNED_HEAD_REGISTRY_HASH}, heads {head_names}"
    )
    return manifest, head_cfg, head_names


def _start_health_server(port: int, metrics: dict, config_view: dict):
    class H(BaseHTTPRequestHandler):
        def do_GET(self):
            if self.path.rstrip("/") == "/config":
                body = json.dumps(config_view, indent=2).encode()

View on GitHub (pinned to 24c60942c5)

Solutions

Check config.json backbone_sha256 in the head checkpoint vs the pinned backbone hash
Redeploy backbone and head checkpoint from the same training pipeline/release
If the backbone was intentionally upgraded, retrain the head on it and redeploy both
Verify --backbone_dir and --head_checkpoint flags point to the same release's artifacts

Example fix

# use a head checkpoint trained on the pinned backbone:
jq .backbone_sha256 /ckpt/config.json  # must equal PINNED_BACKBONE_SHA256

Defensive patterns

Strategy: validation

Validate before calling

cfg = json.load(open(f"{args.head_checkpoint}/config.json"))
assert cfg.get('backbone_sha256') == PINNED_BACKBONE_SHA256

Try / catch

try:
    _verify_pinned_weights(args)
except RuntimeError as e:
    if 'different backbone' in str(e):
        pair_head_with_backbone(); sys.exit(2)
    raise

Prevention

Retrain heads whenever the backbone changes and redeploy both
Record backbone hash in experiment metadata for traceability
Automate pairing checks in the model registry

When it happens

Trigger: Pairing a head checkpoint trained on backbone v1 with a pinned backbone v2; deploying a new backbone without retraining heads; mixing checkpoint and backbone artifacts from different releases.

Common situations: Backbone upgrade rollout that forgot head retraining; reproducibility enforcement at serve time; artifact directory mixups.

Related errors

AI-assisted analysis of xai-org/x-algorithm@24c60942c5 (2026-08-28). Data as JSON: /api/errors/98a3205bb05eacc3. Report an issue: GitHub.