xai-org/x-algorithm · critical · RuntimeError

head names {unknown} are not in HEAD_ORDER — refusing to sta

Error message

head names {unknown} are not in HEAD_ORDER — refusing to start

What it means

After hash checks, the scorer validates that every name in the checkpoint's head_names list exists in the compiled-in HEAD_ORDER. Unknown names mean the checkpoint ships heads the serving code cannot route, so startup aborts.

Source

Thrown at bdsm/runtime/gpu_scorer.py:78

            f"backbone hash mismatch: file={actual} manifest={manifest.get('file_sha256')} "
            f"pinned={PINNED_BACKBONE_SHA256} — refusing to start"
        )

    with open(os.path.join(args.head_checkpoint, "config.json")) as f:
        head_cfg = json.load(f)
    if head_cfg.get("head_registry_hash") != PINNED_HEAD_REGISTRY_HASH:
        raise RuntimeError(
            f"head registry hash {head_cfg.get('head_registry_hash')} != "
            f"{PINNED_HEAD_REGISTRY_HASH} — refusing to start"
        )
    if head_cfg.get("backbone_sha256") != PINNED_BACKBONE_SHA256:
        raise RuntimeError(
            "head checkpoint was trained against a different backbone — refusing to start"
        )
    head_names = list(head_cfg["head_names"])
    unknown = [n for n in head_names if n not in HEAD_ORDER]
    if unknown:
        raise RuntimeError(f"head names {unknown} are not in HEAD_ORDER — refusing to start")
    log.info(
        f"weights pinned OK: backbone sha {actual[:8]}…, "
        f"head registry {PINNED_HEAD_REGISTRY_HASH}, heads {head_names}"
    )
    return manifest, head_cfg, head_names


def _start_health_server(port: int, metrics: dict, config_view: dict):
    class H(BaseHTTPRequestHandler):
        def do_GET(self):
            if self.path.rstrip("/") == "/config":
                body = json.dumps(config_view, indent=2).encode()
            else:
                body = "".join(f"{k}={v}\n" for k, v in sorted(metrics.items())).encode()
            self.send_response(200)
            self.send_header("Content-Type", "text/plain")
            self.end_headers()
            self.wfile.write(body)

View on GitHub (pinned to 24c60942c5)

Solutions

Diff config.json head_names against HEAD_ORDER in the deployed scorer
Upgrade the scorer binary to a version whose HEAD_ORDER includes the new head(s)
Or re-export the checkpoint without the unknown head
Keep head registry, HEAD_ORDER, and checkpoints versioned in one release unit

Example fix

# HEAD_ORDER must contain every entry of config.json head_names:
jq .head_names /ckpt/config.json

Defensive patterns

Strategy: validation

Validate before calling

cfg = json.load(open(f"{args.head_checkpoint}/config.json"))
unknown = [n for n in cfg['head_names'] if n not in HEAD_ORDER]
assert not unknown, unknown

Type guard

def heads_supported(names: list) -> bool:
    return all(n in HEAD_ORDER for n in names)

Try / catch

try:
    _verify_pinned_weights(args)
except RuntimeError as e:
    if 'not in HEAD_ORDER' in str(e):
        upgrade_scorer_or_reexport()
    raise

Prevention

Update HEAD_ORDER and head registry atomically with head additions
Validate exported checkpoints in CI against the serving HEAD_ORDER
Keep head names in a single shared constant module

When it happens

Trigger: A checkpoint containing a newly added head while the scorer binary's HEAD_ORDER predates it; renamed heads between releases; hand-crafted checkpoint with ad-hoc head names.

Common situations: Head added to training config but serving binary not bumped; branch artifacts mixed; typo in head naming during export.

Related errors

AI-assisted analysis of xai-org/x-algorithm@24c60942c5 (2026-08-28). Data as JSON: /api/errors/58ad473ca0bb7be9. Report an issue: GitHub.