{"record":{"id":"98a3205bb05eacc3","repo":"xai-org/x-algorithm","slug":"head-checkpoint-was-trained-against-a-different-ba","errorCode":null,"errorMessage":"head checkpoint was trained against a different backbone — refusing to start","messagePattern":"head checkpoint was trained against a different backbone — refusing to start","errorType":"validation","errorClass":"RuntimeError","httpStatus":null,"severity":"critical","filePath":"bdsm/runtime/gpu_scorer.py","lineNumber":72,"sourceCode":"    with open(manifest_path) as f:\n        manifest = json.load(f)\n    npz_path = os.path.join(args.backbone_dir, \"backbone.npz\")\n    actual = _file_sha256(npz_path)\n    if actual != PINNED_BACKBONE_SHA256 or manifest.get(\"file_sha256\") != PINNED_BACKBONE_SHA256:\n        raise RuntimeError(\n            f\"backbone hash mismatch: file={actual} manifest={manifest.get('file_sha256')} \"\n            f\"pinned={PINNED_BACKBONE_SHA256} — refusing to start\"\n        )\n\n    with open(os.path.join(args.head_checkpoint, \"config.json\")) as f:\n        head_cfg = json.load(f)\n    if head_cfg.get(\"head_registry_hash\") != PINNED_HEAD_REGISTRY_HASH:\n        raise RuntimeError(\n            f\"head registry hash {head_cfg.get('head_registry_hash')} != \"\n            f\"{PINNED_HEAD_REGISTRY_HASH} — refusing to start\"\n        )\n    if head_cfg.get(\"backbone_sha256\") != PINNED_BACKBONE_SHA256:\n        raise RuntimeError(\n            \"head checkpoint was trained against a different backbone — refusing to start\"\n        )\n    head_names = list(head_cfg[\"head_names\"])\n    unknown = [n for n in head_names if n not in HEAD_ORDER]\n    if unknown:\n        raise RuntimeError(f\"head names {unknown} are not in HEAD_ORDER — refusing to start\")\n    log.info(\n        f\"weights pinned OK: backbone sha {actual[:8]}…, \"\n        f\"head registry {PINNED_HEAD_REGISTRY_HASH}, heads {head_names}\"\n    )\n    return manifest, head_cfg, head_names\n\n\ndef _start_health_server(port: int, metrics: dict, config_view: dict):\n    class H(BaseHTTPRequestHandler):\n        def do_GET(self):\n            if self.path.rstrip(\"/\") == \"/config\":\n                body = json.dumps(config_view, indent=2).encode()","sourceCodeStart":54,"sourceCodeEnd":90,"githubUrl":"https://github.com/xai-org/x-algorithm/blob/24c60942c5c5fdad3a6addffb4c6e6d2f228f04f/bdsm/runtime/gpu_scorer.py#L54-L90","documentation":"config.json in the head checkpoint stores backbone_sha256, the backbone the head was trained on. If it differs from PINNED_BACKBONE_SHA256, the head's learned weights correspond to different features than the backbone will produce, so the scorer refuses to start.","triggerScenarios":"Pairing a head checkpoint trained on backbone v1 with a pinned backbone v2; deploying a new backbone without retraining heads; mixing checkpoint and backbone artifacts from different releases.","commonSituations":"Backbone upgrade rollout that forgot head retraining; reproducibility enforcement at serve time; artifact directory mixups.","solutions":["Check config.json backbone_sha256 in the head checkpoint vs the pinned backbone hash","Redeploy backbone and head checkpoint from the same training pipeline/release","If the backbone was intentionally upgraded, retrain the head on it and redeploy both","Verify --backbone_dir and --head_checkpoint flags point to the same release's artifacts"],"exampleFix":"# use a head checkpoint trained on the pinned backbone:\njq .backbone_sha256 /ckpt/config.json  # must equal PINNED_BACKBONE_SHA256","handlingStrategy":"validation","validationCode":"cfg = json.load(open(f\"{args.head_checkpoint}/config.json\"))\nassert cfg.get('backbone_sha256') == PINNED_BACKBONE_SHA256","typeGuard":null,"tryCatchPattern":"try:\n    _verify_pinned_weights(args)\nexcept RuntimeError as e:\n    if 'different backbone' in str(e):\n        pair_head_with_backbone(); sys.exit(2)\n    raise","preventionTips":["Retrain heads whenever the backbone changes and redeploy both","Record backbone hash in experiment metadata for traceability","Automate pairing checks in the model registry"],"tags":["python","model-integrity","backbone","checkpoint"],"backgroundTag":"artifact-checksum-mismatch","analyzedSha":"24c60942c5c5fdad3a6addffb4c6e6d2f228f04f","analyzedAt":"2026-08-28T11:40:14.686Z","schemaVersion":2},"datasetVersion":"2026-08-28T16:17:29.566Z"}