{"record":{"id":"c96725cba8bcaaa3","repo":"xai-org/x-algorithm","slug":"checksum-of-key-key-in-fn-differs-extra-0x-c","errorCode":null,"errorMessage":"Checksum of key {key} in {fn} differs{extra}: 0x{cvalue:08x} vs 0x{lvalue:08x}","messagePattern":"Checksum of key (.+?) in (.+?) differs(.+?): 0x(.+?) vs 0x(.+?)","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"critical","filePath":"phoenix/python/training/xai-checkpointing/xai_checkpointing/checksum.py","lineNumber":441,"sourceCode":"                ldevice = min(lslices2devices[slices])\n            except KeyError:\n                if success:\n                    rank_logger.warning(\n                        \"Sharding of key %s changed from %s to %s, cannot check checksums\",\n                        key,\n                        lsharding,\n                        csharding,\n                    )\n                    success = False\n                continue\n\n            lvalue = lchecksums[ldevice]\n\n            if cvalue != lvalue:\n                extra = \"\"\n                if len(cchecksums) > 1:\n                    extra = f\" at device {cdevice}, slice {format_slices(slices, cshape)}\"\n                raise RuntimeError(\n                    f\"Checksum of key {key} in {fn} differs\"\n                    f\"{extra}: 0x{cvalue:08x} vs 0x{lvalue:08x}\"\n                )\n\n    if version == 2:\n        return success\n\n    for key, cchecksum in computed[\"global_checksums\"].items():\n        if names and key not in names:\n            continue\n\n        try:\n            lchecksum = loaded[\"global_checksums\"][key]\n        except KeyError as e:\n            if success:\n                rank_logger.warning(\"%s reading checksum file %s: %s\", e.__class__.__name__, fn, e)\n                success = False\n            continue","sourceCodeStart":423,"sourceCodeEnd":459,"githubUrl":"https://github.com/xai-org/x-algorithm/blob/24c60942c5c5fdad3a6addffb4c6e6d2f228f04f/phoenix/python/training/xai-checkpointing/xai_checkpointing/checksum.py#L423-L459","documentation":"compare_checksum_dicts compares per-device/per-slice checksums of each tensor between the current state and a loaded checkpoint. Any differing slice raises this RuntimeError showing both hex checksums, the device, and the slice.","triggerScenarios":"maybe_load_checkpoint loads a checkpoint whose tensor contents differ from the just-initialized state for the same key — i.e. same shapes but different bytes, which the loader treats as suspicious rather than silently overwriting semantics (used to catch loading the wrong/partially-restored checkpoint).","commonSituations":"Resuming with a mismatched config (different init seed/shape ordering), loading a checkpoint saved before a refactor renamed or reordered params, or partial restores where only some shards were written.","solutions":["Verify you're loading the checkpoint that matches this config/run","If the difference is expected (fresh init before restore), ensure the load path fully overwrites the tensor rather than comparing pre-load state","Re-save or regenerate the checkpoint if it was partially written","Compare the global checksum (v2 path) to identify which shard is off"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try:\n    maybe_load_checkpoint(...)\nexcept RuntimeError as e:\n    if \"Checksum of key\" in str(e):\n        # wrong or corrupt checkpoint; do not continue training\n        raise SystemExit(f\"Refusing to resume: {e}\")\n    raise","preventionTips":["Pin configs to checkpoints (record config hash with the checkpoint)","Validate checksums right after loading, before training"],"tags":["checkpoint","checksum","distributed","runtime"],"backgroundTag":"checkpoint-checksum-mismatch","analyzedSha":"24c60942c5c5fdad3a6addffb4c6e6d2f228f04f","analyzedAt":"2026-08-28T11:40:14.686Z","schemaVersion":2},"datasetVersion":"2026-08-28T16:17:29.566Z"}