{"record":{"id":"6584140e4b81a73d","repo":"xai-org/x-algorithm","slug":"global-checksum-of-key-key-in-fn-differs-0x-c","errorCode":null,"errorMessage":"Global checksum of key {key} in {fn} differs: 0x{cchecksum:08x} vs 0x{lchecksum:08x}","messagePattern":"Global checksum of key (.+?) in (.+?) differs: 0x(.+?) vs 0x(.+?)","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"critical","filePath":"phoenix/python/training/xai-checkpointing/xai_checkpointing/checksum.py","lineNumber":461,"sourceCode":"                    f\"{extra}: 0x{cvalue:08x} vs 0x{lvalue:08x}\"\n                )\n\n    if version == 2:\n        return success\n\n    for key, cchecksum in computed[\"global_checksums\"].items():\n        if names and key not in names:\n            continue\n\n        try:\n            lchecksum = loaded[\"global_checksums\"][key]\n        except KeyError as e:\n            if success:\n                rank_logger.warning(\"%s reading checksum file %s: %s\", e.__class__.__name__, fn, e)\n                success = False\n            continue\n        if cchecksum != lchecksum:\n            raise RuntimeError(\n                f\"Global checksum of key {key} in {fn} differs: 0x{cchecksum:08x} vs 0x{lchecksum:08x}\"\n            )\n\n    return success\n","sourceCodeStart":443,"sourceCodeEnd":466,"githubUrl":"https://github.com/xai-org/x-algorithm/blob/24c60942c5c5fdad3a6addffb4c6e6d2f228f04f/phoenix/python/training/xai-checkpointing/xai_checkpointing/checksum.py#L443-L466","documentation":"For version-2 checksum files, compare_checksum_dicts also compares a single global (whole-tensor) checksum per key. A difference raises this RuntimeError with both values in hex.","triggerScenarios":"maybe_load_checkpoint on a v2 checkpoint where the aggregate checksum of a tensor in the file differs from the computed aggregate for the in-memory counterpart — indicating bytes differ somewhere even if per-slice info is absent.","commonSituations":"Bit-level corruption of checkpoint files (storage issues), dtype/byte-order changes between save and load environments, or loading a checkpoint written by a different code version with different serialization.","solutions":["Verify file integrity (re-copy/re-download the checkpoint; check fs/obj-store health)","Ensure save and load run the same library versions and dtypes","Regenerate the checkpoint from a trusted source","If corruption is confirmed, fall back to the previous good checkpoint"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try:\n    maybe_load_checkpoint(...)\nexcept RuntimeError as e:\n    if \"Global checksum\" in str(e):\n        restore_from_backup_or_fail(e)\n    raise","preventionTips":["Store checksums/read-back files after writing","Copy checkpoints with integrity verification (rsync -c, etag checks)"],"tags":["checkpoint","checksum","data-corruption","runtime"],"backgroundTag":"checkpoint-checksum-mismatch","analyzedSha":"24c60942c5c5fdad3a6addffb4c6e6d2f228f04f","analyzedAt":"2026-08-28T11:40:14.686Z","schemaVersion":2},"datasetVersion":"2026-08-28T16:17:29.566Z"}