xai-org/x-algorithm · critical · RuntimeError

Global checksum of key {key} in {fn} differs: 0x{cchecksum:0

Error message

Global checksum of key {key} in {fn} differs: 0x{cchecksum:08x} vs 0x{lchecksum:08x}

What it means

For version-2 checksum files, compare_checksum_dicts also compares a single global (whole-tensor) checksum per key. A difference raises this RuntimeError with both values in hex.

Source

Thrown at phoenix/python/training/xai-checkpointing/xai_checkpointing/checksum.py:461

                    f"{extra}: 0x{cvalue:08x} vs 0x{lvalue:08x}"
                )

    if version == 2:
        return success

    for key, cchecksum in computed["global_checksums"].items():
        if names and key not in names:
            continue

        try:
            lchecksum = loaded["global_checksums"][key]
        except KeyError as e:
            if success:
                rank_logger.warning("%s reading checksum file %s: %s", e.__class__.__name__, fn, e)
                success = False
            continue
        if cchecksum != lchecksum:
            raise RuntimeError(
                f"Global checksum of key {key} in {fn} differs: 0x{cchecksum:08x} vs 0x{lchecksum:08x}"
            )

    return success

View on GitHub (pinned to 24c60942c5)

Solutions

1. Verify file integrity (re-copy/re-download the checkpoint; check fs/obj-store health)
2. Ensure save and load run the same library versions and dtypes
3. Regenerate the checkpoint from a trusted source
4. If corruption is confirmed, fall back to the previous good checkpoint

try:
    maybe_load_checkpoint(...)
except RuntimeError as e:
    if "Global checksum" in str(e):
        restore_from_backup_or_fail(e)
    raise

When it happens

Trigger: maybe_load_checkpoint on a v2 checkpoint where the aggregate checksum of a tensor in the file differs from the computed aggregate for the in-memory counterpart — indicating bytes differ somewhere even if per-slice info is absent.

Common situations: Bit-level corruption of checkpoint files (storage issues), dtype/byte-order changes between save and load environments, or loading a checkpoint written by a different code version with different serialization.

Related errors

• Checksums internally inconsistent: Mismatch in {fn} for tens
• Checksum of key {key} in {fn} differs{extra}: 0x{cvalue:08x}
• batch size must be a positive integer
• cannot pass empty collection to Foldl1()
• Date String Format Error:

AI-assisted analysis of xai-org/x-algorithm@24c60942c5 (2026-08-28). Data as JSON: /api/errors/6584140e4b81a73d. Report an issue: GitHub.