xai-org/x-algorithm · critical · RuntimeError

Checksum of key {key} in {fn} differs{extra}: 0x{cvalue:08x}

Error message

Checksum of key {key} in {fn} differs{extra}: 0x{cvalue:08x} vs 0x{lvalue:08x}

What it means

compare_checksum_dicts compares per-device/per-slice checksums of each tensor between the current state and a loaded checkpoint. Any differing slice raises this RuntimeError showing both hex checksums, the device, and the slice.

Source

Thrown at phoenix/python/training/xai-checkpointing/xai_checkpointing/checksum.py:441

                ldevice = min(lslices2devices[slices])
            except KeyError:
                if success:
                    rank_logger.warning(
                        "Sharding of key %s changed from %s to %s, cannot check checksums",
                        key,
                        lsharding,
                        csharding,
                    )
                    success = False
                continue

            lvalue = lchecksums[ldevice]

            if cvalue != lvalue:
                extra = ""
                if len(cchecksums) > 1:
                    extra = f" at device {cdevice}, slice {format_slices(slices, cshape)}"
                raise RuntimeError(
                    f"Checksum of key {key} in {fn} differs"
                    f"{extra}: 0x{cvalue:08x} vs 0x{lvalue:08x}"
                )

    if version == 2:
        return success

    for key, cchecksum in computed["global_checksums"].items():
        if names and key not in names:
            continue

        try:
            lchecksum = loaded["global_checksums"][key]
        except KeyError as e:
            if success:
                rank_logger.warning("%s reading checksum file %s: %s", e.__class__.__name__, fn, e)
                success = False
            continue

View on GitHub (pinned to 24c60942c5)

Solutions

1. Verify you're loading the checkpoint that matches this config/run
2. If the difference is expected (fresh init before restore), ensure the load path fully overwrites the tensor rather than comparing pre-load state
3. Re-save or regenerate the checkpoint if it was partially written
4. Compare the global checksum (v2 path) to identify which shard is off

try:
    maybe_load_checkpoint(...)
except RuntimeError as e:
    if "Checksum of key" in str(e):
        # wrong or corrupt checkpoint; do not continue training
        raise SystemExit(f"Refusing to resume: {e}")
    raise

When it happens

Trigger: maybe_load_checkpoint loads a checkpoint whose tensor contents differ from the just-initialized state for the same key — i.e. same shapes but different bytes, which the loader treats as suspicious rather than silently overwriting semantics (used to catch loading the wrong/partially-restored checkpoint).

Common situations: Resuming with a mismatched config (different init seed/shape ordering), loading a checkpoint saved before a refactor renamed or reordered params, or partial restores where only some shards were written.

Related errors

• Checksums internally inconsistent: Mismatch in {fn} for tens
• Global checksum of key {key} in {fn} differs: 0x{cchecksum:0
• batch size must be a positive integer
• cannot pass empty collection to Foldl1()
• Date String Format Error:

AI-assisted analysis of xai-org/x-algorithm@24c60942c5 (2026-08-28). Data as JSON: /api/errors/c96725cba8bcaaa3. Report an issue: GitHub.