xai-org/x-algorithm · critical · ChecksumInconsistencyError

Checksums internally inconsistent: Mismatch in {fn} for tens

Error message

Checksums internally inconsistent: Mismatch in {fn} for tensor {key!r} ({shape=}).
{errors}

What it means

check_consistent verifies that checksums of the same tensor agree across devices/slices within a single checkpoint or in-memory dict. Disagreements are collected and raised as ChecksumInconsistencyError, naming the file, tensor key, shape, and per-slice value differences.

Source

Thrown at phoenix/python/training/xai-checkpointing/xai_checkpointing/checksum.py:314

        parts.append(f"{start}:{stop}")
    return "[%s]" % ", ".join(parts)


def check_consistent(slices2devices, checksums, fn, key, shape):
    errors = []
    for slices, devices in slices2devices.items():
        subset = [checksums[device] for device in devices]
        values = {}
        for device, checksum in zip(devices, subset, strict=True):
            values.setdefault(f"0x{checksum:08x}", []).append(device)
        if len(values) > 1:
            values = {adler: contract(ranks) for adler, ranks in values.items()}
            errors.append(
                f"In slice {format_slices(slices, shape)}: Seeing values {values} across {len(devices)} devices."
            )
    errors = "\n".join(errors)
    if errors:
        raise ChecksumInconsistencyError(
            "Checksums internally inconsistent: "
            f"Mismatch in {fn} for tensor {key!r} ({shape=}).\n{errors}"
        )


def check_internal_consistency(checksum_dict, fn="<in-memory checksum_dict>", names=None):
    shard_checksums = checksum_dict.get("shard_checksums")
    if not shard_checksums:
        return

    for key, checksums in shard_checksums.items():
        if names is not None and key not in names:
            continue

        sharding = checksum_dict["shardings"][key]
        mesh = sharding.get("mesh")
        if not mesh:
            continue

View on GitHub (pinned to 24c60942c5)

Solutions

1. Re-run check on a known-good older checkpoint to isolate corruption vs divergence
2. Re-save the checkpoint from a consistent state (resume from last good checkpoint)
3. Check NCCL/interconnect logs and set deterministic flags (e.g. XLA/JAX determinism) before retraining
4. If only one tensor mismatches, inspect that parameter's update path for rank-dependent logic

from xai_checkpointing.checksum import ChecksumInconsistencyError
try:
    check_internal_consistency(ck)
except ChecksumInconsistencyError as e:
    logger.critical("Divergent/corrupt checkpoint: %s", e)
    resume_from(last_good_checkpoint)

When it happens

Trigger: Running check_internal_consistency/check_consistency_python/compare_checksum_dicts on a checkpoint where different ranks wrote different values for the same parameter slice — e.g. ranks diverged numerically before the write, or a corrupted/partial write interleaved ranks.

Common situations: Silent NCCL/comm failures producing divergent replicas, a job restarted mid-write leaving a torn checkpoint, or bugs causing non-deterministic updates on some ranks.

Related errors

• Checksum of key {key} in {fn} differs{extra}: 0x{cvalue:08x}
• Global checksum of key {key} in {fn} differs: 0x{cchecksum:0
• head registry hash {head_cfg.get('head_registry_hash')} != {
• head checkpoint was trained against a different backbone — r
• unexpected head param layout {sorted(params)} (expected {sor

AI-assisted analysis of xai-org/x-algorithm@24c60942c5 (2026-08-28). Data as JSON: /api/errors/cfbf31a03b148646. Report an issue: GitHub.