{"record":{"id":"cfbf31a03b148646","repo":"xai-org/x-algorithm","slug":"checksums-internally-inconsistent-mismatch-in-fn","errorCode":null,"errorMessage":"Checksums internally inconsistent: Mismatch in {fn} for tensor {key!r} ({shape=}).\n{errors}","messagePattern":"Checksums internally inconsistent: Mismatch in (.+?) for tensor (.+?) \\((.+?)\\)\\.\n(.+?)","errorType":"exception","errorClass":"ChecksumInconsistencyError","httpStatus":null,"severity":"critical","filePath":"phoenix/python/training/xai-checkpointing/xai_checkpointing/checksum.py","lineNumber":314,"sourceCode":"        parts.append(f\"{start}:{stop}\")\n    return \"[%s]\" % \", \".join(parts)\n\n\ndef check_consistent(slices2devices, checksums, fn, key, shape):\n    errors = []\n    for slices, devices in slices2devices.items():\n        subset = [checksums[device] for device in devices]\n        values = {}\n        for device, checksum in zip(devices, subset, strict=True):\n            values.setdefault(f\"0x{checksum:08x}\", []).append(device)\n        if len(values) > 1:\n            values = {adler: contract(ranks) for adler, ranks in values.items()}\n            errors.append(\n                f\"In slice {format_slices(slices, shape)}: Seeing values {values} across {len(devices)} devices.\"\n            )\n    errors = \"\\n\".join(errors)\n    if errors:\n        raise ChecksumInconsistencyError(\n            \"Checksums internally inconsistent: \"\n            f\"Mismatch in {fn} for tensor {key!r} ({shape=}).\\n{errors}\"\n        )\n\n\ndef check_internal_consistency(checksum_dict, fn=\"<in-memory checksum_dict>\", names=None):\n    shard_checksums = checksum_dict.get(\"shard_checksums\")\n    if not shard_checksums:\n        return\n\n    for key, checksums in shard_checksums.items():\n        if names is not None and key not in names:\n            continue\n\n        sharding = checksum_dict[\"shardings\"][key]\n        mesh = sharding.get(\"mesh\")\n        if not mesh:\n            continue","sourceCodeStart":296,"sourceCodeEnd":332,"githubUrl":"https://github.com/xai-org/x-algorithm/blob/24c60942c5c5fdad3a6addffb4c6e6d2f228f04f/phoenix/python/training/xai-checkpointing/xai_checkpointing/checksum.py#L296-L332","documentation":"check_consistent verifies that checksums of the same tensor agree across devices/slices within a single checkpoint or in-memory dict. Disagreements are collected and raised as ChecksumInconsistencyError, naming the file, tensor key, shape, and per-slice value differences.","triggerScenarios":"Running check_internal_consistency/check_consistency_python/compare_checksum_dicts on a checkpoint where different ranks wrote different values for the same parameter slice — e.g. ranks diverged numerically before the write, or a corrupted/partial write interleaved ranks.","commonSituations":"Silent NCCL/comm failures producing divergent replicas, a job restarted mid-write leaving a torn checkpoint, or bugs causing non-deterministic updates on some ranks.","solutions":["Re-run check on a known-good older checkpoint to isolate corruption vs divergence","Re-save the checkpoint from a consistent state (resume from last good checkpoint)","Check NCCL/interconnect logs and set deterministic flags (e.g. XLA/JAX determinism) before retraining","If only one tensor mismatches, inspect that parameter's update path for rank-dependent logic"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"from xai_checkpointing.checksum import ChecksumInconsistencyError\ntry:\n    check_internal_consistency(ck)\nexcept ChecksumInconsistencyError as e:\n    logger.critical(\"Divergent/corrupt checkpoint: %s\", e)\n    resume_from(last_good_checkpoint)","preventionTips":["Enable determinism flags before training","Verify consistency immediately after every save","Keep multiple generations of checkpoints"],"tags":["checkpoint","checksum","distributed","data-corruption"],"backgroundTag":"checkpoint-checksum-mismatch","analyzedSha":"24c60942c5c5fdad3a6addffb4c6e6d2f228f04f","analyzedAt":"2026-08-28T11:40:14.686Z","schemaVersion":2},"datasetVersion":"2026-08-28T16:17:29.566Z"}