{"record":{"id":"0d2d6ec79e4604e0","repo":"sgl-project/sglang","slug":"integrity-check-failed-joined-errors","errorCode":null,"errorMessage":"Integrity check failed: {joined errors}","messagePattern":"Integrity check failed: (.+?)","errorType":"exception","errorClass":"IntegrityError","httpStatus":null,"severity":"critical","filePath":"python/sglang/srt/utils/model_file_verifier.py","lineNumber":98,"sourceCode":"        max_workers=max_workers,\n    )\n    _compare_manifests(expected=expected, actual=actual)\n    print(f\"[ModelFileVerifier] All {len(expected.files)} files verified successfully.\")\n\n\ndef _compare_manifests(*, expected: Manifest, actual: Manifest) -> None:\n    errors = []\n    for filename, exp in expected.files.items():\n        if filename not in actual.files:\n            errors.append(f\"{filename}: missing (expected size={exp.size})\")\n        elif actual.files[filename].sha256 != exp.sha256:\n            act = actual.files[filename]\n            errors.append(\n                f\"{filename}: mismatch (expected={exp.sha256[:16]}... size={exp.size}, actual={act.sha256[:16]}... size={act.size})\"\n            )\n\n    if errors:\n        raise IntegrityError(\"Integrity check failed: \" + \"; \".join(errors))\n\n\n# ======== Generate ========\n\n\ndef generate_checksums(\n    *, source: str, output_path: str, max_workers: int = 4\n) -> Manifest:\n    if Path(source).is_dir():\n        model_path = Path(source).resolve()\n        files = _discover_files(model_path)\n        if not files:\n            raise IntegrityError(f\"No model files found in {model_path}\")\n        manifest = _compute_manifest_from_folder(\n            model_path=model_path, filenames=files, max_workers=max_workers\n        )\n    else:\n        manifest = Manifest(files=_load_file_infos_from_hf(repo_id=source))","sourceCodeStart":80,"sourceCodeEnd":116,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/utils/model_file_verifier.py#L80-L116","documentation":"The model file verifier compares a stored checksum manifest against freshly computed per-file SHA-256 hashes and sizes. When any file's hash or size differs, or a listed file is missing, it raises IntegrityError with a joined per-file mismatch report. This indicates bit rot or tampering in the model directory or HF repo.","triggerScenarios":"Running verify() where a manifest lists file X with sha256/size but the actual file on disk has different content (corrupted download, partial write, disk corruption, or a since-updated HF file).","commonSituations":"Interrupted model downloads leaving truncated safetensors, silent disk/NFS corruption on large checkpoints, or a manifest generated from an older revision of the HF repo being checked against a newer one.","solutions":["Re-download the affected files (huggingface-cli download --force-reinstall or delete and re-pull the shard) and re-verify.","If the repo revision legitimately changed, regenerate the checksum manifest via generate_checksums and use the new one.","Check filesystem/disk health (dmesg, SMART) if corruption recurs — recurring mismatches point to hardware."],"exampleFix":"# before (stale manifest)\nverify(manifest_path=\"old.json\", source=\"/models/llama\")\n\n# after\n# regenerate after intentional model update\ngenerate_checksums(source=\"/models/llama\", output_path=\"new.json\")\nverify(manifest_path=\"new.json\", source=\"/models/llama\")","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"from sglang.srt.utils.model_file_verifier import IntegrityError\ntry:\n    verify(manifest_path=mf, source=path)\nexcept IntegrityError as e:\n    # quarantine + re-download affected shards listed in str(e)\n    logging.error(\"model corrupted: %s\", e)","preventionTips":["Generate and store manifests right after download.","Run verify() before serving a checkpoint fetched over flaky networks."],"tags":["integrity","checksum","model-files"],"backgroundTag":"file-checksum-mismatch","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}