sgl-project/sglang · critical · IntegrityError

Integrity check failed: {joined errors}

Error message

Integrity check failed: {joined errors}

What it means

The model file verifier compares a stored checksum manifest against freshly computed per-file SHA-256 hashes and sizes. When any file's hash or size differs, or a listed file is missing, it raises IntegrityError with a joined per-file mismatch report. This indicates bit rot or tampering in the model directory or HF repo.

Source

Thrown at python/sglang/srt/utils/model_file_verifier.py:98

        max_workers=max_workers,
    )
    _compare_manifests(expected=expected, actual=actual)
    print(f"[ModelFileVerifier] All {len(expected.files)} files verified successfully.")


def _compare_manifests(*, expected: Manifest, actual: Manifest) -> None:
    errors = []
    for filename, exp in expected.files.items():
        if filename not in actual.files:
            errors.append(f"{filename}: missing (expected size={exp.size})")
        elif actual.files[filename].sha256 != exp.sha256:
            act = actual.files[filename]
            errors.append(
                f"{filename}: mismatch (expected={exp.sha256[:16]}... size={exp.size}, actual={act.sha256[:16]}... size={act.size})"
            )

    if errors:
        raise IntegrityError("Integrity check failed: " + "; ".join(errors))


# ======== Generate ========


def generate_checksums(
    *, source: str, output_path: str, max_workers: int = 4
) -> Manifest:
    if Path(source).is_dir():
        model_path = Path(source).resolve()
        files = _discover_files(model_path)
        if not files:
            raise IntegrityError(f"No model files found in {model_path}")
        manifest = _compute_manifest_from_folder(
            model_path=model_path, filenames=files, max_workers=max_workers
        )
    else:
        manifest = Manifest(files=_load_file_infos_from_hf(repo_id=source))

View on GitHub (pinned to 0132848349)

Solutions

1. Re-download the affected files (huggingface-cli download --force-reinstall or delete and re-pull the shard) and re-verify.
2. If the repo revision legitimately changed, regenerate the checksum manifest via generate_checksums and use the new one.
3. Check filesystem/disk health (dmesg, SMART) if corruption recurs — recurring mismatches point to hardware.

Example fix

# before (stale manifest)
verify(manifest_path="old.json", source="/models/llama")

# after
# regenerate after intentional model update
generate_checksums(source="/models/llama", output_path="new.json")
verify(manifest_path="new.json", source="/models/llama")

from sglang.srt.utils.model_file_verifier import IntegrityError
try:
    verify(manifest_path=mf, source=path)
except IntegrityError as e:
    # quarantine + re-download affected shards listed in str(e)
    logging.error("model corrupted: %s", e)

When it happens

Trigger: Running verify() where a manifest lists file X with sha256/size but the actual file on disk has different content (corrupted download, partial write, disk corruption, or a since-updated HF file).

Common situations: Interrupted model downloads leaving truncated safetensors, silent disk/NFS corruption on large checkpoints, or a manifest generated from an older revision of the HF repo being checked against a newer one.

Related errors

• No model files found in {model_path}
• No files found in HF repo {repo_id}.
• Model directory {model_path} is missing required component d
• Model directory {model_path} does not contain a transformer/
• Kimi expert-pack index SHA-256 does not match manifest

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/0d2d6ec79e4604e0. Report an issue: GitHub.