sgl-project/sglang · critical · RuntimeError

Found {len(corrupted_files)} corrupted safetensors file(s).

Error message

Found {len(corrupted_files)} corrupted safetensors file(s). Files have been removed: {[os.path.basename(f) for f in corrupted_files]}. Please retry - the files will be re-downloaded automatically.

What it means

One or more safetensors files failed integrity/parsing checks and are considered corrupted. The loader has already deleted them from disk and instructs the user to retry so they get re-downloaded, since a clean retry restores a consistent checkpoint.

Source

Thrown at python/sglang/multimodal_gen/runtime/loader/weight_utils.py:282

                    blob_path = os.path.realpath(file_path)
                    os.remove(file_path)
                    logger.info(
                        "Removed corrupted symlink: %s", os.path.basename(file_path)
                    )
                    if os.path.exists(blob_path):
                        os.remove(blob_path)
                        logger.info(
                            "Removed corrupted blob: %s", os.path.basename(blob_path)
                        )
                elif os.path.isfile(file_path):
                    os.remove(file_path)
                    logger.info(
                        "Removed corrupted file: %s", os.path.basename(file_path)
                    )
            except Exception as e:
                logger.warning("Failed to remove corrupted file %s: %s", file_path, e)

        raise RuntimeError(
            f"Found {len(corrupted_files)} corrupted safetensors file(s). "
            f"Files have been removed: {[os.path.basename(f) for f in corrupted_files]}. "
            "Please retry - the files will be re-downloaded automatically."
        )

    _raise_if_duplicate_safetensors_keys(duplicate_files_by_key)

    yield from backend.iter_weights(
        hf_weights_files,
        device=device,
        to_cpu=to_cpu,
        key_filter=key_filter,
        clone_tensors=clone_streamed_tensors,
        show_progress=enable_tqdm,
    )


def _load_pt_file(bin_file: str, device: str) -> dict:

View on GitHub (pinned to 0132848349)

Solutions

1. Simply retry the load — the corrupted files were removed and will be re-downloaded
2. If retry keeps failing, clear the local HF cache for that model and re-download from scratch
3. Check disk health/space if corruption recurs

Example fix

# just rerun the same command
python -m sglang.launch_server --model <model>  # files re-download automatically

import os
from safetensors import safe_open

def all_shards_readable(model_path) -> bool:
    for f in os.listdir(model_path):
        if f.endswith(".safetensors"):
            try:
                with safe_open(os.path.join(model_path, f), framework="pt"):
                    pass
            except Exception:
                return False
    return True

for attempt in range(3):
    try:
        maybe_load_fsdp_model(...)
        break
    except RuntimeError as e:
        if "corrupted safetensors" not in str(e) or attempt == 2:
            raise
        # files were removed; re-download then retry
        subprocess.run(["huggingface-cli", "download", model_name], check=True)

When it happens

Trigger: safetensors_weights_iterator (used by maybe_load_fsdp_model) detects corrupted files while reading headers/streams; after logging removals (or failures to remove) it raises this RuntimeError listing basenames of the removed files.

Common situations: Truncated download, bit rot on disk, transfer interrupted mid-write, or filesystem that silently corrupted large files; previous partial runs that died mid-download.

Related errors

• Checkpoint at '{model_path}' is incomplete — the following s
• Downloaded model files are still corrupted for {model_name_o
• Failed to generate image: {str(e)}
• Network error after {consecutive_errors} consecutive failure
• Quanto checkpoint is missing quantization_map_base64

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/601105c7e302d752. Report an issue: GitHub.