sgl-project/sglang · critical · RuntimeError

Checkpoint at ' ' is incomplete — the following shard(s)…

Error message

Checkpoint at '{model_path}' is incomplete — the following shard(s) listed in the index are missing from disk: {missing}. Re-download the checkpoint (e.g. `huggingface-cli download {os.path.basename(model_path)}`).

What it means

The model index (model.safetensors.index.json) lists shards that are not present on disk — the checkpoint is incomplete. An automatic re-download was attempted and failed, so loading aborts with instructions to re-fetch.

Solutions

1. Re-download: huggingface-cli download <model-name> (name shown in the message)
2. Check HF_TOKEN / gated-repo access and network, then retry
3. Free disk space or clear the partial HF cache (HF_HOME) before retrying

Example fix

huggingface-cli download meta-llama/Llama-3-8B-Instruct

import glob, json, os

def checkpoint_complete(model_path) -> bool:
    idx = os.path.join(model_path, "model.safetensors.index.json")
    if not os.path.exists(idx):
        return True
    shards = json.load(open(idx))["weight_map"].values()
    return all(os.path.exists(os.path.join(model_path, s)) for s in shards)

for attempt in range(3):
    if checkpoint_complete(model_path):
        break
    subprocess.run(["huggingface-cli", "download", model_name], check=True)
else:
    raise RuntimeError("checkpoint still incomplete after 3 downloads")

When it happens

Trigger: _list_safetensors_files finds shards in the index missing from disk, and _try_redownload_missing_shards returns falsy (no network, no credentials, or download failed). Reached via load_customized, load_safetensors_state_dict, or weight iteration/validation helpers.

Common situations: Interrupted huggingface-cli download, partial cache, disk-full during download, gated repo without valid token, or manually deleted shard files.

Related errors

• Comfy layer is missing checkpoint tensors
• Duplicate tensor names detected across safetensors files…
• Found corrupted safetensors file(s). Files have been…
• MiniMax-H3 adaln_t_table must have shape [N, D] with N >=…
• no safetensors files found in

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/cfbeb3dfac7b97ba. Report an issue: GitHub.

Appendix: source

Thrown at python/sglang/multimodal_gen/runtime/loader/utils.py:321

    found = sorted(glob.glob(os.path.join(str(model_path), "*.safetensors")))

    index_path = os.path.join(
        str(model_path), "diffusion_pytorch_model.safetensors.index.json"
    )
    if os.path.exists(index_path):
        with open(index_path) as f:
            index = json.load(f)
        expected_shards = sorted(set(index.get("weight_map", {}).values()))
        found_basenames = {os.path.basename(p) for p in found}
        missing = [s for s in expected_shards if s not in found_basenames]
        if missing:
            repaired = _try_redownload_missing_shards(model_path, missing)
            if repaired:
                found = sorted(
                    glob.glob(os.path.join(str(model_path), "*.safetensors"))
                )
            else:
                raise RuntimeError(
                    f"Checkpoint at '{model_path}' is incomplete — the following "
                    f"shard(s) listed in the index are missing from disk: "
                    f"{missing}. Re-download the checkpoint (e.g. "
                    f"`huggingface-cli download {os.path.basename(model_path)}`)."
                )

    return found


def load_safetensors_state_dict(model_path: str) -> dict[str, torch.Tensor]:
    """Load one safetensors checkpoint, including an indexed sharded set."""
    index_path = os.path.join(
        str(model_path), "diffusion_pytorch_model.safetensors.index.json"
    )
    safetensors_files = _list_safetensors_files(model_path)
    if os.path.exists(index_path):
        with open(index_path) as f:
            index = json.load(f)

View on GitHub (pinned to 0132848349)