{"record":{"id":"cfbeb3dfac7b97ba","repo":"sgl-project/sglang","slug":"checkpoint-at-model-path-is-incomplete-the-f","errorCode":null,"errorMessage":"Checkpoint at '{model_path}' is incomplete — the following shard(s) listed in the index are missing from disk: {missing}. Re-download the checkpoint (e.g. `huggingface-cli download {os.path.basename(model_path)}`).","messagePattern":"Checkpoint at '(.+?)' is incomplete — the following shard\\(s\\) listed in the index are missing from disk: (.+?)\\. Re-download the checkpoint \\(e\\.g\\. `huggingface-cli download (.+?)`\\)\\.","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"critical","filePath":"python/sglang/multimodal_gen/runtime/loader/utils.py","lineNumber":321,"sourceCode":"    found = sorted(glob.glob(os.path.join(str(model_path), \"*.safetensors\")))\n\n    index_path = os.path.join(\n        str(model_path), \"diffusion_pytorch_model.safetensors.index.json\"\n    )\n    if os.path.exists(index_path):\n        with open(index_path) as f:\n            index = json.load(f)\n        expected_shards = sorted(set(index.get(\"weight_map\", {}).values()))\n        found_basenames = {os.path.basename(p) for p in found}\n        missing = [s for s in expected_shards if s not in found_basenames]\n        if missing:\n            repaired = _try_redownload_missing_shards(model_path, missing)\n            if repaired:\n                found = sorted(\n                    glob.glob(os.path.join(str(model_path), \"*.safetensors\"))\n                )\n            else:\n                raise RuntimeError(\n                    f\"Checkpoint at '{model_path}' is incomplete — the following \"\n                    f\"shard(s) listed in the index are missing from disk: \"\n                    f\"{missing}. Re-download the checkpoint (e.g. \"\n                    f\"`huggingface-cli download {os.path.basename(model_path)}`).\"\n                )\n\n    return found\n\n\ndef load_safetensors_state_dict(model_path: str) -> dict[str, torch.Tensor]:\n    \"\"\"Load one safetensors checkpoint, including an indexed sharded set.\"\"\"\n    index_path = os.path.join(\n        str(model_path), \"diffusion_pytorch_model.safetensors.index.json\"\n    )\n    safetensors_files = _list_safetensors_files(model_path)\n    if os.path.exists(index_path):\n        with open(index_path) as f:\n            index = json.load(f)","sourceCodeStart":303,"sourceCodeEnd":339,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/multimodal_gen/runtime/loader/utils.py#L303-L339","documentation":"The model index (model.safetensors.index.json) lists shards that are not present on disk — the checkpoint is incomplete. An automatic re-download was attempted and failed, so loading aborts with instructions to re-fetch.","triggerScenarios":"_list_safetensors_files finds shards in the index missing from disk, and _try_redownload_missing_shards returns falsy (no network, no credentials, or download failed). Reached via load_customized, load_safetensors_state_dict, or weight iteration/validation helpers.","commonSituations":"Interrupted huggingface-cli download, partial cache, disk-full during download, gated repo without valid token, or manually deleted shard files.","solutions":["Re-download: huggingface-cli download <model-name> (name shown in the message)","Check HF_TOKEN / gated-repo access and network, then retry","Free disk space or clear the partial HF cache (HF_HOME) before retrying"],"exampleFix":"huggingface-cli download meta-llama/Llama-3-8B-Instruct","handlingStrategy":"retry","validationCode":"import glob, json, os\n\ndef checkpoint_complete(model_path) -> bool:\n    idx = os.path.join(model_path, \"model.safetensors.index.json\")\n    if not os.path.exists(idx):\n        return True\n    shards = json.load(open(idx))[\"weight_map\"].values()\n    return all(os.path.exists(os.path.join(model_path, s)) for s in shards)","typeGuard":null,"tryCatchPattern":"for attempt in range(3):\n    if checkpoint_complete(model_path):\n        break\n    subprocess.run([\"huggingface-cli\", \"download\", model_name], check=True)\nelse:\n    raise RuntimeError(\"checkpoint still incomplete after 3 downloads\")","preventionTips":["Verify shard completeness before launching long jobs","Download models with huggingface-cli and let it resume rather than manual partial copies","Monitor disk space during downloads"],"tags":["safetensors","checkpoint","download","corrupt-checkpoint"],"backgroundTag":"incomplete-checkpoint-download","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}