{"record":{"id":"601105c7e302d752","repo":"sgl-project/sglang","slug":"found-len-corrupted-files-corrupted-safetensors","errorCode":null,"errorMessage":"Found {len(corrupted_files)} corrupted safetensors file(s). Files have been removed: {[os.path.basename(f) for f in corrupted_files]}. Please retry - the files will be re-downloaded automatically.","messagePattern":"Found (.+?) corrupted safetensors file\\(s\\)\\. Files have been removed: (.+?)\\. Please retry - the files will be re-downloaded automatically\\.","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"critical","filePath":"python/sglang/multimodal_gen/runtime/loader/weight_utils.py","lineNumber":282,"sourceCode":"                    blob_path = os.path.realpath(file_path)\n                    os.remove(file_path)\n                    logger.info(\n                        \"Removed corrupted symlink: %s\", os.path.basename(file_path)\n                    )\n                    if os.path.exists(blob_path):\n                        os.remove(blob_path)\n                        logger.info(\n                            \"Removed corrupted blob: %s\", os.path.basename(blob_path)\n                        )\n                elif os.path.isfile(file_path):\n                    os.remove(file_path)\n                    logger.info(\n                        \"Removed corrupted file: %s\", os.path.basename(file_path)\n                    )\n            except Exception as e:\n                logger.warning(\"Failed to remove corrupted file %s: %s\", file_path, e)\n\n        raise RuntimeError(\n            f\"Found {len(corrupted_files)} corrupted safetensors file(s). \"\n            f\"Files have been removed: {[os.path.basename(f) for f in corrupted_files]}. \"\n            \"Please retry - the files will be re-downloaded automatically.\"\n        )\n\n    _raise_if_duplicate_safetensors_keys(duplicate_files_by_key)\n\n    yield from backend.iter_weights(\n        hf_weights_files,\n        device=device,\n        to_cpu=to_cpu,\n        key_filter=key_filter,\n        clone_tensors=clone_streamed_tensors,\n        show_progress=enable_tqdm,\n    )\n\n\ndef _load_pt_file(bin_file: str, device: str) -> dict:","sourceCodeStart":264,"sourceCodeEnd":300,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/multimodal_gen/runtime/loader/weight_utils.py#L264-L300","documentation":"One or more safetensors files failed integrity/parsing checks and are considered corrupted. The loader has already deleted them from disk and instructs the user to retry so they get re-downloaded, since a clean retry restores a consistent checkpoint.","triggerScenarios":"safetensors_weights_iterator (used by maybe_load_fsdp_model) detects corrupted files while reading headers/streams; after logging removals (or failures to remove) it raises this RuntimeError listing basenames of the removed files.","commonSituations":"Truncated download, bit rot on disk, transfer interrupted mid-write, or filesystem that silently corrupted large files; previous partial runs that died mid-download.","solutions":["Simply retry the load — the corrupted files were removed and will be re-downloaded","If retry keeps failing, clear the local HF cache for that model and re-download from scratch","Check disk health/space if corruption recurs"],"exampleFix":"# just rerun the same command\npython -m sglang.launch_server --model <model>  # files re-download automatically","handlingStrategy":"retry","validationCode":"import os\nfrom safetensors import safe_open\n\ndef all_shards_readable(model_path) -> bool:\n    for f in os.listdir(model_path):\n        if f.endswith(\".safetensors\"):\n            try:\n                with safe_open(os.path.join(model_path, f), framework=\"pt\"):\n                    pass\n            except Exception:\n                return False\n    return True","typeGuard":null,"tryCatchPattern":"for attempt in range(3):\n    try:\n        maybe_load_fsdp_model(...)\n        break\n    except RuntimeError as e:\n        if \"corrupted safetensors\" not in str(e) or attempt == 2:\n            raise\n        # files were removed; re-download then retry\n        subprocess.run([\"huggingface-cli\", \"download\", model_name], check=True)","preventionTips":["Treat this error as transient: corrupted files are auto-deleted, so retry re-downloads them","Use huggingface-cli with resume for large checkpoints to avoid truncated files","Clear the HF cache for the model if corruption recurs on the same shard"],"tags":["safetensors","corruption","download","retry"],"backgroundTag":"corrupted-model-files","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}