{"record":{"id":"a8eb1203440b54c1","repo":"sgl-project/sglang","slug":"downloaded-model-files-are-still-corrupted-for-mo","errorCode":null,"errorMessage":"Downloaded model files are still corrupted for {model_name_or_path} after {max_retries} attempts. This may indicate a persistent issue with the model files on Hugging Face Hub or network problems.","messagePattern":"Downloaded model files are still corrupted for (.+?) after (.+?) attempts\\. This may indicate a persistent issue with the model files on Hugging Face Hub or network problems\\.","errorType":"panic","errorClass":"RuntimeError","httpStatus":null,"severity":"critical","filePath":"python/sglang/srt/model_loader/ci_weight_validation.py","lineNumber":844,"sourceCode":"                ) from e\n\n            # Validate downloaded files to catch corruption early\n            is_valid = _validate_weights_after_download(\n                hf_folder, allow_patterns, model_name_or_path\n            )\n\n            if is_valid:\n                return hf_folder\n\n            # Validation failed, corrupted files were cleaned up\n            if attempt < max_retries - 1:\n                log_info_on_rank0(\n                    logger,\n                    f\"Retrying download for {model_name_or_path} \"\n                    f\"(attempt {attempt + 2}/{max_retries})...\",\n                )\n            else:\n                raise RuntimeError(\n                    f\"Downloaded model files are still corrupted for \"\n                    f\"{model_name_or_path} after {max_retries} attempts. \"\n                    \"This may indicate a persistent issue with the model files \"\n                    \"on Hugging Face Hub or network problems.\"\n                )\n\n        # Should never reach here, but return hf_folder just in case\n        return hf_folder\n\n\ndef ci_validate_and_clean_hf_cache(model_path: str) -> None:\n    \"\"\"Drop corrupted safetensors from the HF cache before a non-SGLang load.\n\n    HFRunner calls transformers' from_pretrained() directly, which bypasses the\n    validation in this module; a corrupted cache surfaces as \"EOF while parsing\".\n    \"\"\"\n    from sglang.utils import is_in_ci\n","sourceCodeStart":826,"sourceCodeEnd":862,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/model_loader/ci_weight_validation.py#L826-L862","documentation":"Raised by ci_download_with_validation_and_retry when downloads technically succeed but the downloaded model files fail integrity validation on every retry attempt. This points to corruption at the source (bad files on Hugging Face Hub) or persistent network corruption rather than a one-off bad transfer, since retries with cleanup did not help.","triggerScenarios":"Calling download_weights_from_hf where each attempt downloads files whose checksums/structure fail validation; _cleanup_incomplete_blobs + redownload loops finish with files still corrupted, so the RuntimeError is raised.","commonSituations":"A repo on HF Hub has corrupted or truncated blobs; a MITM proxy/CDN cache serves corrupted bytes; antivirus modifying files in the cache; or an outdated validation schema flagging valid files.","solutions":["Verify the model repo on HF Hub (file sizes/sha256 vs repo listing) and report if the source is corrupted","Clear the local HF cache completely (rm -rf ~/.cache/huggingface/hub/<model>) and re-download on a clean network","Bypass proxies/VPN/antivirus that may corrupt transfers, or download on another machine and copy the cache","If validation logic is stale (schema changed), update sglang or pin a known-good model revision"],"exampleFix":null,"handlingStrategy":"retry","validationCode":"from huggingface_hub import HfApi\ninfo = HfApi().model_info(model_name_or_path, files_metadata=True)\nfor s in info.siblings:\n    if s.size and s.size % 2:  # example sanity check\n        print('suspicious file size:', s.rfilename, s.size)","typeGuard":null,"tryCatchPattern":"try:\n    download_weights_from_hf(model, cache_dir)\nexcept RuntimeError as e:\n    if 'still corrupted' in str(e):\n        shutil.rmtree(cache_dir, ignore_errors=True)\n        raise  # surface: source corruption needs manual intervention\n    raise","preventionTips":["Pin exact model revisions (commit SHA) known to be intact","Keep HF cache on reliable local disk; avoid cache over flaky network mounts","Compare file hashes against the Hub listing after download","Exclude model caches from antivirus scanning"],"tags":["download","corruption","huggingface","validation","weights"],"backgroundTag":"downloaded-file-corruption","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}