{"record":{"id":"0ec1e3cbc364ed53","repo":"sgl-project/sglang","slug":"download-failed-for-model-name-or-path-after-ma","errorCode":null,"errorMessage":"Download failed for {model_name_or_path} after {max_retries} attempts due to download errors. Last error: {type(e).__name__}: {e}","messagePattern":"Download failed for (.+?) after (.+?) attempts due to download errors\\. Last error: (.+?): (.+?)","errorType":"panic","errorClass":"RuntimeError","httpStatus":null,"severity":"critical","filePath":"python/sglang/srt/model_loader/ci_weight_validation.py","lineNumber":822,"sourceCode":"                    attempt + 1,\n                    max_retries,\n                    model_name_or_path,\n                    type(e).__name__,\n                    e,\n                )\n                if attempt < max_retries - 1:\n                    # Backoff: 10s, 20s, 40s. Clean only the stale\n                    # .incomplete files (not active ones from other processes).\n                    backoff = 10 * (2**attempt)\n                    logger.info(\n                        \"[CI Download] Cleaning up .incomplete files and \"\n                        \"retrying in %ds...\",\n                        backoff,\n                    )\n                    _cleanup_incomplete_blobs(model_name_or_path, cache_dir)\n                    time.sleep(backoff)\n                    continue\n                raise RuntimeError(\n                    f\"Download failed for {model_name_or_path} after \"\n                    f\"{max_retries} attempts due to download errors. \"\n                    f\"Last error: {type(e).__name__}: {e}\"\n                ) from e\n\n            # Validate downloaded files to catch corruption early\n            is_valid = _validate_weights_after_download(\n                hf_folder, allow_patterns, model_name_or_path\n            )\n\n            if is_valid:\n                return hf_folder\n\n            # Validation failed, corrupted files were cleaned up\n            if attempt < max_retries - 1:\n                log_info_on_rank0(\n                    logger,\n                    f\"Retrying download for {model_name_or_path} \"","sourceCodeStart":804,"sourceCodeEnd":840,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/model_loader/ci_weight_validation.py#L804-L840","documentation":"Raised by ci_download_with_validation_and_retry when downloading model weights from Hugging Face Hub exhausts all retry attempts due to download errors (network/HTTP failures, not corruption). The message includes the model name, attempt count, and the last exception. It means transient download errors persisted across the whole backoff schedule.","triggerScenarios":"Calling download_weights_from_hf for a model whose blobs repeatedly fail to download (connection resets, 429/5xx from HF, proxy failures) until max_retries is exhausted; incomplete blobs are cleaned and retried with backoff, then this RuntimeError is raised.","commonSituations":"Flaky networks, rate-limited HF token or anonymous access, corporate proxies blocking CDN, or transient HF Hub outages during CI weight downloads.","solutions":["Retry the job/run later after network recovers; set HF_TOKEN to avoid rate limits","Increase max_retries / backoff, or use HF_HUB_ENABLE_HF_TRANSFER or a mirror endpoint (HF_ENDPOINT)","Pre-download weights with huggingface-cli into a shared cache dir and point the run at it","Check proxy/firewall settings and disk space in the HF cache location"],"exampleFix":null,"handlingStrategy":"retry","validationCode":"import os\nassert os.environ.get('HF_TOKEN') or allow_anonymous, 'set HF_TOKEN to avoid rate-limited downloads'\n# optional: pre-check connectivity\nimport requests\nrequests.head('https://huggingface.co', timeout=10).raise_for_status()","typeGuard":null,"tryCatchPattern":"try:\n    download_weights_from_hf(model, cache_dir)\nexcept RuntimeError as e:\n    if 'Download failed' in str(e):\n        # backoff at the outer layer with a mirror endpoint\n        os.environ['HF_ENDPOINT'] = 'https://hf-mirror.com'\n        time.sleep(300)\n        download_weights_from_hf(model, cache_dir)\n    else:\n        raise","preventionTips":["Pre-download weights into a shared HF cache and mount it read-only in CI","Set HF_TOKEN and generous retry/backoff settings","Use stable network or mirrors; monitor HF status page"],"tags":["network","download","huggingface","retry-exhausted","weights"],"backgroundTag":"model-download-failed","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}