sgl-project/sglang · critical · RuntimeError

Download failed for {model_name_or_path} after {max_retries}

Error message

Download failed for {model_name_or_path} after {max_retries} attempts due to download errors. Last error: {type(e).__name__}: {e}

What it means

Raised by ci_download_with_validation_and_retry when downloading model weights from Hugging Face Hub exhausts all retry attempts due to download errors (network/HTTP failures, not corruption). The message includes the model name, attempt count, and the last exception. It means transient download errors persisted across the whole backoff schedule.

Source

Thrown at python/sglang/srt/model_loader/ci_weight_validation.py:822

                    attempt + 1,
                    max_retries,
                    model_name_or_path,
                    type(e).__name__,
                    e,
                )
                if attempt < max_retries - 1:
                    # Backoff: 10s, 20s, 40s. Clean only the stale
                    # .incomplete files (not active ones from other processes).
                    backoff = 10 * (2**attempt)
                    logger.info(
                        "[CI Download] Cleaning up .incomplete files and "
                        "retrying in %ds...",
                        backoff,
                    )
                    _cleanup_incomplete_blobs(model_name_or_path, cache_dir)
                    time.sleep(backoff)
                    continue
                raise RuntimeError(
                    f"Download failed for {model_name_or_path} after "
                    f"{max_retries} attempts due to download errors. "
                    f"Last error: {type(e).__name__}: {e}"
                ) from e

            # Validate downloaded files to catch corruption early
            is_valid = _validate_weights_after_download(
                hf_folder, allow_patterns, model_name_or_path
            )

            if is_valid:
                return hf_folder

            # Validation failed, corrupted files were cleaned up
            if attempt < max_retries - 1:
                log_info_on_rank0(
                    logger,
                    f"Retrying download for {model_name_or_path} "

View on GitHub (pinned to 0132848349)

Solutions

Retry the job/run later after network recovers; set HF_TOKEN to avoid rate limits
Increase max_retries / backoff, or use HF_HUB_ENABLE_HF_TRANSFER or a mirror endpoint (HF_ENDPOINT)
Pre-download weights with huggingface-cli into a shared cache dir and point the run at it
Check proxy/firewall settings and disk space in the HF cache location

Defensive patterns

Strategy: retry

Validate before calling

import os
assert os.environ.get('HF_TOKEN') or allow_anonymous, 'set HF_TOKEN to avoid rate-limited downloads'
# optional: pre-check connectivity
import requests
requests.head('https://huggingface.co', timeout=10).raise_for_status()

Try / catch

try:
    download_weights_from_hf(model, cache_dir)
except RuntimeError as e:
    if 'Download failed' in str(e):
        # backoff at the outer layer with a mirror endpoint
        os.environ['HF_ENDPOINT'] = 'https://hf-mirror.com'
        time.sleep(300)
        download_weights_from_hf(model, cache_dir)
    else:
        raise

Prevention

Pre-download weights into a shared HF cache and mount it read-only in CI
Set HF_TOKEN and generous retry/backoff settings
Use stable network or mirrors; monitor HF status page

When it happens

Trigger: Calling download_weights_from_hf for a model whose blobs repeatedly fail to download (connection resets, 429/5xx from HF, proxy failures) until max_retries is exhausted; incomplete blobs are cleaned and retried with backoff, then this RuntimeError is raised.

Common situations: Flaky networks, rate-limited HF token or anonymous access, corporate proxies blocking CDN, or transient HF Hub outages during CI weight downloads.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/0ec1e3cbc364ed53. Report an issue: GitHub.