binary-husky/gpt_academic · error · Exception

Failed to download webpage after {self.config.max_retries} a

Error message

Failed to download webpage after {self.config.max_retries} attempts: {e}

What it means

Generic Exception raised at the end of WebReader._download_webpage's retry loop: every attempt (config.max_retries) hit a requests.RequestException (connect error, DNS failure, timeout, 4xx/5xx via raise_for_status). The last exception's text is embedded, and each attempt was already logged as a warning.

Source

Thrown at crazy_functions/doc_fns/read_fns/web_reader.py:110

        Raises:
            Exception: 下载失败时抛出异常
        """
        headers = {'User-Agent': self.config.user_agent}

        for attempt in range(self.config.max_retries):
            try:
                response = requests.get(
                    url,
                    headers=headers,
                    timeout=self.config.timeout
                )
                response.raise_for_status()
                return response.text
            except requests.RequestException as e:
                self.logger.warning(f"Attempt {attempt + 1} failed: {e}")
                if attempt == self.config.max_retries - 1:
                    raise Exception(f"Failed to download webpage after {self.config.max_retries} attempts: {e}")
        return None

    def _cleanup_text(self, text: str) -> str:
        """清理文本

        Args:
            text: 原始文本

        Returns:
            str: 清理后的文本
        """
        if not text:
            return ""

        if self.config.text_cleanup['remove_extra_spaces']:
            text = ' '.join(text.split())

        if self.config.text_cleanup['normalize_whitespace']:

View on GitHub (pinned to d6bde0fa54)

Solutions

1. Inspect the embedded exception and the per-attempt warnings to classify: DNS vs TLS vs status code vs timeout.
2. If 403/429: add proper headers/User-Agent, slow down, or use an API instead of scraping.
3. Increase config.timeout and/or max_retries; add exponential backoff between attempts.
4. Verify network egress (proxy env vars, DNS, firewall) from the host running the reader.

Example fix

# before
html = reader._download_webpage(url)  # Exception after N attempts

# after
cfg = WebReaderConfig(timeout=30, max_retries=4)
reader = WebReader(cfg)
session_headers = {'User-Agent': 'Mozilla/5.0'}
# prefer the public API which uses config; wrap and classify:
try:
    text = reader.read(url)
except Exception as e:
    if '403' in str(e):
        raise RuntimeError('blocked by target; use API or headers') from e
    raise

from urllib.parse import urlparse
u = urlparse(url)
assert u.scheme in ('http', 'https') and u.netloc, 'bad url'
# optional: quick HEAD reachability probe with short timeout
import requests
requests.head(url, timeout=5).raise_for_status()

def url_reachable(url: str, timeout: float = 5.0) -> bool:
    import requests
    from urllib.parse import urlparse
    if urlparse(url).scheme not in ('http', 'https'):
        return False
    try:
        requests.head(url, timeout=timeout, allow_redirects=True)
        return True
    except requests.RequestException:
        return False

try:
    text = web_reader.read(url)
except Exception as e:
    if 'Failed to download webpage after' in str(e):
        text = fetch_via_proxy_or_later(url)  # backoff + alternate route
    else:
        raise

When it happens

Trigger: Calling the extract/read API with a URL that is unreachable for all retries: dead domain, TLS failure, 403/404/500 responses, firewall/proxy blocking, or a timeout shorter than the server's response time. Because requests.RequestException is caught broadly, any HTTP-level failure counts.

Common situations: Scraping targets behind Cloudflare or bots blockers returning 403; flaky corporate proxies; rate-limited endpoints (429) with no backoff; misconfigured timeout in WebReaderConfig; DNS fails in containers with broken resolv.conf.

Related errors

• Failed to download webpage
• 无法下载论文 {self.doi}，所有重试都失败了
• response.content.decode()
• 在线搜索失败！\n{Exceptions}
• 在线搜索失败，状态码: {response.status_code}\t{response.content.decode

AI-assisted analysis of binary-husky/gpt_academic@d6bde0fa54 (2026-08-14). Data as JSON: /api/errors/ad96aea6b67e9685. Report an issue: GitHub.