binary-husky/gpt_academic · error · Exception

Failed to download webpage

Error message

Failed to download webpage

What it means

Generic Exception raised in WebReader.extract when _download_webpage returns a falsy result (empty string or None). In the current code the retry loop raises on the last failure, so hitting this usually means the loop exhausted normally but returned empty body text — an empty 200 response.

Source

Thrown at crazy_functions/doc_fns/read_fns/web_reader.py:158

            url: 网页URL

        Returns:
            str: 提取的文本内容

        Raises:
            ValueError: URL无效时抛出
            Exception: 提取失败时抛出
        """
        try:
            if not self._validate_url(url):
                raise ValueError(f"Invalid URL: {url}")

            self.logger.info(f"Processing URL: {url}")

            # 下载网页
            html_content = self._download_webpage(url)
            if not html_content:
                raise Exception("Failed to download webpage")

            # 配置trafilatura提取选项
            extract_config = {
                'include_comments': self.config.extract_comments,
                'include_tables': self.config.extract_tables,
                'include_links': self.config.extract_links,
                'no_fallback': False,  # 允许使用后备提取器
            }

            # 提取文本
            extracted_text = trafilatura.extract(
                html_content,
                **extract_config
            )

            if not extracted_text:
                raise Exception("No content could be extracted")

View on GitHub (pinned to d6bde0fa54)

Solutions

1. Log/inspect the raw response (status, headers, first bytes) for the URL in question.
2. For JS-heavy pages switch to a headless-browser fetcher; trafilatura cannot render scripts.
3. Check that WebReaderConfig.max_retries >= 1 so the retry loop actually runs.
4. Treat empty-body 200s explicitly (raise or retry with different headers) in your own fetch layer.

Example fix

# before
text = reader.read(url)  # Exception: Failed to download webpage

# after (guard before calling; use a render-capable fetcher for empty shells)
import requests
r = requests.get(url, timeout=30)
if not r.text.strip():
    r = fetch_with_browser(url)  # playwright etc.
text = reader.read(url)

import requests
r = requests.get(url, timeout=30, headers={'User-Agent': 'Mozilla/5.0'})
if not r.text.strip():
    html = render_with_browser(url)  # playwright fallback path

def has_body(url: str) -> bool:
    import requests
    try:
        return bool(requests.get(url, timeout=10).text.strip())
    except requests.RequestException:
        return False

try:
    text = web_reader.read(url)
except Exception as e:
    if str(e) == 'Failed to download webpage':
        text = read_rendered(url)  # headless-browser fallback
    else:
        raise

When it happens

Trigger: A server returns HTTP 200 with an empty body (about:blank-style endpoints, redirects to empty pages, or responses whose .text decodes to ''). Also reachable if config.max_retries is 0/None making the loop skip and fall through to `return None`.

Common situations: Dynamic JS pages where the initial HTML is a near-empty shell; servers redirecting to a consent/empty page; misconfigured max_retries=0; encoding detection producing empty text on binary bodies.

Related errors

• Failed to download webpage after {self.config.max_retries} a
• response.content.decode()
• 在线搜索失败！\n{Exceptions}
• 在线搜索失败，状态码: {response.status_code}\t{response.content.decode
• 无法下载资源{txt}，请检查。

AI-assisted analysis of binary-husky/gpt_academic@d6bde0fa54 (2026-08-14). Data as JSON: /api/errors/80866395b25a4185. Report an issue: GitHub.