{"record":{"id":"ad96aea6b67e9685","repo":"binary-husky/gpt_academic","slug":"failed-to-download-webpage-after-self-config-max","errorCode":null,"errorMessage":"Failed to download webpage after {self.config.max_retries} attempts: {e}","messagePattern":"Failed to download webpage after (.+?) attempts: (.+?)","errorType":"exception","errorClass":"Exception","httpStatus":null,"severity":"error","filePath":"crazy_functions/doc_fns/read_fns/web_reader.py","lineNumber":110,"sourceCode":"\n        Raises:\n            Exception: 下载失败时抛出异常\n        \"\"\"\n        headers = {'User-Agent': self.config.user_agent}\n\n        for attempt in range(self.config.max_retries):\n            try:\n                response = requests.get(\n                    url,\n                    headers=headers,\n                    timeout=self.config.timeout\n                )\n                response.raise_for_status()\n                return response.text\n            except requests.RequestException as e:\n                self.logger.warning(f\"Attempt {attempt + 1} failed: {e}\")\n                if attempt == self.config.max_retries - 1:\n                    raise Exception(f\"Failed to download webpage after {self.config.max_retries} attempts: {e}\")\n        return None\n\n    def _cleanup_text(self, text: str) -> str:\n        \"\"\"清理文本\n\n        Args:\n            text: 原始文本\n\n        Returns:\n            str: 清理后的文本\n        \"\"\"\n        if not text:\n            return \"\"\n\n        if self.config.text_cleanup['remove_extra_spaces']:\n            text = ' '.join(text.split())\n\n        if self.config.text_cleanup['normalize_whitespace']:","sourceCodeStart":92,"sourceCodeEnd":128,"githubUrl":"https://github.com/binary-husky/gpt_academic/blob/d6bde0fa54373309bd05823a49bda8da019d2c77/crazy_functions/doc_fns/read_fns/web_reader.py#L92-L128","documentation":"Generic Exception raised at the end of WebReader._download_webpage's retry loop: every attempt (config.max_retries) hit a requests.RequestException (connect error, DNS failure, timeout, 4xx/5xx via raise_for_status). The last exception's text is embedded, and each attempt was already logged as a warning.","triggerScenarios":"Calling the extract/read API with a URL that is unreachable for all retries: dead domain, TLS failure, 403/404/500 responses, firewall/proxy blocking, or a timeout shorter than the server's response time. Because requests.RequestException is caught broadly, any HTTP-level failure counts.","commonSituations":"Scraping targets behind Cloudflare or bots blockers returning 403; flaky corporate proxies; rate-limited endpoints (429) with no backoff; misconfigured timeout in WebReaderConfig; DNS fails in containers with broken resolv.conf.","solutions":["Inspect the embedded exception and the per-attempt warnings to classify: DNS vs TLS vs status code vs timeout.","If 403/429: add proper headers/User-Agent, slow down, or use an API instead of scraping.","Increase config.timeout and/or max_retries; add exponential backoff between attempts.","Verify network egress (proxy env vars, DNS, firewall) from the host running the reader."],"exampleFix":"# before\nhtml = reader._download_webpage(url)  # Exception after N attempts\n\n# after\ncfg = WebReaderConfig(timeout=30, max_retries=4)\nreader = WebReader(cfg)\nsession_headers = {'User-Agent': 'Mozilla/5.0'}\n# prefer the public API which uses config; wrap and classify:\ntry:\n    text = reader.read(url)\nexcept Exception as e:\n    if '403' in str(e):\n        raise RuntimeError('blocked by target; use API or headers') from e\n    raise","handlingStrategy":"retry","validationCode":"from urllib.parse import urlparse\nu = urlparse(url)\nassert u.scheme in ('http', 'https') and u.netloc, 'bad url'\n# optional: quick HEAD reachability probe with short timeout\nimport requests\nrequests.head(url, timeout=5).raise_for_status()","typeGuard":"def url_reachable(url: str, timeout: float = 5.0) -> bool:\n    import requests\n    from urllib.parse import urlparse\n    if urlparse(url).scheme not in ('http', 'https'):\n        return False\n    try:\n        requests.head(url, timeout=timeout, allow_redirects=True)\n        return True\n    except requests.RequestException:\n        return False","tryCatchPattern":"try:\n    text = web_reader.read(url)\nexcept Exception as e:\n    if 'Failed to download webpage after' in str(e):\n        text = fetch_via_proxy_or_later(url)  # backoff + alternate route\n    else:\n        raise","preventionTips":["Set realistic timeout and retries in WebReaderConfig.","Add backoff between attempts (the built-in loop has none).","Send a real User-Agent to avoid 403s.","Verify proxy/DNS egress from the runtime host.","Add 429-aware throttling when scraping at scale."],"tags":["network","http","retry","scraping"],"backgroundTag":null,"analyzedSha":"d6bde0fa54373309bd05823a49bda8da019d2c77","analyzedAt":"2026-08-14T22:48:35.038Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}