{"record":{"id":"80866395b25a4185","repo":"binary-husky/gpt_academic","slug":"failed-to-download-webpage","errorCode":null,"errorMessage":"Failed to download webpage","messagePattern":"Failed to download webpage","errorType":"exception","errorClass":"Exception","httpStatus":null,"severity":"error","filePath":"crazy_functions/doc_fns/read_fns/web_reader.py","lineNumber":158,"sourceCode":" url: 网页URL\n\n Returns:\n str: 提取的文本内容\n\n Raises:\n ValueError: URL无效时抛出\n Exception: 提取失败时抛出\n \"\"\"\n try:\n if not self._validate_url(url):\n raise ValueError(f\"Invalid URL: {url}\")\n\n self.logger.info(f\"Processing URL: {url}\")\n\n # 下载网页\n html_content = self._download_webpage(url)\n if not html_content:\n raise Exception(\"Failed to download webpage\")\n\n # 配置trafilatura提取选项\n extract_config = {\n 'include_comments': self.config.extract_comments,\n 'include_tables': self.config.extract_tables,\n 'include_links': self.config.extract_links,\n 'no_fallback': False, # 允许使用后备提取器\n }\n\n # 提取文本\n extracted_text = trafilatura.extract(\n html_content,\n **extract_config\n )\n\n if not extracted_text:\n raise Exception(\"No content could be extracted\")\n","sourceCodeStart":140,"sourceCodeEnd":176,"githubUrl":"https://github.com/binary-husky/gpt_academic/blob/d6bde0fa54373309bd05823a49bda8da019d2c77/crazy_functions/doc_fns/read_fns/web_reader.py#L140-L176","documentation":"Generic Exception raised in WebReader.extract when _download_webpage returns a falsy result (empty string or None). In the current code the retry loop raises on the last failure, so hitting this usually means the loop exhausted normally but returned empty body text — an empty 200 response.","triggerScenarios":"A server returns HTTP 200 with an empty body (about:blank-style endpoints, redirects to empty pages, or responses whose .text decodes to ''). Also reachable if config.max_retries is 0/None making the loop skip and fall through to `return None`.","commonSituations":"Dynamic JS pages where the initial HTML is a near-empty shell; servers redirecting to a consent/empty page; misconfigured max_retries=0; encoding detection producing empty text on binary bodies.","solutions":["Log/inspect the raw response (status, headers, first bytes) for the URL in question.","For JS-heavy pages switch to a headless-browser fetcher; trafilatura cannot render scripts.","Check that WebReaderConfig.max_retries >= 1 so the retry loop actually runs.","Treat empty-body 200s explicitly (raise or retry with different headers) in your own fetch layer."],"exampleFix":"# before\ntext = reader.read(url) # Exception: Failed to download webpage\n\n# after (guard before calling; use a render-capable fetcher for empty shells)\nimport requests\nr = requests.get(url, timeout=30)\nif not r.text.strip():\n r = fetch_with_browser(url) # playwright etc.\ntext = reader.read(url)","handlingStrategy":"fallback","validationCode":"import requests\nr = requests.get(url, timeout=30, headers={'User-Agent': 'Mozilla/5.0'})\nif not r.text.strip():\n html = render_with_browser(url) # playwright fallback path","typeGuard":"def has_body(url: str) -> bool:\n import requests\n try:\n return bool(requests.get(url, timeout=10).text.strip())\n except requests.RequestException:\n return False","tryCatchPattern":"try:\n text = web_reader.read(url)\nexcept Exception as e:\n if str(e) == 'Failed to download webpage':\n text = read_rendered(url) # headless-browser fallback\n else:\n raise","preventionTips":["Ensure max_retries >= 1 in WebReaderConfig.","Detect JS-shell pages early (tiny ) and switch fetchers.","Handle empty-200 responses explicitly in your fetch layer.","Log response status+length for every fetch."],"tags":["network","http","empty-response","scraping"],"backgroundTag":null,"analyzedSha":"d6bde0fa54373309bd05823a49bda8da019d2c77","analyzedAt":"2026-08-14T22:48:35.038Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}