{"record":{"id":"e0cb5f6940b94ca6","repo":"binary-husky/gpt_academic","slug":"invalid-url-url","errorCode":null,"errorMessage":"Invalid URL: {url}","messagePattern":"Invalid URL: (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"crazy_functions/doc_fns/read_fns/web_reader.py","lineNumber":151,"sourceCode":"\n        return text.strip()\n\n    def extract_text(self, url: str) -> str:\n        \"\"\"提取网页文本内容\n\n        Args:\n            url: 网页URL\n\n        Returns:\n            str: 提取的文本内容\n\n        Raises:\n            ValueError: URL无效时抛出\n            Exception: 提取失败时抛出\n        \"\"\"\n        try:\n            if not self._validate_url(url):\n                raise ValueError(f\"Invalid URL: {url}\")\n\n            self.logger.info(f\"Processing URL: {url}\")\n\n            # 下载网页\n            html_content = self._download_webpage(url)\n            if not html_content:\n                raise Exception(\"Failed to download webpage\")\n\n            # 配置trafilatura提取选项\n            extract_config = {\n                'include_comments': self.config.extract_comments,\n                'include_tables': self.config.extract_tables,\n                'include_links': self.config.extract_links,\n                'no_fallback': False,  # 允许使用后备提取器\n            }\n\n            # 提取文本\n            extracted_text = trafilatura.extract(","sourceCodeStart":133,"sourceCodeEnd":169,"githubUrl":"https://github.com/binary-husky/gpt_academic/blob/d6bde0fa54373309bd05823a49bda8da019d2c77/crazy_functions/doc_fns/read_fns/web_reader.py#L133-L169","documentation":"ValueError raised by WebReader.extract (read) when the URL fails the reader's _validate_url check before any network activity. It is a fast-fail on malformed or disallowed URLs, distinct from the later download failures.","triggerScenarios":"Calling read() with a URL lacking the http/https scheme, having whitespace or control characters, a bad hostname, or a scheme the validator rejects. Typically: 'example.com/page' without https://, 'ftp://...', or pasted text containing the URL plus extra characters.","commonSituations":"User-submitted URLs straight from a chat box or form with no normalization; strings like 'www.x.com' missing scheme; URLs copied with trailing punctuation or embedded newlines; scheme-casing issues on strict validators.","solutions":["Normalize before calling: strip whitespace, prepend https:// when the scheme is missing.","Validate with urllib.parse and require scheme in ('http','https') and a non-empty netloc.","Reject/escape control characters and angle brackets from pasted input.","Add a URL field type (pydantic HttpUrl) at the API boundary so bad values never reach the reader."],"exampleFix":"# before\nreader.read('www.example.com/article')  # ValueError: Invalid URL\n\n# after\nfrom urllib.parse import urlparse\ndef normalize(u: str) -> str:\n    u = u.strip()\n    if not u.startswith(('http://', 'https://')):\n        u = 'https://' + u\n    p = urlparse(u)\n    assert p.scheme in ('http', 'https') and p.netloc\n    return u\nreader.read(normalize(raw_url))","handlingStrategy":"validation","validationCode":"from urllib.parse import urlparse\ndef normalize_url(u: str) -> str:\n    u = u.strip().rstrip('.,);')\n    if not u.startswith(('http://', 'https://')):\n        u = 'https://' + u\n    p = urlparse(u)\n    if p.scheme not in ('http', 'https') or not p.netloc:\n        raise ValueError(f'bad url: {u!r}')\n    return u","typeGuard":"def is_valid_url(url: str) -> bool:\n    from urllib.parse import urlparse\n    p = urlparse(url.strip())\n    return p.scheme in ('http', 'https') and bool(p.netloc)","tryCatchPattern":"try:\n    web_reader.read(url)\nexcept ValueError as e:\n    if str(e).startswith('Invalid URL'):\n        url = normalize_url(url)\n        return web_reader.read(url)\n    raise","preventionTips":["Normalize user input (strip, add scheme) before calling read().","Use pydantic HttpUrl at API boundaries.","Reject non-http schemes outright.","Trim surrounding punctuation from pasted URLs."],"tags":["validation","url","parsing","python"],"backgroundTag":null,"analyzedSha":"d6bde0fa54373309bd05823a49bda8da019d2c77","analyzedAt":"2026-08-14T22:48:35.038Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}