binary-husky/gpt_academic · error · ValueError

Invalid URL: {url}

Error message

Invalid URL: {url}

What it means

ValueError raised by WebReader.extract (read) when the URL fails the reader's _validate_url check before any network activity. It is a fast-fail on malformed or disallowed URLs, distinct from the later download failures.

Source

Thrown at crazy_functions/doc_fns/read_fns/web_reader.py:151

        return text.strip()

    def extract_text(self, url: str) -> str:
        """提取网页文本内容

        Args:
            url: 网页URL

        Returns:
            str: 提取的文本内容

        Raises:
            ValueError: URL无效时抛出
            Exception: 提取失败时抛出
        """
        try:
            if not self._validate_url(url):
                raise ValueError(f"Invalid URL: {url}")

            self.logger.info(f"Processing URL: {url}")

            # 下载网页
            html_content = self._download_webpage(url)
            if not html_content:
                raise Exception("Failed to download webpage")

            # 配置trafilatura提取选项
            extract_config = {
                'include_comments': self.config.extract_comments,
                'include_tables': self.config.extract_tables,
                'include_links': self.config.extract_links,
                'no_fallback': False,  # 允许使用后备提取器
            }

            # 提取文本
            extracted_text = trafilatura.extract(

View on GitHub (pinned to d6bde0fa54)

Solutions

1. Normalize before calling: strip whitespace, prepend https:// when the scheme is missing.
2. Validate with urllib.parse and require scheme in ('http','https') and a non-empty netloc.
3. Reject/escape control characters and angle brackets from pasted input.
4. Add a URL field type (pydantic HttpUrl) at the API boundary so bad values never reach the reader.

Example fix

# before
reader.read('www.example.com/article')  # ValueError: Invalid URL

# after
from urllib.parse import urlparse
def normalize(u: str) -> str:
    u = u.strip()
    if not u.startswith(('http://', 'https://')):
        u = 'https://' + u
    p = urlparse(u)
    assert p.scheme in ('http', 'https') and p.netloc
    return u
reader.read(normalize(raw_url))

from urllib.parse import urlparse
def normalize_url(u: str) -> str:
    u = u.strip().rstrip('.,);')
    if not u.startswith(('http://', 'https://')):
        u = 'https://' + u
    p = urlparse(u)
    if p.scheme not in ('http', 'https') or not p.netloc:
        raise ValueError(f'bad url: {u!r}')
    return u

def is_valid_url(url: str) -> bool:
    from urllib.parse import urlparse
    p = urlparse(url.strip())
    return p.scheme in ('http', 'https') and bool(p.netloc)

try:
    web_reader.read(url)
except ValueError as e:
    if str(e).startswith('Invalid URL'):
        url = normalize_url(url)
        return web_reader.read(url)
    raise

When it happens

Trigger: Calling read() with a URL lacking the http/https scheme, having whitespace or control characters, a bad hostname, or a scheme the validator rejects. Typically: 'example.com/page' without https://, 'ftp://...', or pasted text containing the URL plus extra characters.

Common situations: User-submitted URLs straight from a chat box or form with no normalization; strings like 'www.x.com' missing scheme; URLs copied with trailing punctuation or embedded newlines; scheme-casing issues on strict validators.

Related errors

• 文件大小 ({file_size_mb:.1f}MB) 超过限制 {max_size_mb}MB
• 没有读取权限: {path}
• 文件大小 ({file_size_mb:.1f}MB) 超过限制 {max_size_mb}MB
• 不支持的文件格式: {path.suffix}. 支持的格式: {', '.join(sorted(self.SUPPO
• File not found: {path}

AI-assisted analysis of binary-husky/gpt_academic@d6bde0fa54 (2026-08-14). Data as JSON: /api/errors/e0cb5f6940b94ca6. Report an issue: GitHub.