searxng/searxng · error · ValueError

Cannot parse url

Error message

Cannot parse url

What it means

ValueError('Cannot parse url') raised by normalize_url when, after urljoin with base_url, the resulting URL has an empty netloc (no host). This means the relative url argument could not be resolved to an absolute URL against the engine's base_url.

Source

Thrown at searx/utils.py:292

        * str: normalized URL
    """
    if url.startswith('//'):
        # add http or https to this kind of url //example.com/
        parsed_search_url = urlparse(base_url)
        url = '{0}:{1}'.format(parsed_search_url.scheme or 'http', url)
    elif url.startswith('/'):
        # fix relative url to the search engine
        url = urljoin(base_url, url)

    # fix relative urls that fall through the crack
    if '://' not in url:
        url = urljoin(base_url, url)

    parsed_url = urlparse(url)

    # add a / at this end of the url if there is no path
    if not parsed_url.netloc:
        raise ValueError('Cannot parse url')
    if not parsed_url.path:
        url += '/'

    return url


def extract_url(xpath_results: list[ElementType] | ElementType | str | Number | bool | None, base_url: str) -> str:
    """Extract and normalize URL from lxml Element

    Example:
        >>> def f(s, search_url):
        >>>    return searx.utils.extract_url(html.fromstring(s), search_url)
        >>> f('<span id="42">https://example.com</span>', 'http://example.com/')
        'https://example.com/'
        >>> f('https://example.com', 'http://example.com/')
        'https://example.com/'
        >>> f('//example.com', 'http://example.com/')
        'http://example.com/'

View on GitHub (pinned to 9fea41204f)

Solutions

1. Verify the engine's base_url in settings.yml is a full absolute URL with scheme and host
2. Filter out non-http hrefs before extraction (skip '#', 'javascript:', empty)
3. Update the engine's XPath selector if the href attribute changed

Example fix

# before
url = extract_url(dom.xpath('//a/@href'), base_url)
# after
hrefs = [h for h in dom.xpath('//a/@href') if h and not h.startswith(('#', 'javascript:'))]
url = extract_url(hrefs, base_url) if hrefs else None

from urllib.parse import urljoin, urlparse
joined = urljoin(base_url, url)
if not urlparse(joined).netloc:
    return None  # skip unusable link

try:
    url = extract_url(nodes, base_url)
except ValueError as e:
    logger.debug('skipping bad url: %s', e)
    url = None

When it happens

Trigger: Calling extract_url/normalize_url with a relative href like '#section', 'javascript:...', or '' against a base_url, or a base_url that is itself malformed so urljoin produces no host.

Common situations: Engine site markup changes producing anchor-only or empty links, base_url misconfigured in settings.yml for an engine, or protocol-relative URLs joined against a bad base.

Related errors

• extract_text(None, allow_none=False)
• Empty url resultset
• URL not found
• len(xpath_str) < {min_len}
• don't know how to extract messages from %s

AI-assisted analysis of searxng/searxng@9fea41204f (2026-08-27). Data as JSON: /api/errors/9d31406d8ec67932. Report an issue: GitHub.