unclecode/crawl4ai · error · HTTPException

Invalid URL format. Must start with http://, https://, or fo

Error message

Invalid URL format. Must start with http://, https://, or for raw HTML (raw:, raw://)

What it means

A 400 from POST /md: the url field must start with http://, https://, or a raw-HTML prefix (raw: or raw://). The endpoint converts pages to Markdown and only accepts remote URLs or inline raw HTML — bare hostnames, file://, data:, etc. are rejected before any crawl.

Source

Thrown at deploy/docker/server.py:583

@mcp_tool("md")
async def get_markdown(
    request: Request,
    body: MarkdownRequest,
    _td: Dict = Depends(token_dep),
):
    """
    Convert a web page into Markdown format.

    Supports multiple extraction modes:
    - fit (default): Readability-based extraction for clean content
    - raw: Direct DOM to Markdown conversion
    - bm25: BM25 relevance ranking with optional query
    - llm: LLM-based summarization with optional query

    Use this tool when you need clean, readable text from web pages.
    """
    if not body.url.startswith(("http://", "https://")) and not body.url.startswith(("raw:", "raw://")):
        raise HTTPException(
            400, "Invalid URL format. Must start with http://, https://, or for raw HTML (raw:, raw://)")
    # base_url is intentionally not accepted from the request (key-exfil vector);
    # the LLM endpoint is server-derived from the provider name only.
    markdown = await handle_markdown_request(
        body.url, body.f, body.q, body.c, config, body.provider,
        body.temperature
    )
    return JSONResponse({
        "url": body.url,
        "filter": body.f,
        "query": body.q,
        "cache": body.c,
        "markdown": markdown,
        "success": True
    })


@app.post("/html")

View on GitHub (pinned to 7e80152142)

Solutions

Prefix the URL with https:// (e.g. 'https://example.com/page').
For inline HTML, pass the content as 'raw:<html>...' or 'raw://<html>...'.
Normalize/strip whitespace on the client before sending.
Use a dedicated URL parser client-side (urllib.parse.urlparse) to confirm scheme is http/https first.

Example fix

# before
body = {'url': 'example.com/article'}
# after
from urllib.parse import urlparse
u = 'example.com/article'
body = {'url': u if urlparse(u).scheme in ('http','https') else 'https://' + u}

Defensive patterns

Strategy: validation

Validate before calling

from urllib.parse import urlparse

def valid_md_url(url: str) -> bool:
    u = url.strip()
    return u.startswith(('http://', 'https://', 'raw:', 'raw://'))

def normalize(url: str) -> str:
    u = url.strip()
    if not u.startswith(('http://', 'https://', 'raw:', 'raw://')):
        u = 'https://' + u
    return u

Type guard

def is_md_url(url: str) -> bool:
    return isinstance(url, str) and url.strip().startswith(('http://', 'https://', 'raw:', 'raw://'))

Prevention

Strip whitespace and always normalize to https:// before sending.
Use 'raw:<html>...' for inline HTML instead of file paths.
Centralize URL normalization in one client helper so every endpoint gets the same treatment.

When it happens

Trigger: POST /md with body.url = 'example.com/page' (no scheme), 'file:///tmp/x.html', 'ftp://...', or any string lacking the four accepted prefixes. The check is a plain startswith, so even whitespace or a leading '/' breaks it.

Common situations: Users pasting URLs without the scheme; passing a local file path expecting the server to read it; forgetting the 'raw:' prefix when submitting inline HTML for conversion.

Related errors

AI-assisted analysis of unclecode/crawl4ai@7e80152142 (2026-08-14). Data as JSON: /api/errors/905314f08295dcf8. Report an issue: GitHub.