modelcontextprotocol/servers · warning · McpError

-32603

-32603

Error message

Failed to fetch robots.txt {robot_txt_url} due to a connection issue

What it means

In check_may_autonomously_fetch_url(), before fetching the target the server GETs {origin}/robots.txt. If that request raises an httpx.HTTPError (DNS failure, connection refused, TLS/cert error, proxy failure, or timeout), it is surfaced as McpError code INTERNAL_ERROR (-32603). The error names the robot_txt_url that could not be reached.

Source

Thrown at src/fetch/src/mcp_server_fetch/server.py:83

async def check_may_autonomously_fetch_url(url: str, user_agent: str, proxy_url: str | None = None) -> None:
    """
    Check if the URL can be fetched by the user agent according to the robots.txt file.
    Raises a McpError if not.
    """
    from httpx import AsyncClient, HTTPError

    robot_txt_url = get_robots_txt_url(url)

    async with AsyncClient(proxy=proxy_url) as client:
        try:
            response = await client.get(
                robot_txt_url,
                follow_redirects=True,
                headers={"User-Agent": user_agent},
            )
        except HTTPError:
            raise McpError(ErrorData(
                code=INTERNAL_ERROR,
                message=f"Failed to fetch robots.txt {robot_txt_url} due to a connection issue",
            ))
        if response.status_code in (401, 403):
            raise McpError(ErrorData(
                code=INTERNAL_ERROR,
                message=f"When fetching robots.txt ({robot_txt_url}), received status {response.status_code} so assuming that autonomous fetching is not allowed, the user can try manually fetching by using the fetch prompt",
            ))
        elif 400 <= response.status_code < 500:
            return
        robot_txt = response.text
    processed_robot_txt = "\n".join(
        line for line in robot_txt.splitlines() if not line.strip().startswith("#")
    )
    robot_parser = Protego.parse(processed_robot_txt)
    if not robot_parser.can_fetch(str(url), user_agent):
        raise McpError(ErrorData(
            code=INTERNAL_ERROR,

View on GitHub (pinned to 76d64c822f)

Solutions

1. Confirm the origin is reachable from the server environment (curl the robots.txt URL).
2. Provide a working proxy_url so httpx can reach the origin.
3. Retry once: transient DNS/TCP failures often clear immediately.
4. If autonomous fetch is not required, use the fetch prompt (manual User-Agent) or set ignore_robots_txt to bypass the robots check.

Example fix

# before
await check_may_autonomously_fetch_url(url, ua, None)  # origin unreachable -> McpError

# after: route through a proxy / fall back to manual
await check_may_autonomously_fetch_url(url, ua, proxy_url='http://proxy:8080')

import httpx
async def robots_reachable(url: str, proxy_url: str | None = None) -> bool:
    from mcp_server_fetch.server import get_robots_txt_url
    try:
        async with httpx.AsyncClient(proxy=proxy_url) as c:
            r = await c.get(get_robots_txt_url(url), headers={'User-Agent':'preflight'}, timeout=10)
            return r.status_code < 500
    except httpx.HTTPError:
        return False

from mcp.shared.exceptions import McpError
for attempt in range(3):
    try:
        await check_may_autonomously_fetch_url(url, ua, proxy_url)
        break
    except McpError as e:
        if 'connection issue' not in str(e) or attempt == 2:
            raise
        await asyncio.sleep(2 ** attempt)

When it happens

Trigger: Origin host offline; DNS cannot resolve; firewall/proxy blocks egress; invalid TLS certificate; the robots.txt endpoint hangs until timeout; proxy_url is misconfigured or unreachable.

Common situations: Server runs in a sandbox/CI without internet egress; corporate proxy required but not configured; intranet host with no route from the server; transient network outage.

Related errors

• No response body
• -32602
• Unsupported URL protocol for ${dataUri}. Only http, https, a
• Error processing file ${dataUri}: ${error instanceof Error ?
• Content-Length for ${url} exceeds max of ${maxBytes}: ${cont

AI-assisted analysis of modelcontextprotocol/servers@76d64c822f (2026-08-12). Data as JSON: /api/errors/4e819e4e87d1929d. Report an issue: GitHub.