{"record":{"id":"4e819e4e87d1929d","repo":"modelcontextprotocol/servers","slug":"32603","errorCode":"-32603","errorMessage":"Failed to fetch robots.txt {robot_txt_url} due to a connection issue","messagePattern":"Failed to fetch robots\\.txt (.+?) due to a connection issue","errorType":"error_code","errorClass":"McpError","httpStatus":null,"severity":"warning","filePath":"src/fetch/src/mcp_server_fetch/server.py","lineNumber":83,"sourceCode":"\nasync def check_may_autonomously_fetch_url(url: str, user_agent: str, proxy_url: str | None = None) -> None:\n    \"\"\"\n    Check if the URL can be fetched by the user agent according to the robots.txt file.\n    Raises a McpError if not.\n    \"\"\"\n    from httpx import AsyncClient, HTTPError\n\n    robot_txt_url = get_robots_txt_url(url)\n\n    async with AsyncClient(proxy=proxy_url) as client:\n        try:\n            response = await client.get(\n                robot_txt_url,\n                follow_redirects=True,\n                headers={\"User-Agent\": user_agent},\n            )\n        except HTTPError:\n            raise McpError(ErrorData(\n                code=INTERNAL_ERROR,\n                message=f\"Failed to fetch robots.txt {robot_txt_url} due to a connection issue\",\n            ))\n        if response.status_code in (401, 403):\n            raise McpError(ErrorData(\n                code=INTERNAL_ERROR,\n                message=f\"When fetching robots.txt ({robot_txt_url}), received status {response.status_code} so assuming that autonomous fetching is not allowed, the user can try manually fetching by using the fetch prompt\",\n            ))\n        elif 400 <= response.status_code < 500:\n            return\n        robot_txt = response.text\n    processed_robot_txt = \"\\n\".join(\n        line for line in robot_txt.splitlines() if not line.strip().startswith(\"#\")\n    )\n    robot_parser = Protego.parse(processed_robot_txt)\n    if not robot_parser.can_fetch(str(url), user_agent):\n        raise McpError(ErrorData(\n            code=INTERNAL_ERROR,","sourceCodeStart":65,"sourceCodeEnd":101,"githubUrl":"https://github.com/modelcontextprotocol/servers/blob/76d64c822f5125032f89eb71dbdb94e42b434821/src/fetch/src/mcp_server_fetch/server.py#L65-L101","documentation":"In check_may_autonomously_fetch_url(), before fetching the target the server GETs {origin}/robots.txt. If that request raises an httpx.HTTPError (DNS failure, connection refused, TLS/cert error, proxy failure, or timeout), it is surfaced as McpError code INTERNAL_ERROR (-32603). The error names the robot_txt_url that could not be reached.","triggerScenarios":"Origin host offline; DNS cannot resolve; firewall/proxy blocks egress; invalid TLS certificate; the robots.txt endpoint hangs until timeout; proxy_url is misconfigured or unreachable.","commonSituations":"Server runs in a sandbox/CI without internet egress; corporate proxy required but not configured; intranet host with no route from the server; transient network outage.","solutions":["Confirm the origin is reachable from the server environment (curl the robots.txt URL).","Provide a working proxy_url so httpx can reach the origin.","Retry once: transient DNS/TCP failures often clear immediately.","If autonomous fetch is not required, use the fetch prompt (manual User-Agent) or set ignore_robots_txt to bypass the robots check."],"exampleFix":"# before\nawait check_may_autonomously_fetch_url(url, ua, None)  # origin unreachable -> McpError\n\n# after: route through a proxy / fall back to manual\nawait check_may_autonomously_fetch_url(url, ua, proxy_url='http://proxy:8080')","handlingStrategy":"retry","validationCode":"import httpx\nasync def robots_reachable(url: str, proxy_url: str | None = None) -> bool:\n    from mcp_server_fetch.server import get_robots_txt_url\n    try:\n        async with httpx.AsyncClient(proxy=proxy_url) as c:\n            r = await c.get(get_robots_txt_url(url), headers={'User-Agent':'preflight'}, timeout=10)\n            return r.status_code < 500\n    except httpx.HTTPError:\n        return False","typeGuard":null,"tryCatchPattern":"from mcp.shared.exceptions import McpError\nfor attempt in range(3):\n    try:\n        await check_may_autonomously_fetch_url(url, ua, proxy_url)\n        break\n    except McpError as e:\n        if 'connection issue' not in str(e) or attempt == 2:\n            raise\n        await asyncio.sleep(2 ** attempt)","preventionTips":["Confirm egress (including proxy) works from the server host before deploying.","Configure proxy_url when the network requires it.","Treat robots.txt connection failures as transient; retry with backoff before giving up.","For air-gapped environments, prefer the manual fetch prompt or set ignore_robots_txt."],"tags":["fetch","python","network","robots-txt","connection-error"],"backgroundTag":null,"analyzedSha":"76d64c822f5125032f89eb71dbdb94e42b434821","analyzedAt":"2026-08-12T10:02:41.718Z","schemaVersion":2},"datasetVersion":"2026-08-12T13:17:24.610Z"}