{"record":{"id":"d98705ac1cf9314b","repo":"BerriAI/litellm","slug":"fault-tag","errorCode":"{fault.tag}","errorMessage":"Failed to list tools from server {get_server_prefix(server)}","messagePattern":"Failed to list tools from server (.+?)","errorType":"http","errorClass":"HTTPException","httpStatus":null,"severity":"error","filePath":"litellm/proxy/_experimental/mcp_server/rest_endpoints.py","lineNumber":647,"sourceCode":"\n        try:\n            list_tools_result: Final = await _get_tools_for_single_server(\n                server,\n                server_auth_header,\n                raw_headers_from_request,\n                user_api_key_dict,\n                extra_headers=user_oauth_extra_headers,\n                apply_tool_filters=apply_tool_filters,\n            )\n        except MCPUpstreamAuthError:\n            # Surface the upstream 401/403 to the caller so it can emit the\n            # matching status code and WWW-Authenticate challenge; that is what\n            # lets standards-compliant MCP clients run the upstream OAuth flow.\n            raise\n        except MCPServerListError as e:\n            fault: Final = classify_list_exception(e)\n            verbose_logger.info(\"Listing tools from %s failed with a %s fault\", server.name, fault.tag)\n            raise HTTPException(\n                status_code=list_fault_http_status(fault),\n                detail={\n                    \"error\": fault.tag,\n                    \"message\": f\"Failed to list tools from server {get_server_prefix(server)}\",\n                },\n            ) from e\n        except Exception as e:\n            verbose_logger.exception(\"Error getting tools from %s: %s\", server.name, e)\n            return {\n                \"tools\": [],\n                \"error\": \"server_error\",\n                \"message\": f\"Failed to get tools from server {server.name}: {e}\",\n            }\n        return {\n            \"tools\": list_tools_result,\n            \"error\": None,\n            \"message\": \"Successfully retrieved tools\",\n        }","sourceCodeStart":629,"sourceCodeEnd":665,"githubUrl":"https://github.com/BerriAI/litellm/blob/77b7c6c40c0c5aa5fbcb1d6a1825ac39ca8829b8/litellm/proxy/_experimental/mcp_server/rest_endpoints.py#L629-L665","documentation":"A single-server tools/list failed and the exception was classified by classify_list_exception into a ServerListFault tag (auth_required, forbidden, timeout, unreachable, upstream_error, internal). The HTTP status follows list_fault_http_status: upstream 401/403 for auth faults, 504 for timeout, 502 for unreachable/upstream_error, 500 for gateway-internal. The body carries error=fault.tag plus this message naming the server prefix. Upstream 401/403 with a challenge is re-raised separately as MCPUpstreamAuthError so WWW-Authenticate can be relayed.","triggerScenarios":"Upstream MCP server down or DNS failing (unreachable -> 502); upstream read timeout (-> 504); upstream returned 5xx (upstream_error -> 502); upstream requires auth the proxy lacks (auth_required/forbidden -> 401/403); unexpected exception in the gateway itself (internal -> 500).","commonSituations":"Flaky internal MCP servers; wrong upstream URL; expired upstream API key; upstream behind a broken gateway returning 502s; slow upstreams exceeding the read timeout under load.","solutions":["Branch on the body's error tag: timeout -> check upstream latency/raise the client timeout; unreachable -> check URL/DNS/firewall; auth_required/forbidden -> fix the upstream credentials for that server; upstream_error -> inspect upstream logs; internal -> check proxy verbose logs.","Retry only the transient tags (timeout, unreachable, upstream_error) with backoff.","For fleet-wide visibility without failing the call, use the all-servers tools/list and read per-server statuses from the result _meta under litellm.ai/server_outcomes."],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"TRANSIENT = {\"timeout\", \"unreachable\", \"upstream_error\"}\nfor attempt in range(4):\n    resp = await client.get(f\"{proxy}/mcp/tools/list\", params={\"server_id\": sid}, headers=headers)\n    if resp.status_code != 200:\n        tag = resp.json().get(\"detail\", {}).get(\"error\")\n        if tag in TRANSIENT and attempt < 3:\n            await asyncio.sleep(2 ** attempt)\n            continue\n    break\nresp.raise_for_status()","preventionTips":["Branch on the body's error tag: only timeout/unreachable/upstream_error are worth retrying; 401/403 need credential fixes.","Use the aggregate tools/list and read _meta['litellm.ai/server_outcomes'] to degrade gracefully instead of hard-failing.","Track per-server fault rates to catch chronically unhealthy upstreams early."],"tags":["mcp","upstream","tools-list","http-502","http-504","fault-classification"],"backgroundTag":"upstream-server-unreachable","analyzedSha":"77b7c6c40c0c5aa5fbcb1d6a1825ac39ca8829b8","analyzedAt":"2026-08-18T11:44:31.656Z","schemaVersion":2},"datasetVersion":"2026-08-25T06:17:31.827Z"}