{"record":{"id":"f719e4500d6c6199","repo":"sgl-project/sglang","slug":"failed-to-connect-to-metadata-server-e","errorCode":null,"errorMessage":"Failed to connect to metadata server: {e}","messagePattern":"Failed to connect to metadata server: (.+?)","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"critical","filePath":"python/sglang/srt/mem_cache/storage/hf3fs/mini_3fs_metadata_server.py","lineNumber":368,"sourceCode":"        adapter = HTTPAdapter(\n            max_retries=retry_strategy, pool_connections=256, pool_maxsize=256\n        )\n        self._session.mount(\"http://\", adapter)\n\n    def _post(self, endpoint: str, json_data: dict) -> dict:\n        try:\n            url = f\"{self.base_url}/{endpoint}\"\n            headers = {\"Content-Type\": \"application/json\"}\n            payload = orjson.dumps(json_data)  # type: ignore[union-attr]\n            response = self._session.post(url, data=payload, headers=headers)\n            response.raise_for_status()\n\n            if response.status_code == 204 or not response.content:\n                return {}\n            return orjson.loads(response.content)  # type: ignore[union-attr]\n        except requests.exceptions.RequestException as e:\n            logging.error(f\"Failed to POST to {endpoint} after retries: {e}\")\n            raise RuntimeError(f\"Failed to connect to metadata server: {e}\") from e\n\n    def initialize(\n        self, rank: int, num_pages: int, namespace: PoolName = PoolName.KV\n    ) -> None:\n        self._post(\n            f\"{rank}/initialize\", {\"num_pages\": num_pages, \"namespace\": str(namespace)}\n        )\n\n    def reserve_and_allocate_page_indices(\n        self, rank: int, keys: List[Tuple[str, str]], namespace: PoolName = PoolName.KV\n    ) -> List[Tuple[bool, int]]:\n        response = self._post(\n            f\"{rank}/reserve_and_allocate_page_indices\",\n            {\"keys\": keys, \"namespace\": str(namespace)},\n        )\n        return [tuple(item) for item in response.get(\"indices\")]\n\n    def confirm_write(","sourceCodeStart":350,"sourceCodeEnd":386,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/mem_cache/storage/hf3fs/mini_3fs_metadata_server.py#L350-L386","documentation":"The client-side _post retries requests.exceptions.RequestException several times; when all retries fail it logs the endpoint error and raises RuntimeError('Failed to connect to metadata server: ...'). This wraps network-level failures (refused, timeout, DNS) into a single startup-friendly error.","triggerScenarios":"initialize/reserve/confirm/delete/exists/clear called when the metadata server URL is unreachable: wrong URL, server not started, port blocked by firewall, or pod-to-pod networking broken in Kubernetes.","commonSituations":"metadata_server_url typo in the 3FS config JSON; the mini_3fs_metadata_server process down or on a different node; network policy blocking the port; server restarted and slow to accept.","solutions":["Verify the server is listening: curl http://<metadata_server_url>/health (or any route) from the same pod","Fix metadata_server_url in the HF3FS env config file and ensure all ranks point at the same server","If using Kubernetes, check NetworkPolicy/endpoint readiness before starting ranks"],"exampleFix":null,"handlingStrategy":"retry","validationCode":"import requests\n\ndef metadata_server_up(url: str) -> bool:\n    try:\n        requests.get(url, timeout=2)\n        return True\n    except requests.RequestException:\n        return False\n\nassert metadata_server_up(metadata_server_url), 'start mini_3fs_metadata_server first'","typeGuard":null,"tryCatchPattern":"for attempt in range(5):\n    try:\n        client.initialize(rank, num_pages)\n        break\n    except RuntimeError as e:\n        if 'Failed to connect to metadata server' in str(e) and attempt < 4:\n            time.sleep(2 ** attempt)\n            continue\n        raise","preventionTips":["Health-check the metadata server URL before launching any rank","Bake the correct metadata_server_url into the config JSON and validate it in deploy pipelines"],"tags":["hf3fs","metadata-server","connection-refused","retries-exhausted"],"backgroundTag":"service-unreachable","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}