sgl-project/sglang · critical · RuntimeError

Failed to connect to metadata server: {e}

Error message

Failed to connect to metadata server: {e}

What it means

The client-side _post retries requests.exceptions.RequestException several times; when all retries fail it logs the endpoint error and raises RuntimeError('Failed to connect to metadata server: ...'). This wraps network-level failures (refused, timeout, DNS) into a single startup-friendly error.

Source

Thrown at python/sglang/srt/mem_cache/storage/hf3fs/mini_3fs_metadata_server.py:368

        adapter = HTTPAdapter(
            max_retries=retry_strategy, pool_connections=256, pool_maxsize=256
        )
        self._session.mount("http://", adapter)

    def _post(self, endpoint: str, json_data: dict) -> dict:
        try:
            url = f"{self.base_url}/{endpoint}"
            headers = {"Content-Type": "application/json"}
            payload = orjson.dumps(json_data)  # type: ignore[union-attr]
            response = self._session.post(url, data=payload, headers=headers)
            response.raise_for_status()

            if response.status_code == 204 or not response.content:
                return {}
            return orjson.loads(response.content)  # type: ignore[union-attr]
        except requests.exceptions.RequestException as e:
            logging.error(f"Failed to POST to {endpoint} after retries: {e}")
            raise RuntimeError(f"Failed to connect to metadata server: {e}") from e

    def initialize(
        self, rank: int, num_pages: int, namespace: PoolName = PoolName.KV
    ) -> None:
        self._post(
            f"{rank}/initialize", {"num_pages": num_pages, "namespace": str(namespace)}
        )

    def reserve_and_allocate_page_indices(
        self, rank: int, keys: List[Tuple[str, str]], namespace: PoolName = PoolName.KV
    ) -> List[Tuple[bool, int]]:
        response = self._post(
            f"{rank}/reserve_and_allocate_page_indices",
            {"keys": keys, "namespace": str(namespace)},
        )
        return [tuple(item) for item in response.get("indices")]

    def confirm_write(

View on GitHub (pinned to 0132848349)

Solutions

  1. Verify the server is listening: curl http://<metadata_server_url>/health (or any route) from the same pod
  2. Fix metadata_server_url in the HF3FS env config file and ensure all ranks point at the same server
  3. If using Kubernetes, check NetworkPolicy/endpoint readiness before starting ranks
Defensive patterns

Strategy: retry

Validate before calling

import requests

def metadata_server_up(url: str) -> bool:
    try:
        requests.get(url, timeout=2)
        return True
    except requests.RequestException:
        return False

assert metadata_server_up(metadata_server_url), 'start mini_3fs_metadata_server first'

Try / catch

for attempt in range(5):
    try:
        client.initialize(rank, num_pages)
        break
    except RuntimeError as e:
        if 'Failed to connect to metadata server' in str(e) and attempt < 4:
            time.sleep(2 ** attempt)
            continue
        raise

Prevention

When it happens

Trigger: initialize/reserve/confirm/delete/exists/clear called when the metadata server URL is unreachable: wrong URL, server not started, port blocked by firewall, or pod-to-pod networking broken in Kubernetes.

Common situations: metadata_server_url typo in the 3FS config JSON; the mini_3fs_metadata_server process down or on a different node; network policy blocking the port; server restarted and slow to accept.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/f719e4500d6c6199. Report an issue: GitHub.