vllm-project/vllm · critical · RuntimeError

Failed to connect to metadata server: {e}

Error message

Failed to connect to metadata server: {e}

What it means

RuntimeError raised by HF3FSMetadataClient._post when a requests POST to the metadata server fails with a RequestException after all retries (connection refused, timeout, DNS failure, etc.). The log line 'Failed to POST to <endpoint> after retries' names the endpoint and the underlying network error. It means the client could not get any HTTP response from the metadata server, not that the server rejected the request.

Source

Thrown at vllm/distributed/kv_transfer/kv_connector/v1/hf3fs/hf3fs_metadata_server.py:469

            headers = {"Content-Type": "application/json"}
            if HAS_ORJSON:
                payload = orjson.dumps(json_data)
            else:
                import json

                payload = json.dumps(json_data).encode("utf-8")
            response = self._session.post(url, data=payload, headers=headers)
            response.raise_for_status()

            if response.status_code == 204 or not response.content:
                return {}
            if HAS_ORJSON:
                return orjson.loads(response.content)
            else:
                return response.json()
        except requests.exceptions.RequestException as e:
            logger.error("Failed to POST to %s after retries: %s", endpoint, e)
            raise RuntimeError(f"Failed to connect to metadata server: {e}") from e

    def initialize(self, rank: int, num_pages: int = 0, role: str = "worker") -> None:
        """Initialize a rank with specified number of pages."""
        self._post(f"rank/{rank}/initialize", {"num_pages": num_pages, "role": role})

    def allocate_pages_for_keys(
        self, rank: int, keys: list[tuple[str, str]]
    ) -> list[tuple[str, int]]:
        """Allocate pages for keys on the specified rank."""
        response = self._post("keys/batch_allocate", {"rank": rank, "keys": keys})

        # Convert response to expected format
        return response.get("results", {})

    def confirm_write_for_keys(
        self,
        rank: int,
        key_confirmations: list[tuple[str, int]],

View on GitHub (pinned to c794754062)

Solutions

  1. Verify the metadata server process is alive and listening on the expected host:port (curl the health/GET endpoint).
  2. Check the metadata server URL in the HF3FS connector extra config matches the server's bind address and port.
  3. Look at the metadata server logs for a crash or OOM kill and restart it if needed.
  4. If retries are exhausting due to load, increase client timeout/retry settings or reduce concurrent workers hammering the server.
Defensive patterns

Strategy: retry

Validate before calling

import socket
def server_reachable(host, port, timeout=2.0):
    try:
        with socket.create_connection((host, port), timeout=timeout):
            return True
    except OSError:
        return False

Try / catch

Catch RuntimeError from HF3FSMetadataClient calls; check server liveness (socket/health probe), restart or wait for the metadata server, then retry the request — the client already exhausted its internal retries.

Prevention

When it happens

Trigger: Metadata server process not started or crashed; wrong host/port in the kv_transfer_config extra config; firewall or network partition between worker and server; server saturated so every retry times out; server bound to a different interface than the client dials.

Common situations: Launching vLLM workers without first starting the HF3FS metadata server; port mismatch between the server's --port and the client's configured URL; metadata server OOM-killed mid-run; container networking where 0.0.0.0 binding is unreachable from another container network.

Related errors

AI-assisted analysis of vllm-project/vllm@c794754062 (2026-08-14). Data as JSON: /api/errors/74e73bc91d862f9e. Report an issue: GitHub.