{"record":{"id":"74e73bc91d862f9e","repo":"vllm-project/vllm","slug":"failed-to-connect-to-metadata-server-e","errorCode":null,"errorMessage":"Failed to connect to metadata server: {e}","messagePattern":"Failed to connect to metadata server: (.+?)","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"critical","filePath":"vllm/distributed/kv_transfer/kv_connector/v1/hf3fs/hf3fs_metadata_server.py","lineNumber":469,"sourceCode":"            headers = {\"Content-Type\": \"application/json\"}\n            if HAS_ORJSON:\n                payload = orjson.dumps(json_data)\n            else:\n                import json\n\n                payload = json.dumps(json_data).encode(\"utf-8\")\n            response = self._session.post(url, data=payload, headers=headers)\n            response.raise_for_status()\n\n            if response.status_code == 204 or not response.content:\n                return {}\n            if HAS_ORJSON:\n                return orjson.loads(response.content)\n            else:\n                return response.json()\n        except requests.exceptions.RequestException as e:\n            logger.error(\"Failed to POST to %s after retries: %s\", endpoint, e)\n            raise RuntimeError(f\"Failed to connect to metadata server: {e}\") from e\n\n    def initialize(self, rank: int, num_pages: int = 0, role: str = \"worker\") -> None:\n        \"\"\"Initialize a rank with specified number of pages.\"\"\"\n        self._post(f\"rank/{rank}/initialize\", {\"num_pages\": num_pages, \"role\": role})\n\n    def allocate_pages_for_keys(\n        self, rank: int, keys: list[tuple[str, str]]\n    ) -> list[tuple[str, int]]:\n        \"\"\"Allocate pages for keys on the specified rank.\"\"\"\n        response = self._post(\"keys/batch_allocate\", {\"rank\": rank, \"keys\": keys})\n\n        # Convert response to expected format\n        return response.get(\"results\", {})\n\n    def confirm_write_for_keys(\n        self,\n        rank: int,\n        key_confirmations: list[tuple[str, int]],","sourceCodeStart":451,"sourceCodeEnd":487,"githubUrl":"https://github.com/vllm-project/vllm/blob/c794754062d49a8fdb63ab3c5215b488b865030c/vllm/distributed/kv_transfer/kv_connector/v1/hf3fs/hf3fs_metadata_server.py#L451-L487","documentation":"RuntimeError raised by HF3FSMetadataClient._post when a requests POST to the metadata server fails with a RequestException after all retries (connection refused, timeout, DNS failure, etc.). The log line 'Failed to POST to <endpoint> after retries' names the endpoint and the underlying network error. It means the client could not get any HTTP response from the metadata server, not that the server rejected the request.","triggerScenarios":"Metadata server process not started or crashed; wrong host/port in the kv_transfer_config extra config; firewall or network partition between worker and server; server saturated so every retry times out; server bound to a different interface than the client dials.","commonSituations":"Launching vLLM workers without first starting the HF3FS metadata server; port mismatch between the server's --port and the client's configured URL; metadata server OOM-killed mid-run; container networking where 0.0.0.0 binding is unreachable from another container network.","solutions":["Verify the metadata server process is alive and listening on the expected host:port (curl the health/GET endpoint).","Check the metadata server URL in the HF3FS connector extra config matches the server's bind address and port.","Look at the metadata server logs for a crash or OOM kill and restart it if needed.","If retries are exhausting due to load, increase client timeout/retry settings or reduce concurrent workers hammering the server."],"exampleFix":null,"handlingStrategy":"retry","validationCode":"import socket\ndef server_reachable(host, port, timeout=2.0):\n    try:\n        with socket.create_connection((host, port), timeout=timeout):\n            return True\n    except OSError:\n        return False","typeGuard":null,"tryCatchPattern":"Catch RuntimeError from HF3FSMetadataClient calls; check server liveness (socket/health probe), restart or wait for the metadata server, then retry the request — the client already exhausted its internal retries.","preventionTips":["Start the metadata server before workers and add a readiness gate in launch scripts","Monitor the metadata server process and auto-restart it","Verify host/port config on both sides at startup"],"tags":["hf3fs","metadata-server","network","connection-failure","kv-transfer"],"backgroundTag":null,"analyzedSha":"c794754062d49a8fdb63ab3c5215b488b865030c","analyzedAt":"2026-08-14T21:17:39.825Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}