{"record":{"id":"dcd6774b85cede3b","repo":"vllm-project/vllm","slug":"initialize-mooncakedistributedstore-failed","errorCode":null,"errorMessage":"Initialize MooncakeDistributedStore failed.","messagePattern":"Initialize MooncakeDistributedStore failed\\.","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"critical","filePath":"vllm/distributed/kv_transfer/kv_connector/v1/mooncake/store/worker.py","lineNumber":1278,"sourceCode":"        local_ip = get_ip()\n        local_hostname = rdma_utils.get_requester_local_hostname(local_ip)\n        setup_kwargs: dict[str, str] = {}\n        if store_config.tenant_id != DEFAULT_TENANT_ID:\n            setup_kwargs[\"tenant_id\"] = store_config.tenant_id\n        ret = self.store.setup(\n            local_hostname,\n            store_config.metadata_server,\n            store_config.global_segment_size,\n            store_config.local_buffer_size,\n            store_config.protocol,\n            store_config.device_name,\n            store_config.master_server_address,\n            **setup_kwargs,\n        )\n        if ret != 0:\n            msg = \"Initialize MooncakeDistributedStore failed.\"\n            logger.error(msg)\n            raise RuntimeError(msg)\n\n        preferred_segment = rdma_utils.get_configured_preferred_segment(extra_config)\n        self.preferred_segment = preferred_segment\n        self.store_replicate_config = ReplicateConfig()\n        self.enable_group_semantics = (\n            str(extra_config.get(\"enable_group_semantics\", \"False\")).strip().lower()\n            == \"true\"\n        )\n        self._supports_group_ids = _replicate_config_supports_group_ids(\n            ReplicateConfig, self.store_replicate_config\n        )\n        if self.enable_group_semantics and not self._supports_group_ids:\n            logger.warning(\n                \"Mooncake group semantics is enabled, but the installed \"\n                \"Mooncake package does not support ReplicateConfig.group_ids. \"\n                \"Falling back to the existing batch_put_from_multi_buffers path.\"\n            )\n        if preferred_segment is not None:","sourceCodeStart":1260,"sourceCodeEnd":1296,"githubUrl":"https://github.com/vllm-project/vllm/blob/c794754062d49a8fdb63ab3c5215b488b865030c/vllm/distributed/kv_transfer/kv_connector/v1/mooncake/store/worker.py#L1260-L1296","documentation":"MooncakeDistributedStore.initialize(...) (setup_store) returned a non-zero code, so the worker wraps it as RuntimeError after logging. The native call spans metadata_server, master_server_address, global_segment_size, local_buffer_size, protocol and device_name, so any of those being unreachable/invalid surfaces here with little extra detail from the C++ side.","triggerScenarios":"Wrong/unreachable metadata_server or master_server_address URL; protocol not matching the transport actually available ('rdma' with no RDMA NIC/verbs, 'nvmf'/'tcp' mismatches); device_name empty or naming a non-existent device; global_segment_size larger than allocatable memory; multi-master_address format errors.","commonSituations":"RDMA devices not exposed in the container (missing /dev/infiniband, wrong device_name); metadata server (e.g. etcd/redis) down or wrong port; K8s network policies blocking the master; segment size exceeding hugepage/memory limits.","solutions":["Verify connectivity to metadata_server and master_server_address from the worker pod (curl/nc) and fix hostnames/ports","Check the transport: for 'rdma' confirm the NIC exists (ibv_devices), device_name matches an RDMA device, and /dev/infiniband is mounted in containers","Lower global_segment_size to fit available memory/hugepages, and confirm local_buffer_size > 0","Run with protocol='tcp' first to isolate RDMA setup issues, then re-enable rdma","Re-run with mooncake/vLLM verbose logging and read the line logged just before — the native error string usually names the failing subsystem"],"exampleFix":"// before\n{ \"protocol\": \"rdma\", \"device_name\": \"\", \"global_segment_size\": \"64GB\" }\n\n// after (isolate transport first, then size)\n{ \"protocol\": \"tcp\",  \"device_name\": \"\", \"global_segment_size\": \"8GB\" }","handlingStrategy":"try-catch","validationCode":"# preflight: reachability and transport checks\nimport socket, subprocess\nfor host_port in (metadata_server, master_server_address):\n    host, _, port = host_port.partition(\":\")\n    socket.create_connection((host, int(port)), timeout=5).close()\nif protocol == \"rdma\":\n    out = subprocess.run([\"ibv_devices\"], capture_output=True)\n    assert out.returncode == 0 and device_name in out.stdout.decode(), \"RDMA device missing\"","typeGuard":null,"tryCatchPattern":"try:\n    store = MooncakeStoreWorker(...)  # initialize() inside\nexcept RuntimeError as e:\n    if \"Initialize MooncakeDistributedStore failed\" in str(e):\n        # check log line above for native cause; retry with protocol='tcp' to isolate\n        raise","preventionTips":["Health-check metadata/master servers before worker startup","Expose RDMA devices and mount /dev/infiniband in containers; verify with ibv_devices","Start with protocol='tcp' to validate addresses, then switch to rdma","Size global_segment_size within hugepage/memory limits"],"tags":["mooncake","rdma","initialization","network","native-error"],"backgroundTag":null,"analyzedSha":"c794754062d49a8fdb63ab3c5215b488b865030c","analyzedAt":"2026-08-14T21:17:39.825Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}