vllm-project/vllm · critical · RuntimeError

Initialize MooncakeDistributedStore failed.

Error message

Initialize MooncakeDistributedStore failed.

What it means

MooncakeDistributedStore.initialize(...) (setup_store) returned a non-zero code, so the worker wraps it as RuntimeError after logging. The native call spans metadata_server, master_server_address, global_segment_size, local_buffer_size, protocol and device_name, so any of those being unreachable/invalid surfaces here with little extra detail from the C++ side.

Source

Thrown at vllm/distributed/kv_transfer/kv_connector/v1/mooncake/store/worker.py:1278

        local_ip = get_ip()
        local_hostname = rdma_utils.get_requester_local_hostname(local_ip)
        setup_kwargs: dict[str, str] = {}
        if store_config.tenant_id != DEFAULT_TENANT_ID:
            setup_kwargs["tenant_id"] = store_config.tenant_id
        ret = self.store.setup(
            local_hostname,
            store_config.metadata_server,
            store_config.global_segment_size,
            store_config.local_buffer_size,
            store_config.protocol,
            store_config.device_name,
            store_config.master_server_address,
            **setup_kwargs,
        )
        if ret != 0:
            msg = "Initialize MooncakeDistributedStore failed."
            logger.error(msg)
            raise RuntimeError(msg)

        preferred_segment = rdma_utils.get_configured_preferred_segment(extra_config)
        self.preferred_segment = preferred_segment
        self.store_replicate_config = ReplicateConfig()
        self.enable_group_semantics = (
            str(extra_config.get("enable_group_semantics", "False")).strip().lower()
            == "true"
        )
        self._supports_group_ids = _replicate_config_supports_group_ids(
            ReplicateConfig, self.store_replicate_config
        )
        if self.enable_group_semantics and not self._supports_group_ids:
            logger.warning(
                "Mooncake group semantics is enabled, but the installed "
                "Mooncake package does not support ReplicateConfig.group_ids. "
                "Falling back to the existing batch_put_from_multi_buffers path."
            )
        if preferred_segment is not None:

View on GitHub (pinned to c794754062)

Solutions

1. Verify connectivity to metadata_server and master_server_address from the worker pod (curl/nc) and fix hostnames/ports
2. Check the transport: for 'rdma' confirm the NIC exists (ibv_devices), device_name matches an RDMA device, and /dev/infiniband is mounted in containers
3. Lower global_segment_size to fit available memory/hugepages, and confirm local_buffer_size > 0
4. Run with protocol='tcp' first to isolate RDMA setup issues, then re-enable rdma
5. Re-run with mooncake/vLLM verbose logging and read the line logged just before — the native error string usually names the failing subsystem

Example fix

// before
{ "protocol": "rdma", "device_name": "", "global_segment_size": "64GB" }

// after (isolate transport first, then size)
{ "protocol": "tcp",  "device_name": "", "global_segment_size": "8GB" }

# preflight: reachability and transport checks
import socket, subprocess
for host_port in (metadata_server, master_server_address):
    host, _, port = host_port.partition(":")
    socket.create_connection((host, int(port)), timeout=5).close()
if protocol == "rdma":
    out = subprocess.run(["ibv_devices"], capture_output=True)
    assert out.returncode == 0 and device_name in out.stdout.decode(), "RDMA device missing"

try:
    store = MooncakeStoreWorker(...)  # initialize() inside
except RuntimeError as e:
    if "Initialize MooncakeDistributedStore failed" in str(e):
        # check log line above for native cause; retry with protocol='tcp' to isolate
        raise

When it happens

Trigger: Wrong/unreachable metadata_server or master_server_address URL; protocol not matching the transport actually available ('rdma' with no RDMA NIC/verbs, 'nvmf'/'tcp' mismatches); device_name empty or naming a non-existent device; global_segment_size larger than allocatable memory; multi-master_address format errors.

Common situations: RDMA devices not exposed in the container (missing /dev/infiniband, wrong device_name); metadata server (e.g. etcd/redis) down or wrong port; K8s network policies blocking the master; segment size exceeding hugepage/memory limits.

Related errors

• Mooncake Transfer Engine initialization failed.
• Mooncake is not available
• Mooncake batch memory registration failed.
• Mooncake preferred_segment override must be a non-empty stri
• embedded mode requires global_segment_size > 0

AI-assisted analysis of vllm-project/vllm@c794754062 (2026-08-14). Data as JSON: /api/errors/dcd6774b85cede3b. Report an issue: GitHub.