sgl-project/sglang · critical · RuntimeError

Failed to setup Mooncake store, error code: {ret_code}

Error message

Failed to setup Mooncake store, error code: {ret_code}

What it means

MooncakeDistributedStore.setup() returned a non-zero code after the kwarg-fallback logic ran; the store (transfer engine session with the metadata service) could not be established. The numeric code is included.

Source

Thrown at python/sglang/srt/mem_cache/storage/mooncake_store/mooncake_store.py:548

                        if not unsupported_kwargs:
                            raise
                        if "tenant_id" in unsupported_kwargs:
                            raise RuntimeError(
                                "The installed Mooncake version does not support "
                                "tenant_id in MooncakeDistributedStore.setup(). "
                                "Please upgrade Mooncake to use non-default "
                                "Mooncake tenants with SGLang."
                            ) from e
                        logger.warning(
                            "The installed Mooncake version does not support the "
                            f"{', '.join(unsupported_kwargs)} parameter(s) in setup(). "
                            f"Retrying without {', '.join(unsupported_kwargs)}. "
                            "Please upgrade Mooncake to enable SSD offload support."
                        )
                        for key in unsupported_kwargs:
                            setup_kwargs.pop(key, None)
            if ret_code:
                raise RuntimeError(
                    f"Failed to setup Mooncake store, error code: {ret_code}"
                )
            logger.info("Mooncake store setup successfully.")

            self.local_rank = (
                storage_config.tp_rank if storage_config is not None else 0
            )
            self.warmup()
            logger.info("Mooncake store warmup successfully.")

            self.enable_storage_metrics = False
            if storage_config is not None:
                self.is_mla_backend = storage_config.is_mla_model
                self.pp_rank = storage_config.pp_rank
                self.pp_size = storage_config.pp_size
                self.attn_cp_rank = storage_config.attn_cp_rank
                self.attn_cp_size = storage_config.attn_cp_size
                self.enable_storage_metrics = storage_config.enable_storage_metrics

View on GitHub (pinned to 0132848349)

Solutions

1. Confirm mooncake master/transfer-engine metadata service is up and reachable at the configured address
2. Match MOONCAKE_PROTOCOL to hardware (use tcp if no RDMA)
3. Re-check config file / env / extra_config values
4. Upgrade mooncake-transfer-engine and retry

import socket
def addr_up(addr: str) -> bool:
    host, _, port = addr.rpartition(':')
    try:
        with socket.create_connection((host, int(port)), timeout=3):
            return True
    except OSError:
        return False
assert addr_up(cfg.master_server_address)

for attempt in range(3):
    try:
        ms = MooncakeStore(...)
        break
    except RuntimeError as e:
        if 'Failed to setup Mooncake store' in str(e) and attempt < 2:
            time.sleep(5)
            continue
        raise

When it happens

Trigger: Wrong master_server_address, unreachable metadata service, unsupported protocol/device combo, or invalid setup kwargs after unsupported ones were dropped (e.g. SSD offload args removed with a warning).

Common situations: Mooncake master down; firewall between SGLang node and master; RDMA protocol without RDMA NIC; degraded capability after retry-without-kwargs fallback still failed.

Related errors

• Failed to setup Mooncake Embedding Store: {ret_code}
• Mooncake transfer_sync failed for {req_id} (session={session
• Failed to register buffer to Mooncake Store, error code: {re
• Failed to get server info. {error_data['error']['message']}
• Failed to get model info: {str(e)}

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/b0d8737e2ed97503. Report an issue: GitHub.