sgl-project/sglang · error · RuntimeError

Namespace '{namespace}' for rank {rank} not initialized

Error message

Namespace '{namespace}' for rank {rank} not initialized

What it means

This is the in-process Hf3fsLocalMetadataClient mirror of the server check: _get_metadata looks up self._metadata['{rank}:{namespace}'] and raises RuntimeError when that namespace was never initialized locally. Single-machine deployments hit this instead of the HTTP 404.

Source

Thrown at python/sglang/srt/mem_cache/storage/hf3fs/mini_3fs_metadata_server.py:439

        response = self._post(
            f"{rank}/get_page_indices", {"keys": keys, "namespace": str(namespace)}
        )
        return response.get("indices")


class Hf3fsLocalMetadataClient(Hf3fsMetadataInterface):
    """Local metadata client that directly operates on RankMetadata in memory without metadata server."""

    def __init__(self):
        self._metadata: Dict[str, RankMetadata] = {}  # key: "rank:namespace"

    def _ns_key(self, rank: int, namespace: PoolName) -> str:
        return f"{rank}:{namespace}"

    def _get_metadata(self, rank: int, namespace) -> RankMetadata:
        key = self._ns_key(rank, namespace)
        if key not in self._metadata:
            raise RuntimeError(
                f"Namespace '{namespace}' for rank {rank} not initialized"
            )
        return self._metadata[key]

    def initialize(
        self, rank: int, num_pages: int, namespace: PoolName = PoolName.KV
    ) -> None:
        key = self._ns_key(rank, namespace)
        if key not in self._metadata:
            self._metadata[key] = RankMetadata(num_pages)

    def reserve_and_allocate_page_indices(
        self, rank: int, keys: List[Tuple[str, str]], namespace: PoolName = PoolName.KV
    ) -> List[Tuple[bool, int]]:
        """Reserve and allocate page indices for keys."""
        return self._get_metadata(rank, namespace).reserve_and_allocate_page_indices(
            keys
        )

View on GitHub (pinned to 0132848349)

Solutions

1. Call initialize(rank, num_pages, namespace) on the same Hf3fsLocalMetadataClient instance before any allocation calls
2. Log the exact namespace string on both sides; keys are '{rank}:{namespace}' and must match exactly
3. Treat RuntimeError from _get_metadata as an init-order bug in your startup sequence, not a transient error — do not retry

key = f'{rank}:{namespace}'
if key not in metadata_client._metadata:
    metadata_client.initialize(rank, num_pages, namespace)

try:
    metadata_client.reserve_and_allocate_page_indices(rank, keys)
except RuntimeError as e:
    if 'not initialized' in str(e):
        metadata_client.initialize(rank, num_pages, namespace)
        metadata_client.reserve_and_allocate_page_indices(rank, keys)
    else:
        raise

When it happens

Trigger: Using the local metadata client (no metadata_server_url in config) and calling reserve_and_allocate_page_indices/confirm_write/delete_keys/exists/clear/get_page_indices before initialize(rank, num_pages, namespace), or with a namespace that differs from the initialized one.

Common situations: Reordered startup where the writer connects before the pool initializes; namespace mismatch between an 'mla' pool config and 'kv' default; object re-created (losing in-memory _metadata) while callers hold stale references.

Related errors

• Rank {rank} namespace '{namespace}' not initialized. Please
• {name} must have dtype torch.int32
• {name} must be on the same device as block sparse tensors
• {name} must live on CUDA
• Invalid attention metadata values.Sparsity should be in [0,

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/b054921140faeffb. Report an issue: GitHub.