{"record":{"id":"b054921140faeffb","repo":"sgl-project/sglang","slug":"namespace-namespace-for-rank-rank-not-initia","errorCode":null,"errorMessage":"Namespace '{namespace}' for rank {rank} not initialized","messagePattern":"Namespace '(.+?)' for rank (.+?) not initialized","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/mem_cache/storage/hf3fs/mini_3fs_metadata_server.py","lineNumber":439,"sourceCode":"        response = self._post(\n            f\"{rank}/get_page_indices\", {\"keys\": keys, \"namespace\": str(namespace)}\n        )\n        return response.get(\"indices\")\n\n\nclass Hf3fsLocalMetadataClient(Hf3fsMetadataInterface):\n    \"\"\"Local metadata client that directly operates on RankMetadata in memory without metadata server.\"\"\"\n\n    def __init__(self):\n        self._metadata: Dict[str, RankMetadata] = {}  # key: \"rank:namespace\"\n\n    def _ns_key(self, rank: int, namespace: PoolName) -> str:\n        return f\"{rank}:{namespace}\"\n\n    def _get_metadata(self, rank: int, namespace) -> RankMetadata:\n        key = self._ns_key(rank, namespace)\n        if key not in self._metadata:\n            raise RuntimeError(\n                f\"Namespace '{namespace}' for rank {rank} not initialized\"\n            )\n        return self._metadata[key]\n\n    def initialize(\n        self, rank: int, num_pages: int, namespace: PoolName = PoolName.KV\n    ) -> None:\n        key = self._ns_key(rank, namespace)\n        if key not in self._metadata:\n            self._metadata[key] = RankMetadata(num_pages)\n\n    def reserve_and_allocate_page_indices(\n        self, rank: int, keys: List[Tuple[str, str]], namespace: PoolName = PoolName.KV\n    ) -> List[Tuple[bool, int]]:\n        \"\"\"Reserve and allocate page indices for keys.\"\"\"\n        return self._get_metadata(rank, namespace).reserve_and_allocate_page_indices(\n            keys\n        )","sourceCodeStart":421,"sourceCodeEnd":457,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/mem_cache/storage/hf3fs/mini_3fs_metadata_server.py#L421-L457","documentation":"This is the in-process Hf3fsLocalMetadataClient mirror of the server check: _get_metadata looks up self._metadata['{rank}:{namespace}'] and raises RuntimeError when that namespace was never initialized locally. Single-machine deployments hit this instead of the HTTP 404.","triggerScenarios":"Using the local metadata client (no metadata_server_url in config) and calling reserve_and_allocate_page_indices/confirm_write/delete_keys/exists/clear/get_page_indices before initialize(rank, num_pages, namespace), or with a namespace that differs from the initialized one.","commonSituations":"Reordered startup where the writer connects before the pool initializes; namespace mismatch between an 'mla' pool config and 'kv' default; object re-created (losing in-memory _metadata) while callers hold stale references.","solutions":["Call initialize(rank, num_pages, namespace) on the same Hf3fsLocalMetadataClient instance before any allocation calls","Log the exact namespace string on both sides; keys are '{rank}:{namespace}' and must match exactly","Treat RuntimeError from _get_metadata as an init-order bug in your startup sequence, not a transient error — do not retry"],"exampleFix":null,"handlingStrategy":"validation","validationCode":"key = f'{rank}:{namespace}'\nif key not in metadata_client._metadata:\n    metadata_client.initialize(rank, num_pages, namespace)","typeGuard":null,"tryCatchPattern":"try:\n    metadata_client.reserve_and_allocate_page_indices(rank, keys)\nexcept RuntimeError as e:\n    if 'not initialized' in str(e):\n        metadata_client.initialize(rank, num_pages, namespace)\n        metadata_client.reserve_and_allocate_page_indices(rank, keys)\n    else:\n        raise","preventionTips":["Call initialize immediately after constructing the local client, on the same instance","Log the exact namespace strings on producer and consumer sides to catch mismatches early"],"tags":["hf3fs","metadata","initialization-order","local-client"],"backgroundTag":"service-not-initialized","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}