sgl-project/sglang · error · HTTPException

Rank {rank} namespace '{namespace}' not initialized. Please

Error message

Rank {rank} namespace '{namespace}' not initialized. Please call /{rank}/initialize first.

What it means

This is the FastAPI metadata server's handler: get_rank_metadata looks up state.ranks['{rank}:{namespace}'] and returns HTTP 404 when the rank/namespace pair was never initialized via POST /{rank}/initialize. It exists to make out-of-order startup fail loudly instead of silently allocating wrong pages.

Source

Thrown at python/sglang/srt/mem_cache/storage/hf3fs/mini_3fs_metadata_server.py:224

        self.app.post("/{rank}/initialize")(self.initialize)
        self.app.post("/{rank}/exists")(self.exists)
        self.app.post("/{rank}/reserve_and_allocate_page_indices")(
            self.reserve_and_allocate_page_indices
        )
        self.app.post("/{rank}/confirm_write")(self.confirm_write)
        self.app.post("/{rank}/delete_keys")(self.delete_keys)
        self.app.post("/{rank}/clear")(self.clear)
        self.app.post("/{rank}/get_page_indices")(self.get_page_indices)

    def _rank_key(self, rank: int, namespace: str) -> str:
        """Generate the composite key for rank+namespace."""
        return f"{rank}:{namespace}"

    def get_rank_metadata(self, rank: int, namespace: str = "kv") -> RankMetadata:
        """Get rank metadata with proper error handling."""
        key = self._rank_key(rank, namespace)
        if key not in self.state.ranks:
            raise HTTPException(
                status_code=404,
                detail=f"Rank {rank} namespace '{namespace}' not initialized. Please call /{rank}/initialize first.",
            )
        return self.state.ranks[key]

    async def _read_json(self, request: Request) -> dict:
        """Parse request JSON using orjson if available."""
        body = await request.body()
        return orjson.loads(body)

    def _json_response(self, content: dict):
        """Return ORJSONResponse when available to bypass jsonable_encoder."""
        return ORJSONResponse(content)

    async def initialize(self, rank: int, request: Request):
        """Initialize a rank with specified number of pages."""
        data = await self._read_json(request)
        num_pages = data["num_pages"]

View on GitHub (pinned to 0132848349)

Solutions

  1. Call POST /{rank}/initialize with num_pages and namespace before any other endpoint for that rank
  2. Verify the namespace string matches exactly what initialize used (PoolName.KV vs others, case-sensitive key '{rank}:{namespace}')
  3. After a metadata-server restart, restart all ranks or re-run initialize — state is in-memory only

Example fix

# before
meta.reserve_and_allocate_page_indices(rank=0, ...)

# after
meta.initialize(rank=0, num_pages=N, namespace=PoolName.KV)
meta.reserve_and_allocate_page_indices(rank=0, ...)
Defensive patterns

Strategy: validation

Validate before calling

resp = requests.get(f'{server}/{rank}/initialized?namespace={ns}')
if resp.status_code == 404:
    requests.post(f'{server}/{rank}/initialize',
                  json={'num_pages': N, 'namespace': ns})

Try / catch

try:
    meta.get_page_indices(rank, keys)
except requests.HTTPError as e:
    if e.response.status_code == 404 and 'not initialized' in e.response.text:
        initialize_rank(server, rank, ns)
        meta.get_page_indices(rank, keys)
    else:
        raise

Prevention

When it happens

Trigger: A client (exists, reserve_and_allocate_page_indices, confirm_write, delete_keys, clear, get_page_indices) hitting the server before that rank called initialize; or using a namespace (e.g. 'mla' vs 'kv') that was never initialized for that rank.

Common situations: Rank process restart that re-connects without re-initializing; mismatched namespace names between config and client; metadata server restarted (in-memory state lost) while ranks kept running.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/b7834a20d2bef6c2. Report an issue: GitHub.