What causes "Rank {rank} not initialized" in vllm-project/vllm?

Ensure each rank completes initialization (POST with role='worker', rank, num_pages>0) before any allocation request

vllm-project/vllm · error · ValueError

Rank {rank} not initialized

Error message

Rank {rank} not initialized

What it means

The HF3FS metadata server's shared state only tracks ranks that were explicitly initialized via initialize_rank(rank, num_pages). allocate_pages_for_keys looks up self.rank_metadata[rank]; if the rank never registered, it raises 'Rank {rank} not initialized' while holding the global lock.

Source

Thrown at vllm/distributed/kv_transfer/kv_connector/v1/hf3fs/hf3fs_metadata_server.py:125

                    rank, num_pages, list(range(num_pages))
                )
                logger.info("Initialized rank %s with %s pages", rank, num_pages)

    def allocate_pages_for_keys(
        self, rank: int, keys: list[tuple[str, str]]
    ) -> dict[str, int]:
        """Allocate one page for each key on the specified rank.

        Args:
            rank: Rank ID to allocate pages on
            keys: List of keys to allocate pages for

        Returns:
            Dictionary mapping key -> allocated page index
        """
        with self.global_lock:
            if rank not in self.rank_metadata:
                raise ValueError(f"Rank {rank} not initialized")

            # Batch allocate pages for all keys
            num_pages_needed = len(keys)
            allocated_pages = self.rank_metadata[rank].allocate_pages(num_pages_needed)

            if len(allocated_pages) < num_pages_needed:
                logger.warning(
                    "Rank %s only allocated %s pages for %s keys",
                    rank,
                    len(allocated_pages),
                    num_pages_needed,
                )

            allocation_results = {}
            for i, (key, prefix_key) in enumerate(keys):
                if key in self.key_metadata:
                    key_meta = self.key_metadata[key]
                    if key_meta.is_complete() and rank in key_meta.rank_to_page:

View on GitHub (pinned to c794754062)

Solutions

Ensure each rank completes initialization (POST with role='worker', rank, num_pages>0) before any allocation request
Verify the rank value sent by the allocator matches the one used at initialization
If the metadata server restarted, re-initialize all ranks before resuming transfers

Example fix

# before: allocate before init
await client.post('/allocate', json={'rank': 3, 'keys': [...]})  # ValueError

# after: init first
await client.post('/initialize', json={'rank': 3, 'role': 'worker', 'num_pages': 1024})
await client.post('/allocate', json={'rank': 3, 'keys': [...]})

Defensive patterns

Strategy: validation

Validate before calling

resp = requests.post(f'{server}/initialize',
                      json={'rank': rank, 'role': 'worker', 'num_pages': n_pages})
resp.raise_for_status()  # only then allocate
resp = requests.post(f'{server}/allocate', json={'rank': rank, 'keys': keys})

Try / catch

try:
    results = state.allocate_pages_for_keys(rank, keys)
except ValueError as e:
    if 'not initialized' in str(e):
        state.initialize_rank(rank, num_pages)
        results = state.allocate_pages_for_keys(rank, keys)
    else:
        raise

Prevention

Make initialization the first request each rank sends to the metadata server
Use one shared rank-numbering helper on producer and consumer sides
Re-initialize ranks after metadata-server restarts

When it happens

Trigger: A worker calls the /allocate (batch_allocate_pages_for_keys) HTTP endpoint before its /initialize call for that rank completed, or with a wrong rank number.

Common situations: Race at startup: worker's first allocation request reaches the metadata server before initialization; rank ID mismatch between initializer and allocator (e.g. TP/PP rank computation differs); metadata server restarted (losing state) while workers kept running.

Related errors

AI-assisted analysis of vllm-project/vllm@c794754062 (2026-08-14). Data as JSON: /api/errors/e9cc079fafe8c2f6. Report an issue: GitHub.