{"record":{"id":"e9cc079fafe8c2f6","repo":"vllm-project/vllm","slug":"rank-rank-not-initialized","errorCode":null,"errorMessage":"Rank {rank} not initialized","messagePattern":"Rank (.+?) not initialized","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"vllm/distributed/kv_transfer/kv_connector/v1/hf3fs/hf3fs_metadata_server.py","lineNumber":125,"sourceCode":"                    rank, num_pages, list(range(num_pages))\n                )\n                logger.info(\"Initialized rank %s with %s pages\", rank, num_pages)\n\n    def allocate_pages_for_keys(\n        self, rank: int, keys: list[tuple[str, str]]\n    ) -> dict[str, int]:\n        \"\"\"Allocate one page for each key on the specified rank.\n\n        Args:\n            rank: Rank ID to allocate pages on\n            keys: List of keys to allocate pages for\n\n        Returns:\n            Dictionary mapping key -> allocated page index\n        \"\"\"\n        with self.global_lock:\n            if rank not in self.rank_metadata:\n                raise ValueError(f\"Rank {rank} not initialized\")\n\n            # Batch allocate pages for all keys\n            num_pages_needed = len(keys)\n            allocated_pages = self.rank_metadata[rank].allocate_pages(num_pages_needed)\n\n            if len(allocated_pages) < num_pages_needed:\n                logger.warning(\n                    \"Rank %s only allocated %s pages for %s keys\",\n                    rank,\n                    len(allocated_pages),\n                    num_pages_needed,\n                )\n\n            allocation_results = {}\n            for i, (key, prefix_key) in enumerate(keys):\n                if key in self.key_metadata:\n                    key_meta = self.key_metadata[key]\n                    if key_meta.is_complete() and rank in key_meta.rank_to_page:","sourceCodeStart":107,"sourceCodeEnd":143,"githubUrl":"https://github.com/vllm-project/vllm/blob/c794754062d49a8fdb63ab3c5215b488b865030c/vllm/distributed/kv_transfer/kv_connector/v1/hf3fs/hf3fs_metadata_server.py#L107-L143","documentation":"The HF3FS metadata server's shared state only tracks ranks that were explicitly initialized via initialize_rank(rank, num_pages). allocate_pages_for_keys looks up self.rank_metadata[rank]; if the rank never registered, it raises 'Rank {rank} not initialized' while holding the global lock.","triggerScenarios":"A worker calls the /allocate (batch_allocate_pages_for_keys) HTTP endpoint before its /initialize call for that rank completed, or with a wrong rank number.","commonSituations":"Race at startup: worker's first allocation request reaches the metadata server before initialization; rank ID mismatch between initializer and allocator (e.g. TP/PP rank computation differs); metadata server restarted (losing state) while workers kept running.","solutions":["Ensure each rank completes initialization (POST with role='worker', rank, num_pages>0) before any allocation request","Verify the rank value sent by the allocator matches the one used at initialization","If the metadata server restarted, re-initialize all ranks before resuming transfers"],"exampleFix":"# before: allocate before init\nawait client.post('/allocate', json={'rank': 3, 'keys': [...]})  # ValueError\n\n# after: init first\nawait client.post('/initialize', json={'rank': 3, 'role': 'worker', 'num_pages': 1024})\nawait client.post('/allocate', json={'rank': 3, 'keys': [...]})","handlingStrategy":"validation","validationCode":"resp = requests.post(f'{server}/initialize',\n                      json={'rank': rank, 'role': 'worker', 'num_pages': n_pages})\nresp.raise_for_status()  # only then allocate\nresp = requests.post(f'{server}/allocate', json={'rank': rank, 'keys': keys})","typeGuard":null,"tryCatchPattern":"try:\n    results = state.allocate_pages_for_keys(rank, keys)\nexcept ValueError as e:\n    if 'not initialized' in str(e):\n        state.initialize_rank(rank, num_pages)\n        results = state.allocate_pages_for_keys(rank, keys)\n    else:\n        raise","preventionTips":["Make initialization the first request each rank sends to the metadata server","Use one shared rank-numbering helper on producer and consumer sides","Re-initialize ranks after metadata-server restarts"],"tags":["kv-transfer","hf3fs","metadata-server","initialization","vllm"],"backgroundTag":null,"analyzedSha":"c794754062d49a8fdb63ab3c5215b488b865030c","analyzedAt":"2026-08-14T21:17:39.825Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}