{"record":{"id":"d4676b1a01a09a71","repo":"vllm-project/vllm","slug":"failed-to-get-key-locations-str-e","errorCode":null,"errorMessage":"Failed to get key locations: {str(e)}","messagePattern":"Failed to get key locations: (.+?)","errorType":"http","errorClass":"HTTPException","httpStatus":500,"severity":"error","filePath":"vllm/distributed/kv_transfer/kv_connector/v1/hf3fs/hf3fs_metadata_server.py","lineNumber":377,"sourceCode":"\n    async def get_key_locations(self, request: Request):\n        \"\"\"Get page indices for keys on a specific rank.\"\"\"\n        data = await self._read_json(request)\n        rank = data.get(\"rank\")\n        keys = data.get(\"keys\", [])\n\n        # Validate input format\n        if rank is None or not isinstance(keys, list):\n            raise HTTPException(\n                status_code=400, detail=\"Invalid request format: need 'rank' and 'keys'\"\n            )\n\n        try:\n            # Get key locations\n            locations = self.state.get_key_locations(rank, keys)\n            return self._json_response({\"locations\": locations})\n        except Exception as e:\n            raise HTTPException(\n                status_code=500, detail=f\"Failed to get key locations: {str(e)}\"\n            ) from e\n\n    async def clear(self, request: Request):\n        \"\"\"Clear the metadata server.\"\"\"\n        self.state.clear()\n        return Response(status_code=204)\n\n    def run(self, host: str = \"0.0.0.0\", port: int = 18000):\n        \"\"\"Run the metadata server.\"\"\"\n        import uvicorn\n\n        logger.info(\"Starting improved metadata server on http://%s:%s\", host, port)\n        uvicorn.run(self.app, host=host, port=port)\n\n\n# --- Client implementation ---\nclass Hf3fsMetadataInterface(ABC):","sourceCodeStart":359,"sourceCodeEnd":395,"githubUrl":"https://github.com/vllm-project/vllm/blob/c794754062d49a8fdb63ab3c5215b488b865030c/vllm/distributed/kv_transfer/kv_connector/v1/hf3fs/hf3fs_metadata_server.py#L359-L395","documentation":"HTTP 500 raised when HF3FSMetadataServerState.get_key_locations(rank, keys) raises after passing format validation. The response detail embeds the original exception text and it is chained, so the root cause (usually a key not registered for that rank or unhashable key values) is visible in the 500 body. This is a server-side lookup failure, not a payload-shape problem.","triggerScenarios":"Querying locations for keys that were never allocated on the given rank; passing non-hashable key entries; rank not initialized via the rank/{rank}/initialize route before the lookup.","commonSituations":"Prefill and decode workers disagreeing on which keys exist (timing race where lookup precedes allocation); querying a rank after it was cleared; client-side key-format drift.","solutions":["Read the '{str(e)}' detail in the 500 response — it carries the underlying state exception.","Call batch_key_exists first to filter keys that are not yet present, avoiding lookups of unknown keys.","Ensure the target rank completed initialize() and allocate_pages_for_keys() before querying locations.","If the race is systematic, check that lookup requests are issued after allocation confirmations."],"exampleFix":"// before\nlocations = client.get_key_locations(rank, keys)\n// after\nexisting = client.batch_key_exists(keys)[\"exists\"]\nlocations = client.get_key_locations(rank, [k for k, ok in zip(keys, existing) if ok])","handlingStrategy":"validation","validationCode":"exists = client.batch_key_exists(keys)[\"exists\"]\nlocations = client.get_key_locations(rank, [k for k, ok in zip(keys, exists) if ok])","typeGuard":null,"tryCatchPattern":"Catch HTTP 500; read the embedded root-cause detail; filter out unknown keys and retry once — persistent failure means server state problems.","preventionTips":["Check key existence before location lookups","Ensure target rank is initialized before querying"],"tags":["hf3fs","metadata-server","http-500","kv-transfer"],"backgroundTag":null,"analyzedSha":"c794754062d49a8fdb63ab3c5215b488b865030c","analyzedAt":"2026-08-14T21:17:39.825Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}