vllm-project/vllm · error · HTTPException

Invalid keys format

Error message

Invalid keys format

What it means

HTTP 400 from the HF3FS metadata server's batch_key_exists endpoint when the POST body's 'keys' field is present but not a JSON list (it defaults to [] when absent, so only a wrong-typed value triggers this). The server validates the shape before touching state, so nothing is mutated when this is raised. It indicates the client serialized the keys array incorrectly.

Source

Thrown at vllm/distributed/kv_transfer/kv_connector/v1/hf3fs/hf3fs_metadata_server.py:350

        try:
            self.state.confirm_write_for_keys(rank, confirmations, pages_to_release)

            return Response(status_code=204)

        except Exception as e:
            logger.error("Confirm write for keys failed: %s", e)
            raise HTTPException(
                status_code=500, detail=f"Confirmation failed: {str(e)}"
            ) from e

    async def batch_key_exists(self, request: Request):
        """Check if multiple keys exist in metadata."""
        data = await self._read_json(request)
        keys = data.get("keys", [])

        if not isinstance(keys, list):
            raise HTTPException(status_code=400, detail="Invalid keys format")

        try:
            exists_results = self.state.batch_key_exists(keys)
            return self._json_response({"exists": exists_results})
        except Exception as e:
            raise HTTPException(
                status_code=500, detail=f"Key existence check failed: {str(e)}"
            ) from e

    async def get_key_locations(self, request: Request):
        """Get page indices for keys on a specific rank."""
        data = await self._read_json(request)
        rank = data.get("rank")
        keys = data.get("keys", [])

        # Validate input format
        if rank is None or not isinstance(keys, list):
            raise HTTPException(

View on GitHub (pinned to c794754062)

Solutions

1. Send 'keys' as a JSON array, e.g. {'keys': ['sess|block0', 'sess|block1']}.
2. Omitting 'keys' entirely is also accepted (defaults to an empty list) if you only want a no-op check.
3. Align connector client and metadata server versions if the wire format changed.

Example fix

// before
{"keys": "sess|blk"}
// after
{"keys": ["sess|blk"]}

assert isinstance(keys, list) and all(isinstance(k, str) for k in keys)
client.post("keys/batch_key_exists", json={"keys": keys})

def is_key_list(v) -> bool:
    return isinstance(v, list) and all(isinstance(k, str) for k in v)

When it happens

Trigger: POST to the key-existence route with {'keys': 'my_key'} or {'keys': {'k': 1}} instead of {'keys': ['my_key']}; sending a JSON string body that decodes to a non-list.

Common situations: Client code that json.dumps a single key instead of a list of keys; schema drift between connector and metadata server versions; ad-hoc curl tests with malformed bodies.

Related errors

• Invalid request format: need 'rank' and 'confirmations'
• Invalid initialization parameters
• Invalid request format: need 'rank' and 'keys'
• Rank {rank} not initialized
• Allocation failed: {str(e)}

AI-assisted analysis of vllm-project/vllm@c794754062 (2026-08-14). Data as JSON: /api/errors/87efbf05d3b370fb. Report an issue: GitHub.