vllm-project/vllm · error · HTTPException

Key existence check failed: {str(e)}

Error message

Key existence check failed: {str(e)}

What it means

HTTP 500 raised when HF3FSMetadataServerState.batch_key_exists(keys) throws while looking up keys in the metadata store. The format check has already passed, so the failure is inside state access — typically unhashable/unserializable key values or internal dict corruption. The original exception is chained via 'from e', and the response detail embeds its str().

Source

Thrown at vllm/distributed/kv_transfer/kv_connector/v1/hf3fs/hf3fs_metadata_server.py:356

        except Exception as e:
            logger.error("Confirm write for keys failed: %s", e)
            raise HTTPException(
                status_code=500, detail=f"Confirmation failed: {str(e)}"
            ) from e

    async def batch_key_exists(self, request: Request):
        """Check if multiple keys exist in metadata."""
        data = await self._read_json(request)
        keys = data.get("keys", [])

        if not isinstance(keys, list):
            raise HTTPException(status_code=400, detail="Invalid keys format")

        try:
            exists_results = self.state.batch_key_exists(keys)
            return self._json_response({"exists": exists_results})
        except Exception as e:
            raise HTTPException(
                status_code=500, detail=f"Key existence check failed: {str(e)}"
            ) from e

    async def get_key_locations(self, request: Request):
        """Get page indices for keys on a specific rank."""
        data = await self._read_json(request)
        rank = data.get("rank")
        keys = data.get("keys", [])

        # Validate input format
        if rank is None or not isinstance(keys, list):
            raise HTTPException(
                status_code=400, detail="Invalid request format: need 'rank' and 'keys'"
            )

        try:
            # Get key locations
            locations = self.state.get_key_locations(rank, keys)

View on GitHub (pinned to c794754062)

Solutions

  1. Inspect the '{str(e)}' part of the 500 detail — it is the underlying state-layer exception.
  2. Ensure all keys are plain hashable values (strings or string-serialized tuples) before sending.
  3. If the state layer is corrupted, restart the metadata server to rebuild state.

Example fix

// before
keys = [["sess", 1], ["sess", 2]]
// after
keys = ["sess|1", "sess|2"]
Defensive patterns

Strategy: type-guard

Validate before calling

keys = [str(k) for k in keys]  # guarantee hashable string keys before POST

Type guard

def all_hashable_keys(keys) -> bool:
    return all(isinstance(k, (str, int)) for k in keys)

Try / catch

Catch the 500 and surface the embedded str(e) detail; treat as non-retryable state errors and alert — the metadata server state needs attention.

Prevention

When it happens

Trigger: Passing a list containing non-string/non-hashable elements such as nested lists or dicts as keys; concurrent mutation of the key map while it is being read; corrupted in-memory state after a partial crash.

Common situations: Client builds keys from unhashable data (e.g. lists instead of tuples/strings); mixed-version clients that send structured key objects where plain strings are expected.

Related errors

AI-assisted analysis of vllm-project/vllm@c794754062 (2026-08-14). Data as JSON: /api/errors/6143500810124f62. Report an issue: GitHub.