vllm-project/vllm · error · HTTPException

Confirmation failed: {str(e)}

Error message

Confirmation failed: {str(e)}

What it means

HTTP 500 returned when HF3FSMetadataServerState.confirm_write_for_keys(rank, confirmations, pages_to_release) raises an unexpected exception while applying write confirmations to the internal allocation state. The original exception is logged ('Confirm write for keys failed: ...') and chained, so the log line immediately above the HTTP error carries the root cause. It signals server-side state corruption or an invalid confirmation payload that passed format validation but broke invariants.

Source

Thrown at vllm/distributed/kv_transfer/kv_connector/v1/hf3fs/hf3fs_metadata_server.py:340

        rank = data.get("rank")
        confirmations = data.get("confirmations", [])
        pages_to_release = data.get("pages_to_release", [])

        # Validate input format
        if rank is None or not isinstance(confirmations, list):
            raise HTTPException(
                status_code=400,
                detail="Invalid request format: need 'rank' and 'confirmations'",
            )

        try:
            self.state.confirm_write_for_keys(rank, confirmations, pages_to_release)

            return Response(status_code=204)

        except Exception as e:
            logger.error("Confirm write for keys failed: %s", e)
            raise HTTPException(
                status_code=500, detail=f"Confirmation failed: {str(e)}"
            ) from e

    async def batch_key_exists(self, request: Request):
        """Check if multiple keys exist in metadata."""
        data = await self._read_json(request)
        keys = data.get("keys", [])

        if not isinstance(keys, list):
            raise HTTPException(status_code=400, detail="Invalid keys format")

        try:
            exists_results = self.state.batch_key_exists(keys)
            return self._json_response({"exists": exists_results})
        except Exception as e:
            raise HTTPException(
                status_code=500, detail=f"Key existence check failed: {str(e)}"
            ) from e

View on GitHub (pinned to c794754062)

Solutions

1. Read the server log line 'Confirm write for keys failed: <e>' to get the underlying state error and act on that message.
2. Verify every key in 'confirmations' was previously allocated via allocate_pages_for_keys for the same rank.
3. If state is stale or corrupted, restart the metadata server (and its state) together with all workers so allocation bookkeeping starts clean.
4. Guard against duplicate confirmation retries in the client after a timeout.

existing = client.batch_key_exists(keys)["exists"]
keys = [k for k, ok in zip(keys, existing) if ok]  # only confirm known keys

Catch the HTTP 500 response; parse the 'Confirmation failed: ...' detail for the root cause; make confirmation idempotent client-side and do not blind-retry non-idempotent confirmations.

When it happens

Trigger: Confirming keys that were never allocated for that rank; confirming the same keys twice so page bookkeeping underflows; passing page indices in pages_to_release that are not currently allocated; races between a clear() call and in-flight confirmations.

Common situations: Worker restarts that replay old confirmation messages; two workers confirming overlapping keys; metadata server restarted without clearing persisted state while workers resume with stale session assumptions.

Related errors

• Allocation failed: {str(e)}
• Key existence check failed: {str(e)}
• Failed to get key locations: {str(e)}
• Rank {rank} not initialized
• Invalid initialization parameters

AI-assisted analysis of vllm-project/vllm@c794754062 (2026-08-14). Data as JSON: /api/errors/eb1b0d98165328ed. Report an issue: GitHub.