{"record":{"id":"eb1b0d98165328ed","repo":"vllm-project/vllm","slug":"confirmation-failed-str-e","errorCode":null,"errorMessage":"Confirmation failed: {str(e)}","messagePattern":"Confirmation failed: (.+?)","errorType":"http","errorClass":"HTTPException","httpStatus":500,"severity":"error","filePath":"vllm/distributed/kv_transfer/kv_connector/v1/hf3fs/hf3fs_metadata_server.py","lineNumber":340,"sourceCode":"        rank = data.get(\"rank\")\n        confirmations = data.get(\"confirmations\", [])\n        pages_to_release = data.get(\"pages_to_release\", [])\n\n        # Validate input format\n        if rank is None or not isinstance(confirmations, list):\n            raise HTTPException(\n                status_code=400,\n                detail=\"Invalid request format: need 'rank' and 'confirmations'\",\n            )\n\n        try:\n            self.state.confirm_write_for_keys(rank, confirmations, pages_to_release)\n\n            return Response(status_code=204)\n\n        except Exception as e:\n            logger.error(\"Confirm write for keys failed: %s\", e)\n            raise HTTPException(\n                status_code=500, detail=f\"Confirmation failed: {str(e)}\"\n            ) from e\n\n    async def batch_key_exists(self, request: Request):\n        \"\"\"Check if multiple keys exist in metadata.\"\"\"\n        data = await self._read_json(request)\n        keys = data.get(\"keys\", [])\n\n        if not isinstance(keys, list):\n            raise HTTPException(status_code=400, detail=\"Invalid keys format\")\n\n        try:\n            exists_results = self.state.batch_key_exists(keys)\n            return self._json_response({\"exists\": exists_results})\n        except Exception as e:\n            raise HTTPException(\n                status_code=500, detail=f\"Key existence check failed: {str(e)}\"\n            ) from e","sourceCodeStart":322,"sourceCodeEnd":358,"githubUrl":"https://github.com/vllm-project/vllm/blob/c794754062d49a8fdb63ab3c5215b488b865030c/vllm/distributed/kv_transfer/kv_connector/v1/hf3fs/hf3fs_metadata_server.py#L322-L358","documentation":"HTTP 500 returned when HF3FSMetadataServerState.confirm_write_for_keys(rank, confirmations, pages_to_release) raises an unexpected exception while applying write confirmations to the internal allocation state. The original exception is logged ('Confirm write for keys failed: ...') and chained, so the log line immediately above the HTTP error carries the root cause. It signals server-side state corruption or an invalid confirmation payload that passed format validation but broke invariants.","triggerScenarios":"Confirming keys that were never allocated for that rank; confirming the same keys twice so page bookkeeping underflows; passing page indices in pages_to_release that are not currently allocated; races between a clear() call and in-flight confirmations.","commonSituations":"Worker restarts that replay old confirmation messages; two workers confirming overlapping keys; metadata server restarted without clearing persisted state while workers resume with stale session assumptions.","solutions":["Read the server log line 'Confirm write for keys failed: <e>' to get the underlying state error and act on that message.","Verify every key in 'confirmations' was previously allocated via allocate_pages_for_keys for the same rank.","If state is stale or corrupted, restart the metadata server (and its state) together with all workers so allocation bookkeeping starts clean.","Guard against duplicate confirmation retries in the client after a timeout."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"existing = client.batch_key_exists(keys)[\"exists\"]\nkeys = [k for k, ok in zip(keys, existing) if ok]  # only confirm known keys","typeGuard":null,"tryCatchPattern":"Catch the HTTP 500 response; parse the 'Confirmation failed: ...' detail for the root cause; make confirmation idempotent client-side and do not blind-retry non-idempotent confirmations.","preventionTips":["Confirm each allocation exactly once, tracking confirmed keys per rank","Restart metadata server and workers together so state stays consistent"],"tags":["hf3fs","metadata-server","http-500","state-corruption","kv-transfer"],"backgroundTag":null,"analyzedSha":"c794754062d49a8fdb63ab3c5215b488b865030c","analyzedAt":"2026-08-14T21:17:39.825Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}