{"record":{"id":"78677618b2abd670","repo":"vllm-project/vllm","slug":"worker-with-dp-rank-payload-dp-rank-tp-rank-pa","errorCode":null,"errorMessage":"Worker with dp_rank={payload.dp_rank}, tp_rank={payload.tp_rank}, pp_rank={payload.pp_rank} is already registered at {tp_entry[payload.pp_rank]}, but still want to register at {payload.addr}","messagePattern":"Worker with dp_rank=(.+?), tp_rank=(.+?), pp_rank=(.+?) is already registered at (.+?), but still want to register at (.+?)","errorType":"http","errorClass":"HTTPException","httpStatus":400,"severity":"error","filePath":"vllm/distributed/kv_transfer/kv_connector/v1/mooncake/mooncake_utils.py","lineNumber":102,"sourceCode":"                engine_id=payload.engine_id,\n                worker_addr={},\n            )\n\n        dp_entry = self.workers[payload.dp_rank]\n        if dp_entry.engine_id != payload.engine_id:\n            raise HTTPException(\n                status_code=400,\n                detail=(\n                    f\"Engine ID mismatch for dp_rank={payload.dp_rank}: \"\n                    f\"expected {dp_entry.engine_id}, got {payload.engine_id}\"\n                ),\n            )\n        if payload.tp_rank not in dp_entry.worker_addr:\n            dp_entry.worker_addr[payload.tp_rank] = {}\n\n        tp_entry = dp_entry.worker_addr[payload.tp_rank]\n        if payload.pp_rank in tp_entry:\n            raise HTTPException(\n                status_code=400,\n                detail=(\n                    f\"Worker with dp_rank={payload.dp_rank}, \"\n                    f\"tp_rank={payload.tp_rank}, pp_rank={payload.pp_rank} \"\n                    f\"is already registered at \"\n                    f\"{tp_entry[payload.pp_rank]}, \"\n                    f\"but still want to register at {payload.addr}\"\n                ),\n            )\n\n        tp_entry[payload.pp_rank] = payload.addr\n        logger.debug(\n            \"Registered worker: engine_id=%s, dp_rank=%d, tp_rank=%d, pp_rank=%d at %s\",\n            payload.engine_id,\n            payload.dp_rank,\n            payload.tp_rank,\n            payload.pp_rank,\n            payload.addr,","sourceCodeStart":84,"sourceCodeEnd":120,"githubUrl":"https://github.com/vllm-project/vllm/blob/c794754062d49a8fdb63ab3c5215b488b865030c/vllm/distributed/kv_transfer/kv_connector/v1/mooncake/mooncake_utils.py#L84-L120","documentation":"Raised as HTTP 400 by the Mooncake KV-transfer metadata server when a worker tries to register a (dp_rank, tp_rank, pp_rank) triple that is already registered at a different address. The registration table (dp_entry.worker_addr[tp_rank][pp_rank]) is keyed by rank coordinates, so a second registration for the same coordinates is treated as a topology conflict, not an update.","triggerScenarios":"Calling the metadata server's registration endpoint twice for the same engine_id/dp/tp/pp ranks with different worker addresses — e.g. a worker restarted with a new address while its old entry was never removed, or a second vLLM engine pointed at the same metadata server with overlapping rank coordinates.","commonSituations":"Rolling restart of a P2P/Mooncake prefill instance where the old worker did not deregister; misconfigured KV_ROLE/--kv-parallel-config so two engines claim identical dp/tp/pp ranks; stale metadata server state across test runs.","solutions":["Verify the old worker for those ranks actually exited; if it did not, kill the stale vLLM process holding the registration","Restart or clear the Mooncake metadata server so stale entries are dropped, then re-register","Check that each engine uses distinct dp_rank/tp_rank/pp_rank coordinates (correct --data-parallel-size, --pipeline-parallel-size, KV role assignment) so no two workers collide","If the old address is actually dead, ensure the engine that owns it deregisters on shutdown instead of reusing its rank coordinates in a new process"],"exampleFix":"# before\n# two engines started with identical rank coords against one metadata server\n# engine A: dp=0,tp=0,pp=0 addr=10.0.0.1:8000\n# engine B: dp=0,tp=0,pp=0 addr=10.0.0.2:8000  # -> HTTP 400\n\n# after\n# give the second engine its own dp coordinate\n# engine B: dp=1,tp=0,pp=0 addr=10.0.0.2:8000","handlingStrategy":"validation","validationCode":"# before registering, probe the metadata server\nimport requests\nr = requests.get(f\"{metadata_server_url}/v1/{engine_id}/workers\")\nexisting = r.json().get(str(dp_rank), {}).get(str(tp_rank), {}).get(str(pp_rank))\nif existing and existing != my_addr:\n    # stale registration: deregister or abort before the 400\n    requests.delete(f\"{metadata_server_url}/v1/register\", json={\n        \"engine_id\": engine_id, \"dp_rank\": dp_rank, \"tp_rank\": tp_rank,\n        \"pp_rank\": pp_rank, \"addr\": existing})","typeGuard":null,"tryCatchPattern":"# around requests.post(register...)\ntry:\n    resp = requests.post(url, json=payload, timeout=10)\n    resp.raise_for_status()\nexcept requests.HTTPError as e:\n    if e.response is not None and e.response.status_code == 400 and \"already registered\" in e.response.text:\n        # stale entry: clear it or pick new rank coords, then retry once\n        ...\n    raise","preventionTips":["Ensure clean shutdown paths deregister workers from the Mooncake metadata server","Assign unique dp/tp/pp coordinates per engine in multi-instance deployments","Clear the metadata server state between full cluster restarts"],"tags":["distributed","kv-transfer","mooncake","registration","http-400"],"backgroundTag":null,"analyzedSha":"c794754062d49a8fdb63ab3c5215b488b865030c","analyzedAt":"2026-08-14T21:17:39.825Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}