vllm-project/vllm · error · HTTPException

Worker with dp_rank={payload.dp_rank}, tp_rank={payload.tp_r

Error message

Worker with dp_rank={payload.dp_rank}, tp_rank={payload.tp_rank}, pp_rank={payload.pp_rank} is already registered at {tp_entry[payload.pp_rank]}, but still want to register at {payload.addr}

What it means

Raised as HTTP 400 by the Mooncake KV-transfer metadata server when a worker tries to register a (dp_rank, tp_rank, pp_rank) triple that is already registered at a different address. The registration table (dp_entry.worker_addr[tp_rank][pp_rank]) is keyed by rank coordinates, so a second registration for the same coordinates is treated as a topology conflict, not an update.

Source

Thrown at vllm/distributed/kv_transfer/kv_connector/v1/mooncake/mooncake_utils.py:102

                engine_id=payload.engine_id,
                worker_addr={},
            )

        dp_entry = self.workers[payload.dp_rank]
        if dp_entry.engine_id != payload.engine_id:
            raise HTTPException(
                status_code=400,
                detail=(
                    f"Engine ID mismatch for dp_rank={payload.dp_rank}: "
                    f"expected {dp_entry.engine_id}, got {payload.engine_id}"
                ),
            )
        if payload.tp_rank not in dp_entry.worker_addr:
            dp_entry.worker_addr[payload.tp_rank] = {}

        tp_entry = dp_entry.worker_addr[payload.tp_rank]
        if payload.pp_rank in tp_entry:
            raise HTTPException(
                status_code=400,
                detail=(
                    f"Worker with dp_rank={payload.dp_rank}, "
                    f"tp_rank={payload.tp_rank}, pp_rank={payload.pp_rank} "
                    f"is already registered at "
                    f"{tp_entry[payload.pp_rank]}, "
                    f"but still want to register at {payload.addr}"
                ),
            )

        tp_entry[payload.pp_rank] = payload.addr
        logger.debug(
            "Registered worker: engine_id=%s, dp_rank=%d, tp_rank=%d, pp_rank=%d at %s",
            payload.engine_id,
            payload.dp_rank,
            payload.tp_rank,
            payload.pp_rank,
            payload.addr,

View on GitHub (pinned to c794754062)

Solutions

1. Verify the old worker for those ranks actually exited; if it did not, kill the stale vLLM process holding the registration
2. Restart or clear the Mooncake metadata server so stale entries are dropped, then re-register
3. Check that each engine uses distinct dp_rank/tp_rank/pp_rank coordinates (correct --data-parallel-size, --pipeline-parallel-size, KV role assignment) so no two workers collide
4. If the old address is actually dead, ensure the engine that owns it deregisters on shutdown instead of reusing its rank coordinates in a new process

Example fix

# before
# two engines started with identical rank coords against one metadata server
# engine A: dp=0,tp=0,pp=0 addr=10.0.0.1:8000
# engine B: dp=0,tp=0,pp=0 addr=10.0.0.2:8000  # -> HTTP 400

# after
# give the second engine its own dp coordinate
# engine B: dp=1,tp=0,pp=0 addr=10.0.0.2:8000

# before registering, probe the metadata server
import requests
r = requests.get(f"{metadata_server_url}/v1/{engine_id}/workers")
existing = r.json().get(str(dp_rank), {}).get(str(tp_rank), {}).get(str(pp_rank))
if existing and existing != my_addr:
    # stale registration: deregister or abort before the 400
    requests.delete(f"{metadata_server_url}/v1/register", json={
        "engine_id": engine_id, "dp_rank": dp_rank, "tp_rank": tp_rank,
        "pp_rank": pp_rank, "addr": existing})

# around requests.post(register...)
try:
    resp = requests.post(url, json=payload, timeout=10)
    resp.raise_for_status()
except requests.HTTPError as e:
    if e.response is not None and e.response.status_code == 400 and "already registered" in e.response.text:
        # stale entry: clear it or pick new rank coords, then retry once
        ...
    raise

When it happens

Trigger: Calling the metadata server's registration endpoint twice for the same engine_id/dp/tp/pp ranks with different worker addresses — e.g. a worker restarted with a new address while its old entry was never removed, or a second vLLM engine pointed at the same metadata server with overlapping rank coordinates.

Common situations: Rolling restart of a P2P/Mooncake prefill instance where the old worker did not deregister; misconfigured KV_ROLE/--kv-parallel-config so two engines claim identical dp/tp/pp ranks; stale metadata server state across test runs.

Related errors

• No KV cache tensors were registered with Mooncake.
• Engine ID mismatch for dp_rank={payload.dp_rank}: expected {
• Invalid initialization parameters
• Invalid request format: need 'rank' and 'keys'
• Invalid request format: need 'rank' and 'confirmations'

AI-assisted analysis of vllm-project/vllm@c794754062 (2026-08-14). Data as JSON: /api/errors/78677618b2abd670. Report an issue: GitHub.