sgl-project/sglang · error · BadRequestError

Inconsistent receive_count for req_id={req_id}: registered {

Error message

Inconsistent receive_count for req_id={req_id}: registered {registered_count}, got {expected_destination_count}

What it means

register_embedding_destinations memoizes the expected receive_count per req_id on first registration and enforces that every subsequent registration for the same req_id declares the identical count. A mismatch raises BadRequestError, guarding the refcount that controls when staged embeddings may be released.

Source

Thrown at python/sglang/srt/disaggregation/encoder/server.py:724

            self.req_states.pop(req_id, None)
        await self.delivery.release(state)
        state.embedding_data = None
        if not state.preserve_metadata_on_release:
            await meta_registry.discard(req_id)

    async def register_embedding_destinations(
        self,
        req_id: str,
        expected_destination_count: int,
        destination_urls: Iterable[str],
    ) -> None:
        async with rid_lock:
            if req_id not in rid_to_receive_endpoint:
                rid_to_receive_endpoint[req_id] = set()
                rid_to_receive_count[req_id] = expected_destination_count
            registered_count = rid_to_receive_count[req_id]
            if registered_count != expected_destination_count:
                raise BadRequestError(
                    f"Inconsistent receive_count for req_id={req_id}: "
                    f"registered {registered_count}, got {expected_destination_count}"
                )
            rid_to_receive_endpoint[req_id].update(destination_urls)

        cond = await _get_receive_condition(req_id)
        async with cond:
            cond.notify_all()

    def _infer_embedding_dims(self) -> dict:
        """Infer per-modality embedding dimensions from hf_config at init time."""
        default = self.model_config.hidden_size
        hf_cfg = self.model_config.hf_config
        thinker_cfg = getattr(hf_cfg, "thinker_config", None)
        dims = {
            Modality.IMAGE: default,
            Modality.VIDEO: default,
            Modality.AUDIO: default,

View on GitHub (pinned to 0132848349)

Solutions

Make all ranks compute receive_count from the same source (decoder TP size / world size) at request time
Fix or restart the component holding a stale TP configuration so all ranks agree
Do not change receive_count between retries for the same req_id; use a fresh req_id if the topology changed

Example fix

# before
rank0.register(req_id, urls, receive_count=1)
rank1.register(req_id, urls, receive_count=tp_size)
# after
all_ranks.register(req_id, urls, receive_count=tp_size)  # consistent

Defensive patterns

Strategy: validation

Validate before calling

if req_id in rid_to_receive_count and rid_to_receive_count[req_id] != expected_count:
    raise BadRequestError('count drift detected before registration')

Try / catch

try:
    await register_embedding_destinations(req_id, urls, count)
except BadRequestError:
    regenerate_req_id_and_replay_request()  # topology changed mid-flight

Prevention

Compute receive_count once from world size and broadcast it
Restart components together after changing TP size
Never vary counts across retries of the same req_id

When it happens

Trigger: Decoder TP ranks registering destination URLs for the same req_id with different receive_count values — e.g. some ranks pass decoder TP size while others pass 1, or a changed TP configuration between calls; also triggered by a buggy client computing the count differently per call.

Common situations: Decoder tensor-parallel size changed or read inconsistently across ranks; a client bug or stale cached config causing rank-dependent counts; retrying registration after a partial failure with an updated count.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/a5259f8458270c81. Report an issue: GitHub.