sgl-project/sglang · critical · RuntimeError

Tag mismatch: expected CMD_STORE_COMPLETE, got {payload.get(

Error message

Tag mismatch: expected CMD_STORE_COMPLETE, got {payload.get('cmd')}

What it means

check_completed_stores polls completed FlexKV store tasks; on the PP receiver path it expects scatter_pp to deliver CMD_STORE_COMPLETE with completed_fk_ids. Any other tag indicates the leader is at a different point in the completion-reporting protocol.

Source

Thrown at python/sglang/srt/mem_cache/storage/flexkv/flexkv_connector.py:567

                except Exception as exc:  # noqa: BLE001
                    logger.debug("[FlexKV] check_completed_stores: %s", exc)
                    completed_dict = {}
                for fk_tid in completed_dict:
                    rid = fk_to_rid[fk_tid]
                    completed_rids.append(rid)
                    self._inflight_stores.pop(rid, None)

        if self._sync_ctx.is_pp_sender:
            self._sync_ctx.scatter_pp(
                {
                    "cmd": CMD_STORE_COMPLETE,
                    "completed_fk_ids": list(completed_dict),
                }
            )
        elif self._sync_ctx.is_pp_receiver:
            payload = self._sync_ctx.scatter_pp(None)
            if payload.get("cmd") != CMD_STORE_COMPLETE:
                raise RuntimeError(
                    f"Tag mismatch: expected CMD_STORE_COMPLETE, got "
                    f"{payload.get('cmd')}"
                )
            fk_ids = payload.get("completed_fk_ids", [])
            if fk_ids and self._inflight_stores:
                fk_to_rid = {v: k for k, v in self._inflight_stores.items()}
                for fk_tid in fk_ids:
                    if fk_tid in fk_to_rid:
                        rid = fk_to_rid[fk_tid]
                        completed_rids.append(rid)
                        self._inflight_stores.pop(rid, None)

        if self._sync_ctx.needs_sync:
            completed_rids = self._sync_ctx.scatter(completed_rids)
        return completed_rids

    def wait_store(self, rid: str, timeout: float = 30.0) -> bool:
        """Block until a single store task identified by ``rid`` finishes."""

View on GitHub (pinned to 0132848349)

Solutions

1. Make the leader always broadcast CMD_STORE_COMPLETE (even with an empty completed list) whenever receivers call check_completed_stores, so the call counts stay symmetric
2. Audit _drain_completed_stores call sites to confirm they run on all ranks of the PP group
3. Restart the whole FlexKV group after any rank failure; do not resume a half-drained protocol

try:
    connector.check_completed_stores()
except RuntimeError as e:
    if 'CMD_STORE_COMPLETE' in str(e):
        logger.error('completion-stream desync: %s', e)
        raise
    raise

When it happens

Trigger: Calling check_completed_stores on a PP receiver when the leader sent a different command (e.g. CMD_LAYERWISE for the next load, or nothing because zero stores were in flight), desynchronizing the completion stream.

Common situations: Asymmetric drain: leader had no in-flight stores so it skipped the STORE_COMPLETE broadcast while receivers still called check_completed_stores; rank restart mid-store; version skew.

Related errors

• Tag mismatch: expected CMD_LAYERWISE, got {payload.get('cmd'
• Tag mismatch: expected CMD_PUT_META, got {payload.get('cmd')
• auxiliary output does not support pipeline-parallel transpor
• world_size must be positive and divide global_heads
• Group {group_name} is destroyed.

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/cbae68b5ad0e1067. Report an issue: GitHub.