sgl-project/sglang · critical · RuntimeError

Tag mismatch: expected CMD_PUT_META, got {payload.get('cmd')

Error message

Tag mismatch: expected CMD_PUT_META, got {payload.get('cmd')}

What it means

On the non-leader (PP receiver) path of store_kv, scatter_pp must deliver the leader's CMD_PUT_META payload (fkv_task_id + unmatched_mask). A different cmd tag means the leader rank executed a different protocol step, so the receiver aborts rather than store under wrong metadata.

Source

Thrown at python/sglang/srt/mem_cache/storage/flexkv/flexkv_connector.py:520

            if int(unmatched_mask.sum()) > 0:
                filtered = kv_indices[unmatched_mask]
                slot_mapping_cpu = self._to_cpu_int64(filtered)
                self.kv_manager.launch(
                    task_ids=[fkv_task_id],
                    slot_mappings=[slot_mapping_cpu],
                    as_batch=False,
                    layerwise_transfer=False,
                )
                self._inflight_stores[rid] = fkv_task_id
                return fkv_task_id
            return -1

        # Non-leader path: receive the unmatched mask + maybe forward
        # slot_mapping to the remote-side TransferManager.
        if self._sync_ctx.is_pp_receiver:
            payload = self._sync_ctx.scatter_pp(None)
            if payload.get("cmd") != CMD_PUT_META:
                raise RuntimeError(
                    f"Tag mismatch: expected CMD_PUT_META, got " f"{payload.get('cmd')}"
                )
            fkv_task_id = int(payload["fkv_task_id"])
            mask_list = payload.get("unmatched_mask", [])
            unmatched_mask = torch.tensor(mask_list, dtype=torch.bool)
            if (
                int(unmatched_mask.sum()) > 0
                and fkv_task_id >= 0
                and self._sync_ctx.should_send_slot_mapping_to_remote
            ):
                filtered = kv_indices[unmatched_mask]
                slot_mapping_cpu = self._to_cpu_int64(filtered)
                self._send_slot_mapping_to_remote(fkv_task_id, slot_mapping_cpu)
                self._inflight_stores[rid] = fkv_task_id
        return fkv_task_id

    def check_completed_stores(self) -> List[str]:
        """Return rids whose stores have completed since the last call."""

View on GitHub (pinned to 0132848349)

Solutions

1. Ensure every rank calls store_kv/check_completed_stores the same number of times per scheduled batch (the protocol assumes symmetric invocation)
2. Check the leader rank's log for the cmd it actually sent at that step and for prior errors that skipped a store
3. Pin identical SGLang versions across all FlexKV ranks

try:
    connector.store_kv(req, token_ids, kv_indices)
except RuntimeError as e:
    if 'CMD_PUT_META' in str(e):
        logger.error('FlexKV store protocol desync on rid=%s: %s', req.rid, e)
        raise  # group-level failure; do not swallow
    raise

When it happens

Trigger: store_kv invoked on a PP receiver when the leader sent a non-CMD_PUT_META message — e.g. the leader skipped a store (nothing to write path), crashed mid-store, or the ranks' scatter/gather call sequences drifted after an exception.

Common situations: Multi-node FlexKV with uneven request distribution causing leaders and receivers to call store_kv a different number of times; version mismatch changing the store path; partial failure of a prior store shifting the message stream.

Related errors

• Tag mismatch: expected CMD_LAYERWISE, got {payload.get('cmd'
• Tag mismatch: expected CMD_STORE_COMPLETE, got {payload.get(
• auxiliary output does not support pipeline-parallel transpor
• world_size must be positive and divide global_heads
• Group {group_name} is destroyed.

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/d75a2d49707048d9. Report an issue: GitHub.