sgl-project/sglang · critical · RuntimeError

Tag mismatch: expected CMD_LAYERWISE, got {payload.get('cmd'

Error message

Tag mismatch: expected CMD_LAYERWISE, got {payload.get('cmd')}

What it means

During layerwise KV loading, the PP receiver rank calls scatter_pp and expects a payload tagged CMD_LAYERWISE carrying the producer's counter id. Receiving any other cmd (or None) means the producer side sent a different synchronization message, indicating the ranks' layerwise protocols have diverged.

Source

Thrown at python/sglang/srt/mem_cache/storage/flexkv/flexkv_connector.py:399

        assert self.enable_layerwise and self.layer_done_counter is not None, (
            "start_load_kv_layerwise called but layerwise transfer is "
            "disabled. Set FLEXKV_ENABLE_LAYERWISE_TRANSFER=1."
        )
        fkv_task_id = self._pending_lookups.pop(rid, -1)
        if fkv_task_id < 0:
            return 0, -1

        slot_mapping_cpu = self._to_cpu_int64(slot_mapping)
        n = slot_mapping_cpu.numel()

        if self._sync_ctx.should_send_slot_mapping_to_remote:
            self._send_slot_mapping_to_remote(fkv_task_id, slot_mapping_cpu)

        # Allocate / receive producer slot.
        if self._sync_ctx.is_pp_receiver:
            payload = self._sync_ctx.scatter_pp(None)
            if payload.get("cmd") != CMD_LAYERWISE:
                raise RuntimeError(
                    f"Tag mismatch: expected CMD_LAYERWISE, got "
                    f"{payload.get('cmd')}"
                )
            producer_id = int(payload["counter_id"])
            self.layer_done_counter.register_task_with_explicit_counter_id(
                fkv_task_id, producer_id
            )
        else:
            producer_id = self.layer_done_counter.update_producer()
            self.layer_done_counter.events[producer_id].reset_for_new_transfer()
            self.layer_done_counter.register_task(fkv_task_id, producer_id)

        if self._sync_ctx.is_pp_sender:
            self._sync_ctx.scatter_pp(
                {
                    "cmd": CMD_LAYERWISE,
                    "fkv_task_id": fkv_task_id,
                    "counter_id": producer_id,

View on GitHub (pinned to 0132848349)

Solutions

1. Verify all nodes run the same SGLang commit so CMD constants match
2. Check scheduler logs on the producer rank for the phase it actually emitted; realign the call order of start_load_kv_layerwise/store_kv/check_completed_stores
3. If a rank crashed mid-protocol, restart the whole FlexKV group rather than resuming one rank

try:
    self.start_load_kv_layerwise(...)
except RuntimeError as e:
    if 'Tag mismatch' in str(e):
        logger.error('FlexKV protocol desync: %s; restarting group', e)
        raise FatalDistributedError(str(e)) from e
    raise

When it happens

Trigger: Calling start_load_kv_layerwise on a PP receiver when the producer rank sent a different command (e.g. a store-complete or shutdown message), or when producer/receiver run different SGLang versions with mismatched CMD tags; also when scatter_pp returns a payload from an unrelated phase of the protocol.

Common situations: Version skew between ranks in multi-node FlexKV offloading; a crash/retry on the producer causing it to skip or repeat a phase; misconfigured pipeline-parallel rank roles.

Related errors

• Tag mismatch: expected CMD_PUT_META, got {payload.get('cmd')
• Tag mismatch: expected CMD_STORE_COMPLETE, got {payload.get(
• auxiliary output does not support pipeline-parallel transpor
• world_size must be positive and divide global_heads
• Group {group_name} is destroyed.

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/48c13c9921b03fdf. Report an issue: GitHub.