{"record":{"id":"48c13c9921b03fdf","repo":"sgl-project/sglang","slug":"tag-mismatch-expected-cmd-layerwise-got-payload","errorCode":null,"errorMessage":"Tag mismatch: expected CMD_LAYERWISE, got {payload.get('cmd')}","messagePattern":"Tag mismatch: expected CMD_LAYERWISE, got (.+?)","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"critical","filePath":"python/sglang/srt/mem_cache/storage/flexkv/flexkv_connector.py","lineNumber":399,"sourceCode":"        assert self.enable_layerwise and self.layer_done_counter is not None, (\n            \"start_load_kv_layerwise called but layerwise transfer is \"\n            \"disabled. Set FLEXKV_ENABLE_LAYERWISE_TRANSFER=1.\"\n        )\n        fkv_task_id = self._pending_lookups.pop(rid, -1)\n        if fkv_task_id < 0:\n            return 0, -1\n\n        slot_mapping_cpu = self._to_cpu_int64(slot_mapping)\n        n = slot_mapping_cpu.numel()\n\n        if self._sync_ctx.should_send_slot_mapping_to_remote:\n            self._send_slot_mapping_to_remote(fkv_task_id, slot_mapping_cpu)\n\n        # Allocate / receive producer slot.\n        if self._sync_ctx.is_pp_receiver:\n            payload = self._sync_ctx.scatter_pp(None)\n            if payload.get(\"cmd\") != CMD_LAYERWISE:\n                raise RuntimeError(\n                    f\"Tag mismatch: expected CMD_LAYERWISE, got \"\n                    f\"{payload.get('cmd')}\"\n                )\n            producer_id = int(payload[\"counter_id\"])\n            self.layer_done_counter.register_task_with_explicit_counter_id(\n                fkv_task_id, producer_id\n            )\n        else:\n            producer_id = self.layer_done_counter.update_producer()\n            self.layer_done_counter.events[producer_id].reset_for_new_transfer()\n            self.layer_done_counter.register_task(fkv_task_id, producer_id)\n\n        if self._sync_ctx.is_pp_sender:\n            self._sync_ctx.scatter_pp(\n                {\n                    \"cmd\": CMD_LAYERWISE,\n                    \"fkv_task_id\": fkv_task_id,\n                    \"counter_id\": producer_id,","sourceCodeStart":381,"sourceCodeEnd":417,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/mem_cache/storage/flexkv/flexkv_connector.py#L381-L417","documentation":"During layerwise KV loading, the PP receiver rank calls scatter_pp and expects a payload tagged CMD_LAYERWISE carrying the producer's counter id. Receiving any other cmd (or None) means the producer side sent a different synchronization message, indicating the ranks' layerwise protocols have diverged.","triggerScenarios":"Calling start_load_kv_layerwise on a PP receiver when the producer rank sent a different command (e.g. a store-complete or shutdown message), or when producer/receiver run different SGLang versions with mismatched CMD tags; also when scatter_pp returns a payload from an unrelated phase of the protocol.","commonSituations":"Version skew between ranks in multi-node FlexKV offloading; a crash/retry on the producer causing it to skip or repeat a phase; misconfigured pipeline-parallel rank roles.","solutions":["Verify all nodes run the same SGLang commit so CMD constants match","Check scheduler logs on the producer rank for the phase it actually emitted; realign the call order of start_load_kv_layerwise/store_kv/check_completed_stores","If a rank crashed mid-protocol, restart the whole FlexKV group rather than resuming one rank"],"exampleFix":null,"handlingStrategy":"validation","validationCode":null,"typeGuard":null,"tryCatchPattern":"try:\n    self.start_load_kv_layerwise(...)\nexcept RuntimeError as e:\n    if 'Tag mismatch' in str(e):\n        logger.error('FlexKV protocol desync: %s; restarting group', e)\n        raise FatalDistributedError(str(e)) from e\n    raise","preventionTips":["Version-lock all ranks to the same SGLang commit in the FlexKV group","Abort and restart the whole group on the first tag mismatch — the message stream is unrecoverable","Add a startup handshake that exchanges protocol/commit ids before layerwise transfers"],"tags":["flexkv","pipeline-parallel","protocol-mismatch","distributed"],"backgroundTag":"protocol-message-mismatch","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}