{"record":{"id":"d75a2d49707048d9","repo":"sgl-project/sglang","slug":"tag-mismatch-expected-cmd-put-meta-got-payload","errorCode":null,"errorMessage":"Tag mismatch: expected CMD_PUT_META, got {payload.get('cmd')}","messagePattern":"Tag mismatch: expected CMD_PUT_META, got (.+?)","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"critical","filePath":"python/sglang/srt/mem_cache/storage/flexkv/flexkv_connector.py","lineNumber":520,"sourceCode":"            if int(unmatched_mask.sum()) > 0:\n                filtered = kv_indices[unmatched_mask]\n                slot_mapping_cpu = self._to_cpu_int64(filtered)\n                self.kv_manager.launch(\n                    task_ids=[fkv_task_id],\n                    slot_mappings=[slot_mapping_cpu],\n                    as_batch=False,\n                    layerwise_transfer=False,\n                )\n                self._inflight_stores[rid] = fkv_task_id\n                return fkv_task_id\n            return -1\n\n        # Non-leader path: receive the unmatched mask + maybe forward\n        # slot_mapping to the remote-side TransferManager.\n        if self._sync_ctx.is_pp_receiver:\n            payload = self._sync_ctx.scatter_pp(None)\n            if payload.get(\"cmd\") != CMD_PUT_META:\n                raise RuntimeError(\n                    f\"Tag mismatch: expected CMD_PUT_META, got \" f\"{payload.get('cmd')}\"\n                )\n            fkv_task_id = int(payload[\"fkv_task_id\"])\n            mask_list = payload.get(\"unmatched_mask\", [])\n            unmatched_mask = torch.tensor(mask_list, dtype=torch.bool)\n            if (\n                int(unmatched_mask.sum()) > 0\n                and fkv_task_id >= 0\n                and self._sync_ctx.should_send_slot_mapping_to_remote\n            ):\n                filtered = kv_indices[unmatched_mask]\n                slot_mapping_cpu = self._to_cpu_int64(filtered)\n                self._send_slot_mapping_to_remote(fkv_task_id, slot_mapping_cpu)\n                self._inflight_stores[rid] = fkv_task_id\n        return fkv_task_id\n\n    def check_completed_stores(self) -> List[str]:\n        \"\"\"Return rids whose stores have completed since the last call.\"\"\"","sourceCodeStart":502,"sourceCodeEnd":538,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/mem_cache/storage/flexkv/flexkv_connector.py#L502-L538","documentation":"On the non-leader (PP receiver) path of store_kv, scatter_pp must deliver the leader's CMD_PUT_META payload (fkv_task_id + unmatched_mask). A different cmd tag means the leader rank executed a different protocol step, so the receiver aborts rather than store under wrong metadata.","triggerScenarios":"store_kv invoked on a PP receiver when the leader sent a non-CMD_PUT_META message — e.g. the leader skipped a store (nothing to write path), crashed mid-store, or the ranks' scatter/gather call sequences drifted after an exception.","commonSituations":"Multi-node FlexKV with uneven request distribution causing leaders and receivers to call store_kv a different number of times; version mismatch changing the store path; partial failure of a prior store shifting the message stream.","solutions":["Ensure every rank calls store_kv/check_completed_stores the same number of times per scheduled batch (the protocol assumes symmetric invocation)","Check the leader rank's log for the cmd it actually sent at that step and for prior errors that skipped a store","Pin identical SGLang versions across all FlexKV ranks"],"exampleFix":null,"handlingStrategy":"validation","validationCode":null,"typeGuard":null,"tryCatchPattern":"try:\n    connector.store_kv(req, token_ids, kv_indices)\nexcept RuntimeError as e:\n    if 'CMD_PUT_META' in str(e):\n        logger.error('FlexKV store protocol desync on rid=%s: %s', req.rid, e)\n        raise  # group-level failure; do not swallow\n    raise","preventionTips":["Guarantee symmetric store_kv invocation counts across PP ranks per batch","Fail fast (crash the group) on tag mismatches instead of retrying individual ranks"],"tags":["flexkv","pipeline-parallel","protocol-mismatch","distributed"],"backgroundTag":"protocol-message-mismatch","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}