{"record":{"id":"cbae68b5ad0e1067","repo":"sgl-project/sglang","slug":"tag-mismatch-expected-cmd-store-complete-got-pa","errorCode":null,"errorMessage":"Tag mismatch: expected CMD_STORE_COMPLETE, got {payload.get('cmd')}","messagePattern":"Tag mismatch: expected CMD_STORE_COMPLETE, got (.+?)","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"critical","filePath":"python/sglang/srt/mem_cache/storage/flexkv/flexkv_connector.py","lineNumber":567,"sourceCode":"                except Exception as exc:  # noqa: BLE001\n                    logger.debug(\"[FlexKV] check_completed_stores: %s\", exc)\n                    completed_dict = {}\n                for fk_tid in completed_dict:\n                    rid = fk_to_rid[fk_tid]\n                    completed_rids.append(rid)\n                    self._inflight_stores.pop(rid, None)\n\n        if self._sync_ctx.is_pp_sender:\n            self._sync_ctx.scatter_pp(\n                {\n                    \"cmd\": CMD_STORE_COMPLETE,\n                    \"completed_fk_ids\": list(completed_dict),\n                }\n            )\n        elif self._sync_ctx.is_pp_receiver:\n            payload = self._sync_ctx.scatter_pp(None)\n            if payload.get(\"cmd\") != CMD_STORE_COMPLETE:\n                raise RuntimeError(\n                    f\"Tag mismatch: expected CMD_STORE_COMPLETE, got \"\n                    f\"{payload.get('cmd')}\"\n                )\n            fk_ids = payload.get(\"completed_fk_ids\", [])\n            if fk_ids and self._inflight_stores:\n                fk_to_rid = {v: k for k, v in self._inflight_stores.items()}\n                for fk_tid in fk_ids:\n                    if fk_tid in fk_to_rid:\n                        rid = fk_to_rid[fk_tid]\n                        completed_rids.append(rid)\n                        self._inflight_stores.pop(rid, None)\n\n        if self._sync_ctx.needs_sync:\n            completed_rids = self._sync_ctx.scatter(completed_rids)\n        return completed_rids\n\n    def wait_store(self, rid: str, timeout: float = 30.0) -> bool:\n        \"\"\"Block until a single store task identified by ``rid`` finishes.\"\"\"","sourceCodeStart":549,"sourceCodeEnd":585,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/mem_cache/storage/flexkv/flexkv_connector.py#L549-L585","documentation":"check_completed_stores polls completed FlexKV store tasks; on the PP receiver path it expects scatter_pp to deliver CMD_STORE_COMPLETE with completed_fk_ids. Any other tag indicates the leader is at a different point in the completion-reporting protocol.","triggerScenarios":"Calling check_completed_stores on a PP receiver when the leader sent a different command (e.g. CMD_LAYERWISE for the next load, or nothing because zero stores were in flight), desynchronizing the completion stream.","commonSituations":"Asymmetric drain: leader had no in-flight stores so it skipped the STORE_COMPLETE broadcast while receivers still called check_completed_stores; rank restart mid-store; version skew.","solutions":["Make the leader always broadcast CMD_STORE_COMPLETE (even with an empty completed list) whenever receivers call check_completed_stores, so the call counts stay symmetric","Audit _drain_completed_stores call sites to confirm they run on all ranks of the PP group","Restart the whole FlexKV group after any rank failure; do not resume a half-drained protocol"],"exampleFix":null,"handlingStrategy":"validation","validationCode":null,"typeGuard":null,"tryCatchPattern":"try:\n    connector.check_completed_stores()\nexcept RuntimeError as e:\n    if 'CMD_STORE_COMPLETE' in str(e):\n        logger.error('completion-stream desync: %s', e)\n        raise\n    raise","preventionTips":["Make the leader broadcast STORE_COMPLETE even when the completed list is empty","Monitor per-rank counts of store_kv/check_completed_stores calls and alert on drift"],"tags":["flexkv","pipeline-parallel","protocol-mismatch","distributed"],"backgroundTag":"protocol-message-mismatch","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}