{"record":{"id":"fccd54d9a304676c","repo":"sgl-project/sglang","slug":"kv-cache-dtype-mismatch-prefill-server-has-kv-cac","errorCode":null,"errorMessage":"KV cache dtype mismatch: prefill server has kv_cache_dtype={info.kv_cache_dtype}, but decode server has kv_cache_dtype={self.kv_cache_dtype_str}. Both servers must use the same --kv-cache-dtype value.","messagePattern":"KV cache dtype mismatch: prefill server has kv_cache_dtype=(.+?), but decode server has kv_cache_dtype=(.+?)\\. Both servers must use the same --kv-cache-dtype value\\.","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"critical","filePath":"python/sglang/srt/disaggregation/common/conn.py","lineNumber":636,"sourceCode":"                )\n                return False\n        except Exception as e:\n            logger.error(f\"Error fetching prefill server info from bootstrap: {e}\")\n            return False\n\n        # Sanity checks\n        if info.page_size is not None and info.page_size != self.kv_args.page_size:\n            raise RuntimeError(\n                f\"Page size mismatch: prefill server has page_size={info.page_size}, \"\n                f\"but decode server has page_size={self.kv_args.page_size}. \"\n                f\"Both servers must use the same --page-size value.\"\n            )\n\n        if (\n            info.kv_cache_dtype is not None\n            and info.kv_cache_dtype != self.kv_cache_dtype_str\n        ):\n            raise RuntimeError(\n                f\"KV cache dtype mismatch: prefill server has kv_cache_dtype={info.kv_cache_dtype}, \"\n                f\"but decode server has kv_cache_dtype={self.kv_cache_dtype_str}. \"\n                f\"Both servers must use the same --kv-cache-dtype value.\"\n            )\n\n        if self.dcp_size > 1:\n            if not (self.is_mla_backend or self.is_hybrid_mla_backend):\n                raise RuntimeError(\n                    \"PD decode DCP requires an MLA or hybrid-MLA KV pool.\"\n                )\n            if info.attn_cp_size != 1:\n                raise RuntimeError(\n                    \"PD decode DCP currently requires prefill attention CP=1, \"\n                    f\"got {info.attn_cp_size}.\"\n                )\n\n        self._resolve_rank_mapping(info)\n        self.prefill_info_table[bootstrap_addr] = info","sourceCodeStart":618,"sourceCodeEnd":654,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/disaggregation/common/conn.py#L618-L654","documentation":"During PD disaggregation startup, the decode server compares the prefill server's reported kv_cache_dtype against its own (self.kv_cache_dtype_str) and raises if they differ. KV tensors transferred between servers must have identical dtype or the raw bytes will be misinterpreted. Thrown in try_ensure_parallel_info after bootstrap info fetch.","triggerScenarios":"Prefill and decode servers launched with different --kv-cache-dtype values (e.g. 'auto' resolving to bf16 on one and fp8_e4m3 explicitly on the other), while info.kv_cache_dtype is not None.","commonSituations":"Enabling fp8 KV cache quantization on the prefill pool for memory savings but forgetting the decode pool; 'auto' resolving differently due to different GPU arch on the two pools; config drift between deployment manifests.","solutions":["Set the identical --kv-cache-dtype on both prefill and decode servers","If using 'auto' on one side, verify what it resolves to per hardware and pin the explicit value on both","Redeploy both pools after changing either side's dtype"],"exampleFix":"# before\n# prefill: --kv-cache-dtype fp8_e5m2\n# decode:  --kv-cache-dtype auto\n# after\n# both:    --kv-cache-dtype fp8_e5m2","handlingStrategy":"validation","validationCode":"assert prefill_args.kv_cache_dtype == decode_args.kv_cache_dtype or (prefill_args.kv_cache_dtype == 'auto' and decode_args.kv_cache_dtype == 'auto')","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Pin explicit --kv-cache-dtype on both pools instead of 'auto'","Diff ServerArgs of both pools in CI for PD deployments"],"tags":["disaggregation","pd-disagg","kv-cache-dtype","config-mismatch","quantization"],"backgroundTag":"server-config-mismatch","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}