sgl-project/sglang · critical · RuntimeError

KV cache dtype mismatch: prefill server has kv_cache_dtype={

Error message

KV cache dtype mismatch: prefill server has kv_cache_dtype={info.kv_cache_dtype}, but decode server has kv_cache_dtype={self.kv_cache_dtype_str}. Both servers must use the same --kv-cache-dtype value.

What it means

During PD disaggregation startup, the decode server compares the prefill server's reported kv_cache_dtype against its own (self.kv_cache_dtype_str) and raises if they differ. KV tensors transferred between servers must have identical dtype or the raw bytes will be misinterpreted. Thrown in try_ensure_parallel_info after bootstrap info fetch.

Source

Thrown at python/sglang/srt/disaggregation/common/conn.py:636

                )
                return False
        except Exception as e:
            logger.error(f"Error fetching prefill server info from bootstrap: {e}")
            return False

        # Sanity checks
        if info.page_size is not None and info.page_size != self.kv_args.page_size:
            raise RuntimeError(
                f"Page size mismatch: prefill server has page_size={info.page_size}, "
                f"but decode server has page_size={self.kv_args.page_size}. "
                f"Both servers must use the same --page-size value."
            )

        if (
            info.kv_cache_dtype is not None
            and info.kv_cache_dtype != self.kv_cache_dtype_str
        ):
            raise RuntimeError(
                f"KV cache dtype mismatch: prefill server has kv_cache_dtype={info.kv_cache_dtype}, "
                f"but decode server has kv_cache_dtype={self.kv_cache_dtype_str}. "
                f"Both servers must use the same --kv-cache-dtype value."
            )

        if self.dcp_size > 1:
            if not (self.is_mla_backend or self.is_hybrid_mla_backend):
                raise RuntimeError(
                    "PD decode DCP requires an MLA or hybrid-MLA KV pool."
                )
            if info.attn_cp_size != 1:
                raise RuntimeError(
                    "PD decode DCP currently requires prefill attention CP=1, "
                    f"got {info.attn_cp_size}."
                )

        self._resolve_rank_mapping(info)
        self.prefill_info_table[bootstrap_addr] = info

View on GitHub (pinned to 0132848349)

Solutions

Set the identical --kv-cache-dtype on both prefill and decode servers
If using 'auto' on one side, verify what it resolves to per hardware and pin the explicit value on both
Redeploy both pools after changing either side's dtype

Example fix

# before
# prefill: --kv-cache-dtype fp8_e5m2
# decode:  --kv-cache-dtype auto
# after
# both:    --kv-cache-dtype fp8_e5m2

Defensive patterns

Strategy: validation

Validate before calling

assert prefill_args.kv_cache_dtype == decode_args.kv_cache_dtype or (prefill_args.kv_cache_dtype == 'auto' and decode_args.kv_cache_dtype == 'auto')

Prevention

Pin explicit --kv-cache-dtype on both pools instead of 'auto'
Diff ServerArgs of both pools in CI for PD deployments

When it happens

Trigger: Prefill and decode servers launched with different --kv-cache-dtype values (e.g. 'auto' resolving to bf16 on one and fp8_e4m3 explicitly on the other), while info.kv_cache_dtype is not None.

Common situations: Enabling fp8 KV cache quantization on the prefill pool for memory savings but forgetting the decode pool; 'auto' resolving differently due to different GPU arch on the two pools; config drift between deployment manifests.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/fccd54d9a304676c. Report an issue: GitHub.