sgl-project/sglang · error · TypeError

Type mismatch: {self.q_dtype} != {self.v_dtype}

Error message

Type mismatch: {self.q_dtype} != {self.v_dtype}

What it means

Unless the kernel is block-scaled (qk_blockscaled) or V is dequantized in-kernel (v_dequant), V must share Q's dtype. This check is skipped when V is fp8 with separate scale factors (v_blockscaled path), where bf16 output is expected.

Source

Thrown at python/sglang/kernels/ops/attention/flash_attn/cute/flash_fwd_sm100.py:870

            if const_expr(self.kv_sf_interleaved):
                mSFV = cute.make_tensor(mSFV.iterator, sfv_layout)
            else:
                assert (
                    not self.use_tma_KV
                ), "can't use TMA to load SFV if not interleaved in gmem"
                mSFV = cute.make_tensor(
                    mSFV.iterator, cute.select(mSFV.layout, mode=KV_layout_transpose)
                )

        # check type consistency
        if const_expr(self.q_dtype != self.k_dtype):
            raise TypeError(f"Type mismatch: {self.q_dtype} != {self.k_dtype}")
        if const_expr(
            not self.qk_blockscaled
            and not self.v_dequant
            and self.q_dtype != self.v_dtype
        ):
            raise TypeError(f"Type mismatch: {self.q_dtype} != {self.v_dtype}")
        if const_expr(self.qk_blockscaled and self.sfq_dtype != self.sfk_dtype):
            raise TypeError(f"Type mismatch: {self.sfq_dtype} != {self.sfk_dtype}")
        if const_expr(self.q_dtype.width == 8):
            paged_kv_non_tma = not self.use_tma_KV
            if const_expr(self.head_dim_padded < 96):
                fp8_regs = _FP8_SMALL_HDIM_REGS[paged_kv_non_tma]
                self.num_regs_softmax = fp8_regs["num_regs_softmax"]
                self.num_regs_correction = fp8_regs["num_regs_correction"]
                self.num_regs_other = fp8_regs["num_regs_other"]
            else:
                fp8_tune = _FP8_TUNING_CONFIG.get(
                    (
                        self.use_2cta_instrs,
                        self.is_causal,
                        self.head_dim_padded,
                        self.is_sm103,
                    ),
                    {},

View on GitHub (pinned to 0132848349)

Solutions

1. If quantizing V to fp8, enable the v_blockscaled/v_dequant path with sfv scale factors
2. Otherwise cast V to match Q's dtype (bf16)

Example fix

// before
attn(q_bf16, k_bf16, v_fp8)  # no sfv
// after
v = v.to(torch.bfloat16)
attn(q_bf16, k_bf16, v)

uses_sf = v_blockscaled or v_dequant
assert uses_sf or q.dtype == v.dtype

def v_dtype_ok(q, v, v_blockscaled=False, v_dequant=False) -> bool:
    return v_blockscaled or v_dequant or q.dtype == v.dtype

When it happens

Trigger: Calling the SM100 kernel with q_dtype != v_dtype while neither qk_blockscaled nor v_dequant is enabled — e.g. bf16 Q/K with fp8 V and no SFV/scale configuration.

Common situations: Trying to save memory by quantizing only V to fp8 without wiring up the block-scaled (scale factor) path.

Related errors

• Type mismatch: {self.q_dtype} != {self.k_dtype}
• Type mismatch: {self.sfq_dtype} != {self.sfk_dtype}
• Type mismatch: {self.q_dtype} != {self.k_dtype}
• O tensor must match dtype
• LSE partial tensor must be Float32

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/acc018ef5e4f360c. Report an issue: GitHub.