sgl-project/sglang · error · TypeError

Type mismatch: {self.q_dtype} != {self.k_dtype}

Error message

Type mismatch: {self.q_dtype} != {self.k_dtype}

What it means

The SM100 FA4 kernel requires Q and K to have the same dtype. QK^T math in the MMA pipeline is specialized per dtype pair, and mixed Q/K dtypes are unsupported.

Source

Thrown at python/sglang/kernels/ops/attention/flash_attn/cute/flash_fwd_sm100.py:864

                self.v_sf_vec_size
            ).layout
            # Tile over V's token-major shape (like sfk over mK.shape): M=tokens,
            # sf_k = head_dim_v/32. sfv is now symmetric with sfk (per-token,
            # per-head-dim), which is what an incremental KV cache can produce.
            sfv_layout = cute.tile_to_shape(sfv_atom, mV_sf_shape, _SF_TILE_ORDER)
            if const_expr(self.kv_sf_interleaved):
                mSFV = cute.make_tensor(mSFV.iterator, sfv_layout)
            else:
                assert (
                    not self.use_tma_KV
                ), "can't use TMA to load SFV if not interleaved in gmem"
                mSFV = cute.make_tensor(
                    mSFV.iterator, cute.select(mSFV.layout, mode=KV_layout_transpose)
                )

        # check type consistency
        if const_expr(self.q_dtype != self.k_dtype):
            raise TypeError(f"Type mismatch: {self.q_dtype} != {self.k_dtype}")
        if const_expr(
            not self.qk_blockscaled
            and not self.v_dequant
            and self.q_dtype != self.v_dtype
        ):
            raise TypeError(f"Type mismatch: {self.q_dtype} != {self.v_dtype}")
        if const_expr(self.qk_blockscaled and self.sfq_dtype != self.sfk_dtype):
            raise TypeError(f"Type mismatch: {self.sfq_dtype} != {self.sfk_dtype}")
        if const_expr(self.q_dtype.width == 8):
            paged_kv_non_tma = not self.use_tma_KV
            if const_expr(self.head_dim_padded < 96):
                fp8_regs = _FP8_SMALL_HDIM_REGS[paged_kv_non_tma]
                self.num_regs_softmax = fp8_regs["num_regs_softmax"]
                self.num_regs_correction = fp8_regs["num_regs_correction"]
                self.num_regs_other = fp8_regs["num_regs_other"]
            else:
                fp8_tune = _FP8_TUNING_CONFIG.get(
                    (

View on GitHub (pinned to 0132848349)

Solutions

1. Quantize both Q and K to the same dtype (typically torch.float8_e4m3fn)
2. Or keep both in bf16/fp16
3. Inspect your quantization step to confirm K was actually cast

Example fix

// before
q = q.to(torch.float8_e4m3fn)  # k left bf16
// after
q = q.to(torch.float8_e4m3fn)
k = k.to(torch.float8_e4m3fn)

assert q.dtype == k.dtype, f'{q.dtype} != {k.dtype}'

def qk_dtype_consistent(q, k) -> bool: return q.dtype == k.dtype

When it happens

Trigger: Constructing/calling the SM100 flash attention kernel with q_dtype != k_dtype (e.g. fp8 Q with bf16 K).

Common situations: Partial FP8 quantization where only Q or only K was quantized; bugs in quantization pipelines leaving K in bf16.

Related errors

• Type mismatch: {self.q_dtype} != {self.v_dtype}
• Type mismatch: {self.sfq_dtype} != {self.sfk_dtype}
• Type mismatch: {self.q_dtype} != {self.k_dtype}
• O tensor must match dtype
• LSE partial tensor must be Float32

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/defb21e077a22f10. Report an issue: GitHub.