sgl-project/sglang · error · TypeError

Type mismatch: {self.sfq_dtype} != {self.sfk_dtype}

Error message

Type mismatch: {self.sfq_dtype} != {self.sfk_dtype}

What it means

In block-scaled FP8 mode, the Q scale-factor tensor (SFQ) and K scale-factor tensor (SFK) must share dtype (e.g. both float8_e8m0fnu for MXFP8). Mismatched scale formats cannot be combined in the MMA.

Source

Thrown at python/sglang/kernels/ops/attention/flash_attn/cute/flash_fwd_sm100.py:872

            else:
                assert (
                    not self.use_tma_KV
                ), "can't use TMA to load SFV if not interleaved in gmem"
                mSFV = cute.make_tensor(
                    mSFV.iterator, cute.select(mSFV.layout, mode=KV_layout_transpose)
                )

        # check type consistency
        if const_expr(self.q_dtype != self.k_dtype):
            raise TypeError(f"Type mismatch: {self.q_dtype} != {self.k_dtype}")
        if const_expr(
            not self.qk_blockscaled
            and not self.v_dequant
            and self.q_dtype != self.v_dtype
        ):
            raise TypeError(f"Type mismatch: {self.q_dtype} != {self.v_dtype}")
        if const_expr(self.qk_blockscaled and self.sfq_dtype != self.sfk_dtype):
            raise TypeError(f"Type mismatch: {self.sfq_dtype} != {self.sfk_dtype}")
        if const_expr(self.q_dtype.width == 8):
            paged_kv_non_tma = not self.use_tma_KV
            if const_expr(self.head_dim_padded < 96):
                fp8_regs = _FP8_SMALL_HDIM_REGS[paged_kv_non_tma]
                self.num_regs_softmax = fp8_regs["num_regs_softmax"]
                self.num_regs_correction = fp8_regs["num_regs_correction"]
                self.num_regs_other = fp8_regs["num_regs_other"]
            else:
                fp8_tune = _FP8_TUNING_CONFIG.get(
                    (
                        self.use_2cta_instrs,
                        self.is_causal,
                        self.head_dim_padded,
                        self.is_sm103,
                    ),
                    {},
                )
                if const_expr("ex2_emu_freq" in fp8_tune):

View on GitHub (pinned to 0132848349)

Solutions

1. Re-quantize so SFQ and SFK use the same scale dtype (typically torch.float8_e8m0fnu for MXFP8)
2. Check the quantization recipe produces symmetric scale formats for Q and K

Example fix

// before
sfq_f32, sfk_e8m0 = quant(q), quant(k)
// after
sfq_e8m0, sfk_e8m0 = quant_mxfp8(q), quant_mxfp8(k)

if qk_blockscaled:
    assert sfq.dtype == sfk.dtype, f'{sfq.dtype} != {sfk.dtype}'

def scale_factors_consistent(sfq, sfk) -> bool: return sfq.dtype == sfk.dtype

When it happens

Trigger: Invoking the kernel with qk_blockscaled=True and sfq.dtype != sfk.dtype, e.g. one e8m0 and the other float32.

Common situations: Custom quantizers emitting float32 scales for one side and e8m0 for the other; mixing MXFP8 and per-tensor scaled quantization between Q and K.

Related errors

• Type mismatch: {self.q_dtype} != {self.k_dtype}
• Type mismatch: {self.q_dtype} != {self.v_dtype}
• int32-packed scale buffers require scale_ue8m0=True
• scale_ue8m0=True requires an int32-packed output_s
• O tensor must match dtype

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/0fe17dc7b6e2e44e. Report an issue: GitHub.