{"record":{"id":"defb21e077a22f10","repo":"sgl-project/sglang","slug":"type-mismatch-self-q-dtype-self-k-dtype","errorCode":null,"errorMessage":"Type mismatch: {self.q_dtype} != {self.k_dtype}","messagePattern":"Type mismatch: (.+?) != (.+?)","errorType":"validation","errorClass":"TypeError","httpStatus":null,"severity":"error","filePath":"python/sglang/kernels/ops/attention/flash_attn/cute/flash_fwd_sm100.py","lineNumber":864,"sourceCode":"                self.v_sf_vec_size\n            ).layout\n            # Tile over V's token-major shape (like sfk over mK.shape): M=tokens,\n            # sf_k = head_dim_v/32. sfv is now symmetric with sfk (per-token,\n            # per-head-dim), which is what an incremental KV cache can produce.\n            sfv_layout = cute.tile_to_shape(sfv_atom, mV_sf_shape, _SF_TILE_ORDER)\n            if const_expr(self.kv_sf_interleaved):\n                mSFV = cute.make_tensor(mSFV.iterator, sfv_layout)\n            else:\n                assert (\n                    not self.use_tma_KV\n                ), \"can't use TMA to load SFV if not interleaved in gmem\"\n                mSFV = cute.make_tensor(\n                    mSFV.iterator, cute.select(mSFV.layout, mode=KV_layout_transpose)\n                )\n\n        # check type consistency\n        if const_expr(self.q_dtype != self.k_dtype):\n            raise TypeError(f\"Type mismatch: {self.q_dtype} != {self.k_dtype}\")\n        if const_expr(\n            not self.qk_blockscaled\n            and not self.v_dequant\n            and self.q_dtype != self.v_dtype\n        ):\n            raise TypeError(f\"Type mismatch: {self.q_dtype} != {self.v_dtype}\")\n        if const_expr(self.qk_blockscaled and self.sfq_dtype != self.sfk_dtype):\n            raise TypeError(f\"Type mismatch: {self.sfq_dtype} != {self.sfk_dtype}\")\n        if const_expr(self.q_dtype.width == 8):\n            paged_kv_non_tma = not self.use_tma_KV\n            if const_expr(self.head_dim_padded < 96):\n                fp8_regs = _FP8_SMALL_HDIM_REGS[paged_kv_non_tma]\n                self.num_regs_softmax = fp8_regs[\"num_regs_softmax\"]\n                self.num_regs_correction = fp8_regs[\"num_regs_correction\"]\n                self.num_regs_other = fp8_regs[\"num_regs_other\"]\n            else:\n                fp8_tune = _FP8_TUNING_CONFIG.get(\n                    (","sourceCodeStart":846,"sourceCodeEnd":882,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/kernels/ops/attention/flash_attn/cute/flash_fwd_sm100.py#L846-L882","documentation":"The SM100 FA4 kernel requires Q and K to have the same dtype. QK^T math in the MMA pipeline is specialized per dtype pair, and mixed Q/K dtypes are unsupported.","triggerScenarios":"Constructing/calling the SM100 flash attention kernel with q_dtype != k_dtype (e.g. fp8 Q with bf16 K).","commonSituations":"Partial FP8 quantization where only Q or only K was quantized; bugs in quantization pipelines leaving K in bf16.","solutions":["Quantize both Q and K to the same dtype (typically torch.float8_e4m3fn)","Or keep both in bf16/fp16","Inspect your quantization step to confirm K was actually cast"],"exampleFix":"// before\nq = q.to(torch.float8_e4m3fn)  # k left bf16\n// after\nq = q.to(torch.float8_e4m3fn)\nk = k.to(torch.float8_e4m3fn)","handlingStrategy":"validation","validationCode":"assert q.dtype == k.dtype, f'{q.dtype} != {k.dtype}'","typeGuard":"def qk_dtype_consistent(q, k) -> bool: return q.dtype == k.dtype","tryCatchPattern":null,"preventionTips":["Quantize Q and K together in one function so they can't diverge","Add a dtype assertion in your attention wrapper before dispatching to FA4"],"tags":["flash-attention","dtype-mismatch","sm100","fp8"],"backgroundTag":"dtype-mismatch","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}