{"record":{"id":"d42fc53fbaa60061","repo":"sgl-project/sglang","slug":"only-float16-or-bfloat16-is-supported","errorCode":null,"errorMessage":"Only Float16 or BFloat16 is supported","messagePattern":"Only Float16 or BFloat16 is supported","errorType":"validation","errorClass":"TypeError","httpStatus":null,"severity":"error","filePath":"python/sglang/kernels/ops/attention/flash_attn/cute/flash_fwd.py","lineNumber":213,"sourceCode":"        mV_type: Type[cutlass.Numeric],\n        mO_type: Type[cutlass.Numeric],\n        mLSE_type: Type[cutlass.Numeric] | None,\n        mCuSeqlensQ_type: Type[cutlass.Numeric] | None,\n        mCuSeqlensK_type: Type[cutlass.Numeric] | None,\n        mSeqUsedQ_type: Type[cutlass.Numeric] | None,\n        mSeqUsedK_type: Type[cutlass.Numeric] | None,\n    ):\n        # Get the data type and check if it is fp16 or bf16\n        if const_expr(self.is_split_kv):\n            # SplitKV writes float32 partial outputs; Q/K/V still fp16/bf16.\n            if const_expr(not (mQ_type == mK_type == mV_type)):\n                raise TypeError(\"Q/K/V must have the same data type\")\n            if const_expr(mO_type != Float32):\n                raise TypeError(\"SplitKV partial output (mO) must be Float32\")\n        elif const_expr(not (mQ_type == mK_type == mV_type == mO_type)):\n            raise TypeError(\"All tensors must have the same data type\")\n        if const_expr(mQ_type not in [cutlass.Float16, cutlass.BFloat16]):\n            raise TypeError(\"Only Float16 or BFloat16 is supported\")\n        if const_expr(mLSE_type not in [None, Float32]):\n            raise TypeError(\"LSE tensor must be Float32\")\n        if const_expr(mCuSeqlensQ_type not in [None, Int32]):\n            raise TypeError(\"cu_seqlens_q tensor must be Int32\")\n        if const_expr(mCuSeqlensK_type not in [None, Int32]):\n            raise TypeError(\"cu_seqlens_k tensor must be Int32\")\n        if const_expr(mSeqUsedQ_type not in [None, Int32]):\n            raise TypeError(\"seqused_q tensor must be Int32\")\n        if const_expr(mSeqUsedK_type not in [None, Int32]):\n            raise TypeError(\"seqused_k tensor must be Int32\")\n        assert mQ_type == self.dtype\n\n    def _setup_attributes(self):\n        # ///////////////////////////////////////////////////////////////////////////////\n        # Shared memory layout: Q/K/V\n        # ///////////////////////////////////////////////////////////////////////////////\n        (\n            sQ_layout_atom,","sourceCodeStart":195,"sourceCodeEnd":231,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/kernels/ops/attention/flash_attn/cute/flash_fwd.py#L195-L231","documentation":"The CUTLASS DSL flash-attention forward kernel is only instantiated for Float16 and BFloat16 element types; the Q tensor's dtype failed this check. The kernel templates/gemm configurations do not exist for fp32, fp8, or integer types.","triggerScenarios":"Passing Q (and by extension K/V/O, which must already match) as torch.float32, torch.float8_*, or any non-fp16/bf16 dtype to FlashAttentionForward.","commonSituations":"Feeding un-cast model weights or activations in fp32 (e.g. a model loaded without half precision); testing with toy float32 tensors; accidentally using fp8 tensors intended for a different attention backend.","solutions":["Cast inputs: Q = Q.to(torch.bfloat16) (or torch.float16) before calling the op","Ensure the model runs in half precision (--dtype bfloat16/half or equivalent server config)","For fp8 attention use the dedicated fp8 attention backend, not this kernel"],"exampleFix":"// before\nout = fa_fwd(Q, K, V)  # Q,K,V are float32\n// after\nout = fa_fwd(Q.to(torch.bfloat16), K.to(torch.bfloat16), V.to(torch.bfloat16))","handlingStrategy":"type-guard","validationCode":"assert Q.dtype in (torch.float16, torch.bfloat16), f'flash fwd supports fp16/bf16 only, got {Q.dtype}'","typeGuard":"def is_half_dtype(t) -> bool:\n    return t.dtype in (torch.float16, torch.bfloat16)","tryCatchPattern":null,"preventionTips":["Cast at the attention-wrapper boundary: Q=Q.to(torch.bfloat16), etc.","Validate once in the backend-selection code which dtypes each backend supports"],"tags":["cuda","dtype","flash-attention","cutlass","unsupported-dtype"],"backgroundTag":null,"analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}