{"record":{"id":"f0e711a56df09dcc","repo":"sgl-project/sglang","slug":"all-tensors-must-have-the-same-data-type","errorCode":null,"errorMessage":"All tensors must have the same data type","messagePattern":"All tensors must have the same data type","errorType":"validation","errorClass":"TypeError","httpStatus":null,"severity":"error","filePath":"python/sglang/kernels/ops/attention/flash_attn/cute/flash_fwd.py","lineNumber":211,"sourceCode":"        mQ_type: Type[cutlass.Numeric],\n        mK_type: Type[cutlass.Numeric],\n        mV_type: Type[cutlass.Numeric],\n        mO_type: Type[cutlass.Numeric],\n        mLSE_type: Type[cutlass.Numeric] | None,\n        mCuSeqlensQ_type: Type[cutlass.Numeric] | None,\n        mCuSeqlensK_type: Type[cutlass.Numeric] | None,\n        mSeqUsedQ_type: Type[cutlass.Numeric] | None,\n        mSeqUsedK_type: Type[cutlass.Numeric] | None,\n    ):\n        # Get the data type and check if it is fp16 or bf16\n        if const_expr(self.is_split_kv):\n            # SplitKV writes float32 partial outputs; Q/K/V still fp16/bf16.\n            if const_expr(not (mQ_type == mK_type == mV_type)):\n                raise TypeError(\"Q/K/V must have the same data type\")\n            if const_expr(mO_type != Float32):\n                raise TypeError(\"SplitKV partial output (mO) must be Float32\")\n        elif const_expr(not (mQ_type == mK_type == mV_type == mO_type)):\n            raise TypeError(\"All tensors must have the same data type\")\n        if const_expr(mQ_type not in [cutlass.Float16, cutlass.BFloat16]):\n            raise TypeError(\"Only Float16 or BFloat16 is supported\")\n        if const_expr(mLSE_type not in [None, Float32]):\n            raise TypeError(\"LSE tensor must be Float32\")\n        if const_expr(mCuSeqlensQ_type not in [None, Int32]):\n            raise TypeError(\"cu_seqlens_q tensor must be Int32\")\n        if const_expr(mCuSeqlensK_type not in [None, Int32]):\n            raise TypeError(\"cu_seqlens_k tensor must be Int32\")\n        if const_expr(mSeqUsedQ_type not in [None, Int32]):\n            raise TypeError(\"seqused_q tensor must be Int32\")\n        if const_expr(mSeqUsedK_type not in [None, Int32]):\n            raise TypeError(\"seqused_k tensor must be Int32\")\n        assert mQ_type == self.dtype\n\n    def _setup_attributes(self):\n        # ///////////////////////////////////////////////////////////////////////////////\n        # Shared memory layout: Q/K/V\n        # ///////////////////////////////////////////////////////////////////////////////","sourceCodeStart":193,"sourceCodeEnd":229,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/kernels/ops/attention/flash_attn/cute/flash_fwd.py#L193-L229","documentation":"Raised by the flash-attention forward op type checker in the non-SplitKV path: Q, K, V, and O must all share the same data type for the kernel to compile (mixed-dtype tensors are unsupported). This is a static const_expr check performed on tensor element types during __call__.","triggerScenarios":"Calling FlashAttentionForward without split-KV where any of Q/K/V/O differs in dtype, e.g. Q,K,V in bfloat16 but O preallocated as float16, or one input cast differently.","commonSituations":"Preallocating the output tensor with a default dtype (torch.float32) while inputs are fp16; mixing autocast-produced tensors with manually cast ones; partial refactors that cast only some inputs.","solutions":["Unify all four tensors: cast O to match Q/K/V (o = torch.empty_like(q) or dtype=q.dtype)","Verify no tensor was created under torch.get_default_dtype() == float32","If you intentionally want fp32 partials, use the SplitKV path where O is fp32 by design"],"exampleFix":"// before\nO = torch.empty((b,h,s,d), dtype=torch.float32, device='cuda')\n// after\nO = torch.empty((b,h,s,d), dtype=Q.dtype, device='cuda')","handlingStrategy":"type-guard","validationCode":"assert Q.dtype == K.dtype == V.dtype == O.dtype, 'Q/K/V/O dtypes must match for non-split flash attention'","typeGuard":"def dtypes_match(*ts) -> bool:\n    return len({t.dtype for t in ts}) == 1","tryCatchPattern":null,"preventionTips":["Allocate O with torch.empty_like(Q) or dtype=Q.dtype","Never rely on default dtype for output buffers"],"tags":["cuda","dtype","flash-attention","cutlass","type-mismatch"],"backgroundTag":null,"analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}