{"record":{"id":"64268f2733ed942c","repo":"sgl-project/sglang","slug":"fa4-cute-fp8-backward-is-not-supported-yet-forwar","errorCode":null,"errorMessage":"FA4 CuTe FP8 backward is not supported yet (forward-only).","messagePattern":"FA4 CuTe FP8 backward is not supported yet \\(forward-only\\)\\.","errorType":"exception","errorClass":"NotImplementedError","httpStatus":null,"severity":"error","filePath":"python/sglang/kernels/ops/attention/flash_attn/cute/interface.py","lineNumber":606,"sourceCode":"        softmax_scale = (\n            1.0 / math.sqrt(head_dim)\n            if qv is None or q is None\n            else 1.0 / math.sqrt(head_dim + head_dim_v)\n        )\n    if softcap == 0.0:\n        softcap = None\n    qhead_per_kvhead = num_head // num_head_kv\n    if pack_gqa is None:\n        pack_gqa = qhead_per_kvhead > 1\n    if pack_gqa:\n        # pack_gqa reshapes SFQ's head/token layout, which the interleaved atom\n        # can't express; fall back to the dense (non-interleaved) SFQ path.\n        q_sf_interleaved = False\n\n    is_fp8 = v.dtype in (torch.float8_e4m3fn, torch.float8_e5m2)\n    requires_grad = any(t is not None and t.requires_grad for t in [q, k, v, qv])\n    if is_fp8 and requires_grad:\n        raise NotImplementedError(\n            \"FA4 CuTe FP8 backward is not supported yet (forward-only).\"\n        )\n    # qk_blockscaled (fp8 Q/K, bf16 V): output follows V's dtype. v_blockscaled\n    # (fp8 V dequanted in-kernel): output is bf16.\n    if qk_blockscaled:\n        out_torch_dtype = torch.bfloat16 if v_blockscaled else v.dtype\n    else:\n        out_torch_dtype = torch.bfloat16 if is_fp8 else q_dtype\n    device = v.device\n    q_batch_seqlen_shape = (\n        (batch_size, seqlen_q) if cu_seqlens_q is None else (total_q,)\n    )\n\n    if qv is None:\n        lse_shape = (\n            (batch_size, num_head, seqlen_q)\n            if cu_seqlens_q is None\n            else (num_head, total_q)","sourceCodeStart":588,"sourceCodeEnd":624,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/kernels/ops/attention/flash_attn/cute/interface.py#L588-L624","documentation":"The CuTe (FA4) flash attention interface supports FP8 inputs only in the forward direction; no FP8 backward kernel exists yet. Any input tensor requiring grad with FP8 Q or V triggers this NotImplementedError.","triggerScenarios":"Calling flash attention with v (or q) in torch.float8_e4m3fn/float8_e5m2 while q/k/v/qv have requires_grad=True (e.g. under autograd.backward or training loop).","commonSituations":"Fine-tuning or training with FP8 quantized activations without detaching; using FlexAttention-style APIs that assume differentiable paths in FP8 inference code.","solutions":["Detach the FP8 tensors: pass q.detach() etc. or run under torch.no_grad()","Switch Q/V to bf16 if backward is genuinely needed","Wait for upstream FP8 backward support in FA4 CuTe"],"exampleFix":"// before\nloss = fa(q_fp8.requires_grad_(True), k, v_fp8).sum(); loss.backward()\n// after\nout = fa(q_fp8.detach(), k, v_fp8.detach())  # inference only","handlingStrategy":"type-guard","validationCode":"is_fp8 = any(t is not None and t.dtype in (torch.float8_e4m3fn, torch.float8_e5m2) for t in (q, k, v))\nneeds_grad = any(t is not None and t.requires_grad for t in (q, k, v, qv))\nassert not (is_fp8 and needs_grad)","typeGuard":"def fp8_backward_safe(q, k, v) -> bool:\n    fp8 = any(t is not None and t.dtype in (torch.float8_e4m3fn, torch.float8_e5m2) for t in (q, k, v))\n    grad = any(t is not None and t.requires_grad for t in (q, k, v))\n    return not (fp8 and grad)","tryCatchPattern":"try:\n    out = fa(q, k, v)\nexcept NotImplementedError:\n    out = fa(q.to(torch.bfloat16), k.to(torch.bfloat16), v.to(torch.bfloat16))  # bf16 fallback supports backward","preventionTips":["Treat FP8 attention as inference-only in current FA4","Wrap training step so quantized tensors are detached before attention"],"tags":["flash-attention","fp8","autograd","not-implemented"],"backgroundTag":"unsupported-operation","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}