{"record":{"id":"8b1faf1de4cd1a64","repo":"sgl-project/sglang","slug":"out-is-only-supported-for-forward-only-inference","errorCode":null,"errorMessage":"out is only supported for forward-only inference","messagePattern":"out is only supported for forward-only inference","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/kernels/ops/attention/flash_attn/cute/interface.py","lineNumber":2299,"sourceCode":"            k,\n            v,\n            qv,\n            learnable_sink,\n            q_descale,\n            k_descale,\n            v_descale,\n            rel_bias,\n            sfq,\n            sfk,\n            sfv,\n            *(aux_tensors or ()),\n        )\n        needs_autograd = torch.is_grad_enabled() and any(\n            tensor is not None and tensor.requires_grad\n            for tensor in differentiable_tensors\n        )\n    if needs_autograd and out is not None:\n        raise ValueError(\"out is only supported for forward-only inference\")\n    if not needs_autograd and forward_host is not None:\n        return FlashAttnVarlenFunc.forward(None, *autograd_args)\n    return FlashAttnVarlenFunc.apply(*autograd_args)\n\n\ndef _compile_fwd_combine(\n    dtype,\n    dtype_partial,\n    head_dim,\n    tile_m,\n    k_block_size,\n    log_max_splits,\n    has_cu_seqlens,\n    has_seqused,\n    has_lse,\n    has_varlen_batch_idx,\n    *,\n    use_pdl,","sourceCodeStart":2281,"sourceCodeEnd":2317,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/kernels/ops/attention/flash_attn/cute/interface.py#L2281-L2317","documentation":"flash_attn_varlen_func supports the `out` parameter only in forward-only inference mode. If autograd is enabled and any input (q/k/v or related differentiable tensors) requires grad, the function raises because the custom autograd path cannot honor a preallocated output buffer.","triggerScenarios":"Calling flash_attn_varlen_func with out=... while torch.is_grad_enabled() is True and at least one of q, k, v (or other differentiable inputs) has requires_grad=True.","commonSituations":"Using the inference path inside a training loop or a torch.compile region without torch.no_grad(); accidentally leaving requires_grad=True on frozen model weights; running under an autograd-enabled profiler/tracer.","solutions":["Wrap the call in with torch.no_grad(): (or torch.inference_mode()) and ensure it's on the inference path.","Drop the out= argument if you actually need gradients through attention.","Call .detach() on q/k/v before the call so needs_autograd is False."],"exampleFix":"# before\nout = torch.empty(...)\nflash_attn_varlen_func(q, k, v, ..., out=out)  # q.requires_grad == True\n\n# after\nout = torch.empty(...)\nwith torch.no_grad():\n    flash_attn_varlen_func(q, k, v, ..., out=out)","handlingStrategy":"validation","validationCode":"import torch\n\ndef can_use_out(q, k, v, out):\n    needs_grad = torch.is_grad_enabled() and any(\n        t is not None and t.requires_grad for t in (q, k, v)\n    )\n    return out is None or not needs_grad\n\nassert can_use_out(q, k, v, out)","typeGuard":null,"tryCatchPattern":"try:\n    flash_attn_varlen_func(q, k, v, ..., out=out)\nexcept ValueError as e:\n    if \"forward-only\" in str(e):\n        with torch.no_grad():\n            flash_attn_varlen_func(q.detach(), k.detach(), v.detach(), ..., out=out)\n    else:\n        raise","preventionTips":["Run inference under torch.inference_mode() by default in serving code.","Detach q/k/v before passing to inference-only kernels when out= is used."],"tags":["autograd","attention","inference","flash-attention"],"backgroundTag":"out-parameter-requires-grad-conflict","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}