What causes "out is only supported for forward-only inference" in sgl-project/sglang?

Wrap the call in with torch.no_grad(): (or torch.inference_mode()) and ensure it's on the inference path.

sgl-project/sglang · error · ValueError

out is only supported for forward-only inference

Error message

out is only supported for forward-only inference

What it means

flash_attn_varlen_func supports the `out` parameter only in forward-only inference mode. If autograd is enabled and any input (q/k/v or related differentiable tensors) requires grad, the function raises because the custom autograd path cannot honor a preallocated output buffer.

Source

Thrown at python/sglang/kernels/ops/attention/flash_attn/cute/interface.py:2299

            k,
            v,
            qv,
            learnable_sink,
            q_descale,
            k_descale,
            v_descale,
            rel_bias,
            sfq,
            sfk,
            sfv,
            *(aux_tensors or ()),
        )
        needs_autograd = torch.is_grad_enabled() and any(
            tensor is not None and tensor.requires_grad
            for tensor in differentiable_tensors
        )
    if needs_autograd and out is not None:
        raise ValueError("out is only supported for forward-only inference")
    if not needs_autograd and forward_host is not None:
        return FlashAttnVarlenFunc.forward(None, *autograd_args)
    return FlashAttnVarlenFunc.apply(*autograd_args)


def _compile_fwd_combine(
    dtype,
    dtype_partial,
    head_dim,
    tile_m,
    k_block_size,
    log_max_splits,
    has_cu_seqlens,
    has_seqused,
    has_lse,
    has_varlen_batch_idx,
    *,
    use_pdl,

View on GitHub (pinned to 0132848349)

Solutions

Wrap the call in with torch.no_grad(): (or torch.inference_mode()) and ensure it's on the inference path.
Drop the out= argument if you actually need gradients through attention.
Call .detach() on q/k/v before the call so needs_autograd is False.

Example fix

# before
out = torch.empty(...)
flash_attn_varlen_func(q, k, v, ..., out=out)  # q.requires_grad == True

# after
out = torch.empty(...)
with torch.no_grad():
    flash_attn_varlen_func(q, k, v, ..., out=out)

Defensive patterns

Strategy: validation

Validate before calling

import torch

def can_use_out(q, k, v, out):
    needs_grad = torch.is_grad_enabled() and any(
        t is not None and t.requires_grad for t in (q, k, v)
    )
    return out is None or not needs_grad

assert can_use_out(q, k, v, out)

Try / catch

try:
    flash_attn_varlen_func(q, k, v, ..., out=out)
except ValueError as e:
    if "forward-only" in str(e):
        with torch.no_grad():
            flash_attn_varlen_func(q.detach(), k.detach(), v.detach(), ..., out=out)
    else:
        raise

Prevention

Run inference under torch.inference_mode() by default in serving code.
Detach q/k/v before passing to inference-only kernels when out= is used.

When it happens

Trigger: Calling flash_attn_varlen_func with out=... while torch.is_grad_enabled() is True and at least one of q, k, v (or other differentiable inputs) has requires_grad=True.

Common situations: Using the inference path inside a training loop or a torch.compile region without torch.no_grad(); accidentally leaving requires_grad=True on frozen model weights; running under an autograd-enabled profiler/tracer.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/8b1faf1de4cd1a64. Report an issue: GitHub.