sgl-project/sglang · error · ValueError

out must not require gradients

Error message

out must not require gradients

What it means

When the caller supplies a preallocated out tensor, the FA kernel writes in-place and cannot propagate gradients through it — autograd would silently be wrong. Hence out.requires_grad must be False.

Source

Thrown at python/sglang/kernels/ops/attention/flash_attn/cute/interface.py:644

    else:
        # num_head contiguous better for MQA in MLA absorbed
        lse_shape = (
            (batch_size, seqlen_q, num_head)
            if cu_seqlens_q is None
            else (total_q, num_head)
        )

    if out is None:
        out = torch.empty(
            *q_batch_seqlen_shape,
            num_head,
            head_dim_v,
            dtype=out_torch_dtype,
            device=device,
        )
    else:
        if out.requires_grad:
            raise ValueError("out must not require gradients")
        if out.stride(-1) != 1:
            raise ValueError("out must have stride 1 in the last dimension")
        _validate_tensor(
            out,
            "out",
            (*q_batch_seqlen_shape, num_head, head_dim_v),
            out_torch_dtype,
            device,
        )

    if lse is None:
        lse = (
            torch.empty(lse_shape, dtype=torch.float32, device=device)
            if requires_grad or return_lse
            else None
        )
    elif lse is not None:
        _validate_tensor(lse, "lse", lse_shape, torch.float32, device)

View on GitHub (pinned to 0132848349)

Solutions

1. Allocate out with requires_grad=False (default)
2. Call out.detach() (or out.requires_grad_(False)) before passing
3. Omit out and let the kernel allocate, so autograd flows normally

Example fix

// before
out = torch.empty(shape, requires_grad=True)
fa(..., out=out)
// after
out = torch.empty(shape)
fa(..., out=out)

if out is not None and out.requires_grad:
    out = out.detach()

def out_tensor_safe(out) -> bool: return out is None or not out.requires_grad

When it happens

Trigger: Passing out= tensor that was created with requires_grad=True (directly or via a differentiable factory like torch.zeros(..., requires_grad=True)).

Common situations: Training loops where all buffers are allocated with requires_grad; reusing a leaf parameter as the output buffer.

Related errors

• out must not require gradients
• v_cache must be provided
• q can only be None when only_qv=True
• q must be provided unless qv is provided with only_qv=True
• out must have stride 1 in the last dimension

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/d441c32a6beff2fd. Report an issue: GitHub.