sgl-project/sglang · error · ValueError

out must not require gradients

Error message

out must not require gradients

What it means

The FA4 sm120 wrapper can write into a caller-provided out tensor, but requires it to be a plain non-autograd buffer: out.requires_grad must be False. Otherwise it raises ValueError to avoid autograd silent-breakage.

Source

Thrown at python/sglang/kernels/ops/attention/flash_attention_v4_sm120.py:71

        )
    num_splits, decode_num_splits, decode_uses_static_max_seqlen_k = (
        resolve_runtime_policy(
            device_capability=device_capability,
            deterministic=deterministic,
        )
    )
    return FlashAttentionV4SM120RuntimePolicy(
        num_splits=num_splits,
        decode_num_splits=decode_num_splits,
        decode_uses_static_max_seqlen_k=decode_uses_static_max_seqlen_k,
    )


def _validate_out_contract(out: Optional[torch.Tensor]) -> None:
    if out is None:
        return
    if out.requires_grad:
        raise ValueError("out must not require gradients")
    if out.stride(-1) != 1:
        raise ValueError("out must have stride 1 in the last dimension")


@debug_kernel_api
def flash_attn_varlen_func(
    q: torch.Tensor,
    k: torch.Tensor,
    v: torch.Tensor,
    cu_seqlens_q: Optional[torch.Tensor] = None,
    cu_seqlens_k: Optional[torch.Tensor] = None,
    qv: Optional[torch.Tensor] = None,
    seqused_q: Optional[torch.Tensor] = None,
    seqused_k: Optional[torch.Tensor] = None,
    max_seqlen_q: Optional[int] = None,
    max_seqlen_k: Optional[int] = None,
    page_table: Optional[torch.Tensor] = None,
    softmax_scale: Optional[float] = None,

View on GitHub (pinned to 0132848349)

Solutions

1. Detach the buffer: out = out.detach() before passing
2. Allocate out with torch.empty(..., requires_grad=False)
3. Wrap the call in torch.no_grad() so buffers don't track grad

Example fix

# before
out = q.new_empty(...); out.requires_grad_(True)
flash_attn_varlen_func(..., out=out)
# after
out = out.detach()
flash_attn_varlen_func(..., out=out)

if out is not None and out.requires_grad:\n    out = out.detach()

def valid_out(out) -> bool:\n    return out is None or (not out.requires_grad and out.stride(-1) == 1)

When it happens

Trigger: Calling flash_attn_varlen_func/flash_attn_with_kvcache(out=t) where t.requires_grad is True (e.g. a leaf tensor with grad enabled).

Common situations: Passing a parameters-derived or grad-tracking buffer as out during training/debugging; running under autograd-enabled contexts without torch.no_grad().

Related errors

• out must not require gradients
• out must have stride 1 in the last dimension
• v_cache must be provided
• q can only be None when only_qv=True
• q must be provided unless qv is provided with only_qv=True

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/1a6dd0b0cc7b7b5d. Report an issue: GitHub.