sgl-project/sglang · error · NotImplementedError

Backward pass is not implemented yet and we do not have plan

Error message

Backward pass is not implemented yet and we do not have plans to implement it because we haven't figured out how to compute dg without materializing the full hidden states for all time steps.

What it means

The autograd Function wrapping fused_recurrent (gated delta rule / KDA) explicitly does not implement backward: computing gradients w.r.t. gates would require materializing hidden states for all timesteps, which is memory-prohibitive. Calling .backward() or loss.backward() on outputs of this op raises NotImplementedError.

Source

Thrown at python/sglang/kernels/ops/attention/fla/fused_recurrent.py:746

        o, final_state = fused_recurrent_gated_delta_rule_fwd(
            q=q,
            k=k,
            v=v,
            g=g,
            beta=beta,
            scale=scale,
            initial_state=initial_state,
            output_final_state=output_final_state,
            use_qk_l2norm_in_kernel=use_qk_l2norm_in_kernel,
            cu_seqlens=cu_seqlens,
        )

        return o, final_state

    @staticmethod
    @input_guard
    def backward(ctx, do, dht):
        raise NotImplementedError(
            "Backward pass is not implemented yet and we do not have plans to implement it "
            "because we haven't figured out how to compute dg without materializing the full "
            "hidden states for all time steps."
        )


def fused_recurrent_gated_delta_rule(
    q: torch.Tensor,
    k: torch.Tensor,
    v: torch.Tensor,
    g: torch.Tensor,
    beta: torch.Tensor = None,
    scale: float = None,
    initial_state: torch.Tensor = None,
    output_final_state: bool = False,
    cu_seqlens: Optional[torch.LongTensor] = None,
    use_qk_l2norm_in_kernel: bool = False,
) -> Tuple[torch.Tensor, torch.Tensor]:

View on GitHub (pinned to 0132848349)

Solutions

Wrap inference calls in torch.no_grad() / torch.inference_mode() so backward is never invoked
For training, use a differentiable reference implementation or a chunked kernel that supports backward
Detach outputs if downstream code computes losses you don't actually need to backprop

Example fix

// before
o, s = fused_recurrent_gated_delta_rule(q, k, v, ...)
loss = o.sum(); loss.backward()  # NotImplementedError
// after
with torch.no_grad():
    o, s = fused_recurrent_gated_delta_rule(q, k, v, ...)
# inference only; use a differentiable path for training

Defensive patterns

Strategy: validation

Validate before calling

assert not torch.is_grad_enabled(), 'kernel is inference-only'

Prevention

Wrap all fused_recurrent calls in torch.inference_mode()
Detach outputs before returning from inference modules

When it happens

Trigger: Using fused_recurrent outputs in a loss and calling backward() during training; running an autograd-enabled graph that reaches this op's backward node.

Common situations: Trying to fine-tune/train a linear-attention model through the inference-optimized recurrent kernel; tests that accidentally build a grad graph; forgetting torch.no_grad() in benchmark/profiling code.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/823f568468c1aa88. Report an issue: GitHub.