sgl-project/sglang · error · NotImplementedError

{type(self).__name__} does not implement packed varlen atten

Error message

{type(self).__name__} does not implement packed varlen attention

What it means

AttentionBackend.forward_varlen is an abstract hook for packed variable-length (THD) attention. Backends that only support batched inputs deliberately raise NotImplementedError so the gap surfaces early instead of silently producing wrong results.

Source

Thrown at python/sglang/multimodal_gen/runtime/layers/attention/backends/attention_backend.py:208

        self,
        query: torch.Tensor,
        key: torch.Tensor,
        value: torch.Tensor,
        attn_metadata: T,
    ) -> torch.Tensor:
        raise NotImplementedError

    def forward_varlen(
        self,
        query: torch.Tensor,
        key: torch.Tensor,
        value: torch.Tensor,
        *,
        cu_seqlens: torch.Tensor,
        max_seqlen: int,
        cu_seqlens_host: tuple[int, ...] | None = None,
    ) -> torch.Tensor:
        raise NotImplementedError(
            f"{type(self).__name__} does not implement packed varlen attention"
        )

    def forward_ring_kv_chunk(
        self,
        query: torch.Tensor,
        key: torch.Tensor,
        value: torch.Tensor,
    ) -> tuple[torch.Tensor, torch.Tensor]:
        """Attend local queries to one rotated KV chunk for ring merging.

        Inputs use packed ``[T, H, D]`` layout. The returned attention output
        has the query shape and softmax LSE uses ``[H, Tq]`` layout.
        """
        raise NotImplementedError(
            f"{type(self).__name__} does not implement ring KV-chunk attention"
        )

View on GitHub (pinned to 0132848349)

Solutions

1. Switch to a backend that implements packed varlen attention (flash_attn, ascend_fa, etc.) via server args / attention_backend_config
2. If you own the backend, implement forward_varlen handling cu_seqlens/max_seqlen packed [T, H, D] tensors
3. Route layers needing varlen to a supported backend and keep the limited backend only for forward() layers

Example fix

# before
out = backend.forward_varlen(q, k, v, cu_seqlens=cu, max_seqlen=m)  # NotImplementedError
# after
if type(backend).forward_varlen is AttentionBackend.forward_varlen:
    out = run_padded_attention(backend, q, k, v, cu)
else:
    out = backend.forward_varlen(q, k, v, cu_seqlens=cu, max_seqlen=m)

from sglang.multimodal_gen.runtime.layers.attention.backends.attention_backend import AttentionBackend

def has_varlen(backend) -> bool:
    return type(backend).forward_varlen is not AttentionBackend.forward_varlen

def implements_varlen(b: AttentionBackend) -> bool:
    return type(b).forward_varlen is not AttentionBackend.forward_varlen

try:
    out = backend.forward_varlen(q, k, v, cu_seqlens=cu, max_seqlen=m)
except NotImplementedError:
    out = run_padded_attention(backend, q, k, v, cu)

When it happens

Trigger: Calling forward_varlen(query, key, value, cu_seqlens=..., max_seqlen=...) on a backend subclass that did not override it, e.g. SlidingTileAttentionBackend or a custom backend implementing only forward().

Common situations: Switching attention_backend to one without varlen support while the workload uses packed sequences; new custom backends copied from a batched-only template; multimodal/diffusion models routed to a backend with only forward().

Related errors

• AITer backend does not have a metadata builder.
• AITER Sage backend does not have a metadata builder.
• NPU packed attention does not support a sequence that is emp
• {type(self).__name__} does not implement ring KV-chunk atten
• K/V-gather SP does not support varlen UlyssesAttention.

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/f29368ca3fc19717. Report an issue: GitHub.