sgl-project/sglang · error · NotImplementedError

K/V-gather SP does not support varlen UlyssesAttention.

Error message

K/V-gather SP does not support varlen UlyssesAttention.

What it means

The K/V-gather sequence-parallel path of UlyssesAttention only supports uniform (non-varlen) batches. If per-row sequence lengths (seq_lens) are passed to _forward_with_kv_gather, it raises rather than producing wrong results across the gather.

Source

Thrown at python/sglang/multimodal_gen/runtime/layers/attention/layer.py:397

        self.sp_attention_mode, self.sp_attention_mode_is_auto = (
            _resolve_sp_attention_mode(
                causal=causal, sparse_backend=self.backend.is_sparse
            )
        )

    def _forward_with_kv_gather(
        self,
        q: torch.Tensor,
        k: torch.Tensor,
        v: torch.Tensor,
        ctx_attn_metadata,
        replicated_q: torch.Tensor | None,
        replicated_k: torch.Tensor | None,
        replicated_v: torch.Tensor | None,
        seq_lens: list[int] | None,
    ) -> tuple[torch.Tensor, torch.Tensor | None]:
        if seq_lens is not None:
            raise NotImplementedError(
                "K/V-gather SP does not support varlen UlyssesAttention."
            )
        if any(x is not None for x in (replicated_q, replicated_k, replicated_v)):
            if any(x is None for x in (replicated_q, replicated_k, replicated_v)):
                raise ValueError("Replicated Q, K, and V must be provided together.")

        k = sequence_model_parallel_all_gather(k, dim=1)
        v = sequence_model_parallel_all_gather(v, dim=1)

        local_query_len = q.shape[1]
        if replicated_q is not None:
            q = torch.cat([q, replicated_q], dim=1)
            k = torch.cat([k, replicated_k], dim=1)
            v = torch.cat([v, replicated_v], dim=1)

        output = self.attn_impl.forward(q, k, v, ctx_attn_metadata)
        if replicated_q is None:
            return output, None

View on GitHub (pinned to 0132848349)

Solutions

1. Switch sp_attention_mode away from kv_gather to a mode that supports varlen (e.g. the default all-to-all/USP path)
2. Pad/bucket the batch to uniform sequence lengths so seq_lens is None
3. Upgrade the model code to USPAttention which handles varlen under SP

Example fix

# before
out = attn(q, k, v, seq_lens=[5, 17, 9])  # attn uses kv_gather SP
# after
out = attn(q, k, v, seq_lens=None)  # uniform batch, or use USPAttention for varlen

if seq_lens is not None and attn.sp_attention_mode == "kv_gather":
    raise ValueError("kv_gather SP requires uniform lengths; pad the batch or switch mode")

def kv_gather_ok(attn, seq_lens) -> bool:
    return seq_lens is None or getattr(attn, "sp_attention_mode", None) != "kv_gather"

try:
    out = attn(q, k, v, seq_lens=seq_lens)
except NotImplementedError as e:
    if "kv_gather" in str(e):
        seq_lens = None  # after padding to uniform length
        out = attn(q, k, v, seq_lens=None)
    else:
        raise

When it happens

Trigger: Calling UlyssesAttention.forward with seq_lens not None while the layer is configured with sp_attention_mode == 'kv_gather', on a batch with variable sequence lengths (e.g. multimodal batches with mixed image/text sizes).

Common situations: Serving variable-length multimodal prompts under K/V-gather SP mode; switching a workload from fixed-length synthetic batches to real ragged batches; enabling kv_gather attention mode in server args and sending a batch that triggers the varlen path.

Related errors

• Varlen USPAttention does not support ring parallelism yet.
• UlyssesAttention's all-to-all spans the combined sequence pa
• K/V-gather SP does not support video sparse attention.
• USPAttention's masked path does not support replicated prefi
• The batch size is expected to be 1 rather than {q.shape[0]}

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/a12b9b507d575f1f. Report an issue: GitHub.