sgl-project/sglang · error · NotImplementedError

Varlen USPAttention does not support ring parallelism yet.

Error message

Varlen USPAttention does not support ring parallelism yet.

What it means

The varlen path of USPAttention performs an all-to-all over the combined sequence-parallel group; under ring parallelism (ring world size > 1) this would shuffle rows across ring ranks instead of rotating KV, silently corrupting output, so it raises NotImplementedError instead.

Source

Thrown at python/sglang/multimodal_gen/runtime/layers/attention/layer.py:878

        ctx_attn_metadata = forward_context.attn_metadata
        effective_skip_sp = (
            self.skip_sequence_parallel or skip_sequence_parallel_override
        )
        if seq_lens is not None:
            assert (
                attn_mask is None
                and attn_mask_meta is None
                and not num_replicated_prefix
                and not num_replicated_suffix
                and not num_replicated_kv_prefix
            ), "Varlen USPAttention does not support masks or replicated tokens"
            if effective_skip_sp or get_sequence_parallel_world_size() == 1:
                return self.attn_impl.forward(q, k, v, ctx_attn_metadata)
            if get_ring_parallel_world_size() > 1:
                # The varlen all-to-all spans the combined SP group and is not
                # ring-aware; it would shuffle rows across ring ranks instead
                # of rotating KV, corrupting the output silently.
                raise NotImplementedError(
                    "Varlen USPAttention does not support ring parallelism yet."
                )
            qkv = torch.cat([q, k, v], dim=0)
            qkv = _usp_input_all_to_all_varlen(qkv, seq_lens, head_dim=2)
            qkv = self.attn_impl.preprocess_qkv(qkv, ctx_attn_metadata)
            q, k, v = qkv.chunk(3, dim=0)
            out = self.attn_impl.forward(q, k, v, ctx_attn_metadata)
            out = self.attn_impl.postprocess_output(out, ctx_attn_metadata)
            return _usp_output_all_to_all_varlen(out, seq_lens, head_dim=2)

        if isinstance(attn_mask_meta, DynamicVarlenMaskMeta):
            attn_mask_meta = attn_mask_meta.resolve(attn_mask)

        # Tail-pad meta alone (sp_shard.tail_attn_meta; mask derivable from the
        # pad span) also opts into the masked SP branch. gap_* = legacy alias.
        meta_pad_start = meta_pad_end = None
        if attn_mask_meta is not None:
            meta_pad_start = attn_mask_meta.get(

View on GitHub (pinned to 0132848349)

Solutions

1. Disable ring parallelism (run with ring world size 1) for varlen workloads
2. Or pad/bucket requests to uniform lengths so the non-varlen ring path is used
3. Track the upstream task implementing varlen ring support and upgrade once available

Example fix

# before
# launch with ring parallelism, batch has variable lengths
out = attn(q, k, v, seq_lens=lens)
# after
# disable ring parallelism in server args, keep SP + varlen
out = attn(q, k, v, seq_lens=lens)

if seq_lens is not None and get_ring_parallel_world_size() > 1 and get_sequence_parallel_world_size() > 1:
    raise ValueError("varlen + ring parallelism unsupported; disable ring or pad batch")

def varlen_ring_ok(seq_lens, ring_ws: int) -> bool:
    return seq_lens is None or ring_ws == 1

When it happens

Trigger: Calling USPAttention.forward with seq_lens not None (ragged/varlen batch), sequence parallel world size > 1, effective_skip_sp False, and ring parallel world size > 1.

Common situations: Serving variable-length multimodal prompts on a ring-attention-enabled deployment; enabling ring parallelism for throughput on a workload that produces varlen batches; a config change that turns on ring attention for a model whose batches are ragged.

Related errors

• UlyssesAttention's all-to-all spans the combined sequence pa
• K/V-gather SP does not support varlen UlyssesAttention.
• USPAttention's masked path does not support replicated prefi
• USPAttention masked path supports ring parallelism only for
• The batch size is expected to be 1 rather than {q.shape[0]}

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/354d2c9af887182f. Report an issue: GitHub.