sgl-project/sglang · critical · NotImplementedError

UlyssesAttention's all-to-all spans the combined sequence pa

Error message

UlyssesAttention's all-to-all spans the combined sequence parallel group and is not ring-aware; it would silently shuffle across ring ranks instead of rotating KV within them. Ring parallelism is not supported for models still using UlyssesAttention -- use USPAttention instead.

What it means

UlyssesAttention uses an all-to-all over the combined sequence-parallel group, which is incompatible with ring parallelism: it would shuffle tokens across ring ranks instead of rotating KV within them, silently corrupting outputs. The constructor therefore hard-fails when ring parallel world size > 1 and directs you to the ring-aware USPAttention.

Source

Thrown at python/sglang/multimodal_gen/runtime/layers/attention/layer.py:338


class UlyssesAttention(nn.Module):
    """Ulysses-style SequenceParallelism attention layer."""

    def __init__(
        self,
        num_heads: int,
        head_size: int,
        num_kv_heads: int | None = None,
        softmax_scale: float | None = None,
        causal: bool = False,
        supported_attention_backends: set[AttentionBackendEnum] | None = None,
        prefix: str = "",
        **extra_impl_args,
    ) -> None:
        super().__init__()
        if get_ring_parallel_world_size() > 1:
            raise NotImplementedError(
                "UlyssesAttention's all-to-all spans the combined sequence "
                "parallel group and is not ring-aware; it would silently "
                "shuffle across ring ranks instead of rotating KV within "
                "them. Ring parallelism is not supported for models still "
                "using UlyssesAttention -- use USPAttention instead."
            )
        if softmax_scale is None:
            self.softmax_scale = head_size**-0.5
        else:
            self.softmax_scale = softmax_scale

        if num_kv_heads is None:
            num_kv_heads = num_heads

        dtype = get_compute_dtype()
        attn_backend = get_attn_backend(
            head_size, dtype, supported_attention_backends=supported_attention_backends
        )

View on GitHub (pinned to 0132848349)

Solutions

Replace UlyssesAttention with USPAttention in the model's attention layers (USPAttention is ring-aware)
Or disable ring parallelism (run with ring parallel world size 1) if ring attention is not required
Check the server/launcher args that set the ring parallel group size and reduce it to 1

Example fix

# before
class MyAttn(nn.Module):
    self.attn = UlyssesAttention(...)
# after
class MyAttn(nn.Module):
    self.attn = USPAttention(...)

Defensive patterns

Strategy: validation

Validate before calling

from sglang... import get_ring_parallel_world_size
if get_ring_parallel_world_size() > 1:
    assert not isinstance(self.attn, UlyssesAttention), "use USPAttention under ring parallelism"

Type guard

def ring_safe_attn_cls(ring_ws: int):
    return USPAttention if ring_ws > 1 else UlyssesAttention

Prevention

Gate model attention class selection on ring world size at build time
Run a 2-rank smoke test whenever enabling ring parallelism on a new model
Search model code for UlyssesAttention before enabling ring flags

When it happens

Trigger: Instantiating UlyssesAttention while the ring parallel world size (get_ring_parallel_world_size()) is greater than 1, i.e. running with ring attention enabled in the distributed launch config.

Common situations: Enabling ring attention flags in the server args for a model whose code still uses the legacy UlyssesAttention class; upgrading a launch script to ring parallelism without updating model code; porting a model from USPAttention-based template but keeping the old class name.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/e60ff63a39daf09a. Report an issue: GitHub.