sgl-project/sglang · critical · RuntimeError

Ring Attention requires a backend whose kernel exposes the s

Error message

Ring Attention requires a backend whose kernel exposes the softmax LSE for the per-hop merge; {attn_backend.get_enum().name} does not declare support (see AttentionBackend.supports_ring_rotation).

What it means

Ring attention merges results across ring hops using the softmax LSE (log-sum-exp) statistics from each backend kernel. If the selected attention backend does not declare supports_ring_rotation(), the per-hop merge cannot be done correctly, so the layer constructor raises at init time when ring parallel world size > 1.

Source

Thrown at python/sglang/multimodal_gen/runtime/layers/attention/layer.py:774

        if softmax_scale is None:
            self.softmax_scale = head_size**-0.5
        else:
            self.softmax_scale = softmax_scale

        if num_kv_heads is None:
            num_kv_heads = num_heads

        dtype = get_compute_dtype()
        attn_backend = get_attn_backend(
            head_size,
            dtype,
            supported_attention_backends=supported_attention_backends,
            default_attention_backend=default_attention_backend,
            is_cross_attention=is_cross_attention,
        )
        if not skip_sequence_parallel and get_ring_parallel_world_size() > 1:
            if not attn_backend.supports_ring_rotation():
                raise RuntimeError(
                    f"Ring Attention requires a backend whose kernel exposes the "
                    f"softmax LSE for the per-hop merge; "
                    f"{attn_backend.get_enum().name} does not declare support "
                    f"(see AttentionBackend.supports_ring_rotation)."
                )
        impl_cls: Type[AttentionImpl] = attn_backend.get_impl_cls()
        self.allow_cudnn_sdp = bool(extra_impl_args.get("allow_cudnn_sdp", False))
        self._attn_impl_ctor_kwargs = dict(
            num_heads=num_heads,
            head_size=head_size,
            causal=causal,
            softmax_scale=self.softmax_scale,
            num_kv_heads=num_kv_heads,
            prefix=f"{prefix}.impl",
            **extra_impl_args,
        )
        self.attn_impl = impl_cls(**self._attn_impl_ctor_kwargs)
        wrap_attention_impl_forward(self.attn_impl)

View on GitHub (pinned to 0132848349)

Solutions

1. Switch the attention backend to one that declares supports_ring_rotation() (typically the FA backend)
2. Or disable ring parallelism (ring world size 1) if the backend must stay as-is
3. Check AttentionBackend.supports_ring_rotation for your chosen backend and align backend selection with ring parallel settings in server args

Example fix

# before
# ring parallelism enabled, attn_backend=SDPA
# after
server_args.attn_backend = "fa"  # FA exposes softmax LSE for ring rotation merge

from sglang... import get_ring_parallel_world_size
if get_ring_parallel_world_size() > 1:
    assert attn_backend.supports_ring_rotation(), f"{attn_backend.get_enum().name} lacks ring rotation support; use FA"

def backend_ring_ok(backend) -> bool:
    return backend.supports_ring_rotation()

try:
    layer = USPAttention(...)
except RuntimeError as e:
    if "supports_ring_rotation" in str(e):
        # fall back: disable ring or switch backend before re-init
        raise
    raise

When it happens

Trigger: Constructing this attention layer (e.g. USPAttention) with ring parallelism enabled (ring world size > 1) while the chosen/default attention backend (attn_backend) is one whose supports_ring_rotation() returns False, e.g. a backend without LSE exposure like some SDPA/Triton paths.

Common situations: Enabling ring attention with a backend override to a non-supporting backend; default backend resolution on a platform (e.g. ROCm or CPU) picking a backend without LSE; upgrading sglang where a backend's ring support flag changed; misconfigured attn_backend in server args combined with ring parallel flags.

Related errors

• UlyssesAttention's all-to-all spans the combined sequence pa
• Varlen USPAttention does not support ring parallelism yet.
• USPAttention masked path supports ring parallelism only for
• num_heads must be divisible by num_epi_subtiles
• num_heads // num_epi_subtiles must be divisible by 4 (FMA un

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/3dfcad4127187301. Report an issue: GitHub.