{"record":{"id":"3dfcad4127187301","repo":"sgl-project/sglang","slug":"ring-attention-requires-a-backend-whose-kernel-exp","errorCode":null,"errorMessage":"Ring Attention requires a backend whose kernel exposes the softmax LSE for the per-hop merge; {attn_backend.get_enum().name} does not declare support (see AttentionBackend.supports_ring_rotation).","messagePattern":"Ring Attention requires a backend whose kernel exposes the softmax LSE for the per-hop merge; (.+?) does not declare support \\(see AttentionBackend\\.supports_ring_rotation\\)\\.","errorType":"error_code","errorClass":"RuntimeError","httpStatus":null,"severity":"critical","filePath":"python/sglang/multimodal_gen/runtime/layers/attention/layer.py","lineNumber":774,"sourceCode":"        if softmax_scale is None:\n            self.softmax_scale = head_size**-0.5\n        else:\n            self.softmax_scale = softmax_scale\n\n        if num_kv_heads is None:\n            num_kv_heads = num_heads\n\n        dtype = get_compute_dtype()\n        attn_backend = get_attn_backend(\n            head_size,\n            dtype,\n            supported_attention_backends=supported_attention_backends,\n            default_attention_backend=default_attention_backend,\n            is_cross_attention=is_cross_attention,\n        )\n        if not skip_sequence_parallel and get_ring_parallel_world_size() > 1:\n            if not attn_backend.supports_ring_rotation():\n                raise RuntimeError(\n                    f\"Ring Attention requires a backend whose kernel exposes the \"\n                    f\"softmax LSE for the per-hop merge; \"\n                    f\"{attn_backend.get_enum().name} does not declare support \"\n                    f\"(see AttentionBackend.supports_ring_rotation).\"\n                )\n        impl_cls: Type[AttentionImpl] = attn_backend.get_impl_cls()\n        self.allow_cudnn_sdp = bool(extra_impl_args.get(\"allow_cudnn_sdp\", False))\n        self._attn_impl_ctor_kwargs = dict(\n            num_heads=num_heads,\n            head_size=head_size,\n            causal=causal,\n            softmax_scale=self.softmax_scale,\n            num_kv_heads=num_kv_heads,\n            prefix=f\"{prefix}.impl\",\n            **extra_impl_args,\n        )\n        self.attn_impl = impl_cls(**self._attn_impl_ctor_kwargs)\n        wrap_attention_impl_forward(self.attn_impl)","sourceCodeStart":756,"sourceCodeEnd":792,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/multimodal_gen/runtime/layers/attention/layer.py#L756-L792","documentation":"Ring attention merges results across ring hops using the softmax LSE (log-sum-exp) statistics from each backend kernel. If the selected attention backend does not declare supports_ring_rotation(), the per-hop merge cannot be done correctly, so the layer constructor raises at init time when ring parallel world size > 1.","triggerScenarios":"Constructing this attention layer (e.g. USPAttention) with ring parallelism enabled (ring world size > 1) while the chosen/default attention backend (attn_backend) is one whose supports_ring_rotation() returns False, e.g. a backend without LSE exposure like some SDPA/Triton paths.","commonSituations":"Enabling ring attention with a backend override to a non-supporting backend; default backend resolution on a platform (e.g. ROCm or CPU) picking a backend without LSE; upgrading sglang where a backend's ring support flag changed; misconfigured attn_backend in server args combined with ring parallel flags.","solutions":["Switch the attention backend to one that declares supports_ring_rotation() (typically the FA backend)","Or disable ring parallelism (ring world size 1) if the backend must stay as-is","Check AttentionBackend.supports_ring_rotation for your chosen backend and align backend selection with ring parallel settings in server args"],"exampleFix":"# before\n# ring parallelism enabled, attn_backend=SDPA\n# after\nserver_args.attn_backend = \"fa\"  # FA exposes softmax LSE for ring rotation merge","handlingStrategy":"validation","validationCode":"from sglang... import get_ring_parallel_world_size\nif get_ring_parallel_world_size() > 1:\n    assert attn_backend.supports_ring_rotation(), f\"{attn_backend.get_enum().name} lacks ring rotation support; use FA\"","typeGuard":"def backend_ring_ok(backend) -> bool:\n    return backend.supports_ring_rotation()","tryCatchPattern":"try:\n    layer = USPAttention(...)\nexcept RuntimeError as e:\n    if \"supports_ring_rotation\" in str(e):\n        # fall back: disable ring or switch backend before re-init\n        raise\n    raise","preventionTips":["Cross-check attn_backend and ring parallel flags in server args before launch","Add a startup config validator: ring_ws>1 implies backend.supports_ring_rotation()"],"tags":["attention","ring-parallelism","backend-support","lse","initialization"],"backgroundTag":"backend-capability-unsupported","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}