{"record":{"id":"e60ff63a39daf09a","repo":"sgl-project/sglang","slug":"ulyssesattention-s-all-to-all-spans-the-combined-s","errorCode":null,"errorMessage":"UlyssesAttention's all-to-all spans the combined sequence parallel group and is not ring-aware; it would silently shuffle across ring ranks instead of rotating KV within them. Ring parallelism is not supported for models still using UlyssesAttention -- use USPAttention instead.","messagePattern":"UlyssesAttention's all-to-all spans the combined sequence parallel group and is not ring-aware; it would silently shuffle across ring ranks instead of rotating KV within them\\. Ring parallelism is not supported for models still using UlyssesAttention -- use USPAttention instead\\.","errorType":"exception","errorClass":"NotImplementedError","httpStatus":null,"severity":"critical","filePath":"python/sglang/multimodal_gen/runtime/layers/attention/layer.py","lineNumber":338,"sourceCode":"\n\nclass UlyssesAttention(nn.Module):\n    \"\"\"Ulysses-style SequenceParallelism attention layer.\"\"\"\n\n    def __init__(\n        self,\n        num_heads: int,\n        head_size: int,\n        num_kv_heads: int | None = None,\n        softmax_scale: float | None = None,\n        causal: bool = False,\n        supported_attention_backends: set[AttentionBackendEnum] | None = None,\n        prefix: str = \"\",\n        **extra_impl_args,\n    ) -> None:\n        super().__init__()\n        if get_ring_parallel_world_size() > 1:\n            raise NotImplementedError(\n                \"UlyssesAttention's all-to-all spans the combined sequence \"\n                \"parallel group and is not ring-aware; it would silently \"\n                \"shuffle across ring ranks instead of rotating KV within \"\n                \"them. Ring parallelism is not supported for models still \"\n                \"using UlyssesAttention -- use USPAttention instead.\"\n            )\n        if softmax_scale is None:\n            self.softmax_scale = head_size**-0.5\n        else:\n            self.softmax_scale = softmax_scale\n\n        if num_kv_heads is None:\n            num_kv_heads = num_heads\n\n        dtype = get_compute_dtype()\n        attn_backend = get_attn_backend(\n            head_size, dtype, supported_attention_backends=supported_attention_backends\n        )","sourceCodeStart":320,"sourceCodeEnd":356,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/multimodal_gen/runtime/layers/attention/layer.py#L320-L356","documentation":"UlyssesAttention uses an all-to-all over the combined sequence-parallel group, which is incompatible with ring parallelism: it would shuffle tokens across ring ranks instead of rotating KV within them, silently corrupting outputs. The constructor therefore hard-fails when ring parallel world size > 1 and directs you to the ring-aware USPAttention.","triggerScenarios":"Instantiating UlyssesAttention while the ring parallel world size (get_ring_parallel_world_size()) is greater than 1, i.e. running with ring attention enabled in the distributed launch config.","commonSituations":"Enabling ring attention flags in the server args for a model whose code still uses the legacy UlyssesAttention class; upgrading a launch script to ring parallelism without updating model code; porting a model from USPAttention-based template but keeping the old class name.","solutions":["Replace UlyssesAttention with USPAttention in the model's attention layers (USPAttention is ring-aware)","Or disable ring parallelism (run with ring parallel world size 1) if ring attention is not required","Check the server/launcher args that set the ring parallel group size and reduce it to 1"],"exampleFix":"# before\nclass MyAttn(nn.Module):\n    self.attn = UlyssesAttention(...)\n# after\nclass MyAttn(nn.Module):\n    self.attn = USPAttention(...)","handlingStrategy":"validation","validationCode":"from sglang... import get_ring_parallel_world_size\nif get_ring_parallel_world_size() > 1:\n    assert not isinstance(self.attn, UlyssesAttention), \"use USPAttention under ring parallelism\"","typeGuard":"def ring_safe_attn_cls(ring_ws: int):\n    return USPAttention if ring_ws > 1 else UlyssesAttention","tryCatchPattern":null,"preventionTips":["Gate model attention class selection on ring world size at build time","Run a 2-rank smoke test whenever enabling ring parallelism on a new model","Search model code for UlyssesAttention before enabling ring flags"],"tags":["attention","ring-parallelism","sequence-parallel","distributed","not-implemented"],"backgroundTag":"parallelism-strategy-unsupported","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}