{"record":{"id":"a12b9b507d575f1f","repo":"sgl-project/sglang","slug":"k-v-gather-sp-does-not-support-varlen-ulyssesatten","errorCode":null,"errorMessage":"K/V-gather SP does not support varlen UlyssesAttention.","messagePattern":"K/V-gather SP does not support varlen UlyssesAttention\\.","errorType":"exception","errorClass":"NotImplementedError","httpStatus":null,"severity":"error","filePath":"python/sglang/multimodal_gen/runtime/layers/attention/layer.py","lineNumber":397,"sourceCode":"        self.sp_attention_mode, self.sp_attention_mode_is_auto = (\n            _resolve_sp_attention_mode(\n                causal=causal, sparse_backend=self.backend.is_sparse\n            )\n        )\n\n    def _forward_with_kv_gather(\n        self,\n        q: torch.Tensor,\n        k: torch.Tensor,\n        v: torch.Tensor,\n        ctx_attn_metadata,\n        replicated_q: torch.Tensor | None,\n        replicated_k: torch.Tensor | None,\n        replicated_v: torch.Tensor | None,\n        seq_lens: list[int] | None,\n    ) -> tuple[torch.Tensor, torch.Tensor | None]:\n        if seq_lens is not None:\n            raise NotImplementedError(\n                \"K/V-gather SP does not support varlen UlyssesAttention.\"\n            )\n        if any(x is not None for x in (replicated_q, replicated_k, replicated_v)):\n            if any(x is None for x in (replicated_q, replicated_k, replicated_v)):\n                raise ValueError(\"Replicated Q, K, and V must be provided together.\")\n\n        k = sequence_model_parallel_all_gather(k, dim=1)\n        v = sequence_model_parallel_all_gather(v, dim=1)\n\n        local_query_len = q.shape[1]\n        if replicated_q is not None:\n            q = torch.cat([q, replicated_q], dim=1)\n            k = torch.cat([k, replicated_k], dim=1)\n            v = torch.cat([v, replicated_v], dim=1)\n\n        output = self.attn_impl.forward(q, k, v, ctx_attn_metadata)\n        if replicated_q is None:\n            return output, None","sourceCodeStart":379,"sourceCodeEnd":415,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/multimodal_gen/runtime/layers/attention/layer.py#L379-L415","documentation":"The K/V-gather sequence-parallel path of UlyssesAttention only supports uniform (non-varlen) batches. If per-row sequence lengths (seq_lens) are passed to _forward_with_kv_gather, it raises rather than producing wrong results across the gather.","triggerScenarios":"Calling UlyssesAttention.forward with seq_lens not None while the layer is configured with sp_attention_mode == 'kv_gather', on a batch with variable sequence lengths (e.g. multimodal batches with mixed image/text sizes).","commonSituations":"Serving variable-length multimodal prompts under K/V-gather SP mode; switching a workload from fixed-length synthetic batches to real ragged batches; enabling kv_gather attention mode in server args and sending a batch that triggers the varlen path.","solutions":["Switch sp_attention_mode away from kv_gather to a mode that supports varlen (e.g. the default all-to-all/USP path)","Pad/bucket the batch to uniform sequence lengths so seq_lens is None","Upgrade the model code to USPAttention which handles varlen under SP"],"exampleFix":"# before\nout = attn(q, k, v, seq_lens=[5, 17, 9])  # attn uses kv_gather SP\n# after\nout = attn(q, k, v, seq_lens=None)  # uniform batch, or use USPAttention for varlen","handlingStrategy":"validation","validationCode":"if seq_lens is not None and attn.sp_attention_mode == \"kv_gather\":\n    raise ValueError(\"kv_gather SP requires uniform lengths; pad the batch or switch mode\")","typeGuard":"def kv_gather_ok(attn, seq_lens) -> bool:\n    return seq_lens is None or getattr(attn, \"sp_attention_mode\", None) != \"kv_gather\"","tryCatchPattern":"try:\n    out = attn(q, k, v, seq_lens=seq_lens)\nexcept NotImplementedError as e:\n    if \"kv_gather\" in str(e):\n        seq_lens = None  # after padding to uniform length\n        out = attn(q, k, v, seq_lens=None)\n    else:\n        raise","preventionTips":["Check sp_attention_mode before enabling kv_gather in server args","Pad multimodal batches to a common length when using kv_gather","Prefer USPAttention for ragged production traffic"],"tags":["attention","sequence-parallel","kv-gather","varlen","not-implemented"],"backgroundTag":"varlen-batch-unsupported","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}