{"record":{"id":"4bb7de3f7e7cf936","repo":"sgl-project/sglang","slug":"uspattention-s-masked-path-does-not-support-replic","errorCode":null,"errorMessage":"USPAttention's masked path does not support replicated prefix/suffix tokens under sequence parallelism; drop attn_mask/attn_mask_meta or the replicated segment.","messagePattern":"USPAttention's masked path does not support replicated prefix/suffix tokens under sequence parallelism; drop attn_mask/attn_mask_meta or the replicated segment\\.","errorType":"exception","errorClass":"NotImplementedError","httpStatus":null,"severity":"error","filePath":"python/sglang/multimodal_gen/runtime/layers/attention/layer.py","lineNumber":953,"sourceCode":"                raise NotImplementedError(unsupported)\n\n        if attn_mask is not None or meta_only_pad:\n            if (\n                (\n                    num_replicated_prefix\n                    or num_replicated_suffix\n                    or num_replicated_kv_prefix\n                )\n                and not effective_skip_sp\n                and get_sequence_parallel_world_size() > 1\n            ):\n                # Under SP this path shards every row through the all-to-all;\n                # a replicated prefix/suffix would be duplicated across ranks\n                # and silently corrupt the output, so refuse loudly instead.\n                # On a single rank the mask already describes the full\n                # sequence and the replicated counts are meaningless, so the\n                # call is legal.\n                raise NotImplementedError(\n                    \"USPAttention's masked path does not support replicated \"\n                    \"prefix/suffix tokens under sequence parallelism; drop \"\n                    \"attn_mask/attn_mask_meta or the replicated segment.\"\n                )\n\n            def _prepare_sdpa_mask(\n                mask: torch.Tensor, *, dtype: torch.dtype, device: torch.device\n            ) -> torch.Tensor:\n                mask = mask.to(device=device)\n                if torch.is_floating_point(mask):\n                    mask = mask.to(dtype=dtype)\n                    if mask.dim() == 2:\n                        mask = mask[:, None, None, :]\n                    elif mask.dim() == 3:\n                        mask = mask[:, None, :, :]\n                    return mask\n\n                mask = mask.to(dtype=dtype)","sourceCodeStart":935,"sourceCodeEnd":971,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/multimodal_gen/runtime/layers/attention/layer.py#L935-L971","documentation":"The masked path of USPAttention shards every row through the SP all-to-all. Replicated prefix/suffix tokens (e.g. shared text around an image) would be duplicated across ranks and silently corrupt the output, so under sequence parallelism the combination of an explicit attn_mask/attn_mask_meta and replicated tokens is refused. On a single rank it is legal.","triggerScenarios":"Calling USPAttention.forward with both an attn_mask or attn_mask_meta AND replicated prefix/suffix tokens, while sequence parallel world size > 1 (and not on the supported ring tail-pad path).","commonSituations":"Multimodal batches with a shared text prompt plus an explicit attention mask under SP; migrating a single-GPU masked attention path to multi-GPU SP without dropping the replicated segment; adding attn_mask_meta for prefix masking while keeping replicated_q/k/v arguments populated.","solutions":["Drop the attn_mask/attn_mask_meta and rely on the layer's native prefix handling","Or remove the replicated prefix/suffix segment from the batch construction (let every rank own its tokens)","Or run with sequence parallel world size 1 if the masked+replicated combination is required"],"exampleFix":"# before\nout = attn(q, k, v, attn_mask=mask, replicated_q=rq, replicated_k=rk, replicated_v=rv)\n# after\nout = attn(q, k, v, replicated_q=rq, replicated_k=rk, replicated_v=rv)  # no mask","handlingStrategy":"validation","validationCode":"has_replicated = any(x is not None for x in (replicated_q, replicated_k, replicated_v))\nhas_mask = attn_mask is not None or attn_mask_meta is not None\nif has_replicated and has_mask and get_sequence_parallel_world_size() > 1:\n    raise ValueError(\"masked path + replicated tokens unsupported under SP; drop one\")","typeGuard":"def masked_replicated_sp_ok(attn_mask, attn_mask_meta, rq, rk, rv, sp_ws: int) -> bool:\n    if sp_ws == 1:\n        return True\n    has_rep = any(x is not None for x in (rq, rk, rv))\n    return not (has_rep and (attn_mask is not None or attn_mask_meta is not None))","tryCatchPattern":null,"preventionTips":["Encode prefix masking via the layer's native prefix args, not explicit masks, under SP","Run the single-rank vs multi-rank output equality test when adding replicated segments"],"tags":["attention","sequence-parallel","attention-mask","replicated-tokens","not-implemented"],"backgroundTag":"parallelism-strategy-unsupported","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}