sgl-project/sglang · error · NotImplementedError

USPAttention masked path supports ring parallelism only for

Error message

USPAttention masked path supports ring parallelism only for batch-1 tail-pad metadata on the FA backend.

What it means

Under ring parallelism, the USPAttention masked path only has a correct ring implementation for the narrow case of batch size 1, tail-padding-only mask metadata, on the FA backend. Any other masked configuration (batch > 1, non-tail-pad metadata, or a non-FA backend) has no ring-aware implementation and is rejected.

Source

Thrown at python/sglang/multimodal_gen/runtime/layers/attention/layer.py:1053

                with sdpa_context:
                    return torch.nn.functional.scaled_dot_product_attention(
                        q_,
                        k_,
                        v_,
                        attn_mask=mask,
                        dropout_p=0.0,
                        is_causal=False,
                        scale=self.softmax_scale,
                    ).transpose(1, 2)

            if get_ring_parallel_world_size() > 1:
                if (
                    meta_only_pad
                    and q.shape[0] == 1
                    and self.backend == AttentionBackendEnum.FA
                ):
                    return self._forward_ring_tail_pad(q, k, v, attn_mask_meta)
                raise NotImplementedError(
                    "USPAttention masked path supports ring parallelism only "
                    "for batch-1 tail-pad metadata on the FA backend."
                )
            if attn_mask is not None and attn_mask.dim() != 2:
                raise NotImplementedError(
                    "USPAttention masked SP path currently expects a [B, S_local] key mask."
                )

            sp_size = get_ulysses_parallel_world_size()
            if sp_size > 1 and not qkv_pre_all_to_all:
                qkv_fast = _ipc_input_a2a_qkv(q, k, v)
                if qkv_fast is not None:
                    q, k, v = qkv_fast
                else:
                    q, k, v = _usp_input_all_to_all_qkv(q, k, v)

            if (
                _VARLEN_FA_ENABLED

View on GitHub (pinned to 0132848349)

Solutions

1. Restructure to batch size 1 with tail-padding and express the mask as pad-only attn_mask_meta on the FA backend
2. Or disable ring parallelism for masked batched workloads
3. Or switch to the unmasked/varlen path which has its own (non-ring) SP handling

Example fix

# before
out = attn(q, k, v, attn_mask=bool_mask)  # ring parallel, batch=4
# after
# single request, tail pad metadata, FA backend
out = attn(q, k, v[:1], k, v, attn_mask_meta=tail_pad_meta)

ring = get_ring_parallel_world_size() > 1
if ring and attn_mask is not None:
    supported = meta_only_pad and q.shape[0] == 1 and layer.backend == AttentionBackendEnum.FA
    if not supported:
        raise ValueError("ring masked path needs batch=1 tail-pad meta on FA")

def ring_masked_ok(q, meta_only_pad: bool, backend) -> bool:
    return meta_only_pad and q.shape[0] == 1 and backend == AttentionBackendEnum.FA

When it happens

Trigger: Calling USPAttention.forward with an attn_mask/attn_mask_meta under ring parallel world size > 1 when NOT (meta_only_pad and q.shape[0] == 1 and backend == FA) — e.g. batch size > 1, a general 2D mask instead of pad metadata, or a non-FA backend.

Common situations: Batched video/image generation with padding masks under ring attention; switching the backend from FA to FlashInfer/Triton while ring parallelism is on; passing a boolean [B, S] mask instead of pad-only metadata.

Related errors

• UlyssesAttention's all-to-all spans the combined sequence pa
• Varlen USPAttention does not support ring parallelism yet.
• USPAttention's masked path does not support replicated prefi
• K/V-gather SP does not support varlen UlyssesAttention.
• Ring Attention requires a backend whose kernel exposes the s

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/eed84493cccce534. Report an issue: GitHub.