sgl-project/sglang · error · NotImplementedError

Varlen USPAttention does not support ring parallelism yet.

Error message

Varlen USPAttention does not support ring parallelism yet.

What it means

The varlen path of USPAttention performs an all-to-all over the combined sequence-parallel group; under ring parallelism (ring world size > 1) this would shuffle rows across ring ranks instead of rotating KV, silently corrupting output, so it raises NotImplementedError instead.

Solutions

  1. Disable ring parallelism (run with ring world size 1) for varlen workloads
  2. Or pad/bucket requests to uniform lengths so the non-varlen ring path is used
  3. Track the upstream task implementing varlen ring support and upgrade once available

Example fix

# before
# launch with ring parallelism, batch has variable lengths
out = attn(q, k, v, seq_lens=lens)
# after
# disable ring parallelism in server args, keep SP + varlen
out = attn(q, k, v, seq_lens=lens)
Defensive patterns

Strategy: validation

Validate before calling

if seq_lens is not None and get_ring_parallel_world_size() > 1 and get_sequence_parallel_world_size() > 1:
    raise ValueError("varlen + ring parallelism unsupported; disable ring or pad batch")

Type guard

def varlen_ring_ok(seq_lens, ring_ws: int) -> bool:
    return seq_lens is None or ring_ws == 1

Prevention

When it happens

Trigger: Calling USPAttention.forward with seq_lens not None (ragged/varlen batch), sequence parallel world size > 1, effective_skip_sp False, and ring parallel world size > 1.

Common situations: Serving variable-length multimodal prompts on a ring-attention-enabled deployment; enabling ring parallelism for throughput on a workload that produces varlen batches; a config change that turns on ring attention for a model whose batches are ragged.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/354d2c9af887182f. Report an issue: GitHub.

Appendix: source

Thrown at python/sglang/multimodal_gen/runtime/layers/attention/layer.py:878

        ctx_attn_metadata = forward_context.attn_metadata
        effective_skip_sp = (
            self.skip_sequence_parallel or skip_sequence_parallel_override
        )
        if seq_lens is not None:
            assert (
                attn_mask is None
                and attn_mask_meta is None
                and not num_replicated_prefix
                and not num_replicated_suffix
                and not num_replicated_kv_prefix
            ), "Varlen USPAttention does not support masks or replicated tokens"
            if effective_skip_sp or get_sequence_parallel_world_size() == 1:
                return self.attn_impl.forward(q, k, v, ctx_attn_metadata)
            if get_ring_parallel_world_size() > 1:
                # The varlen all-to-all spans the combined SP group and is not
                # ring-aware; it would shuffle rows across ring ranks instead
                # of rotating KV, corrupting the output silently.
                raise NotImplementedError(
                    "Varlen USPAttention does not support ring parallelism yet."
                )
            qkv = torch.cat([q, k, v], dim=0)
            qkv = _usp_input_all_to_all_varlen(qkv, seq_lens, head_dim=2)
            qkv = self.attn_impl.preprocess_qkv(qkv, ctx_attn_metadata)
            q, k, v = qkv.chunk(3, dim=0)
            out = self.attn_impl.forward(q, k, v, ctx_attn_metadata)
            out = self.attn_impl.postprocess_output(out, ctx_attn_metadata)
            return _usp_output_all_to_all_varlen(out, seq_lens, head_dim=2)

        if isinstance(attn_mask_meta, DynamicVarlenMaskMeta):
            attn_mask_meta = attn_mask_meta.resolve(attn_mask)

        # Tail-pad meta alone (sp_shard.tail_attn_meta; mask derivable from the
        # pad span) also opts into the masked SP branch. gap_* = legacy alias.
        meta_pad_start = meta_pad_end = None
        if attn_mask_meta is not None:
            meta_pad_start = attn_mask_meta.get(

View on GitHub (pinned to 0132848349)