sgl-project/sglang · error · ValueError

packed seq_len {seq_len} not divisible by the combined seque

Error message

packed seq_len {seq_len} not divisible by the combined sequence-parallel world size {sp_ws} (ulysses={ulysses_ws} x ring={ring_ws})

What it means

build_rope_cache computes local sequence chunks under combined sequence parallelism: the packed seq_len must be divisible by sp_ws = ulysses_ws * ring_ws (read from the distributed context via get_ulysses_ctx/get_ring_ctx). A non-divisible length means the ring chunk / Ulysses slice boundaries would cut through a token position, so it raises showing seq_len and both world sizes.

Source

Thrown at python/sglang/multimodal_gen/runtime/models/dits/minimax_h3.py:2189

        """Build request-static RoPE inputs for this rank's row shard.

        Same 2D row split as forward(): ring first (outer, contiguous
        ring_chunk_len slice), Ulysses second (inner slice within that
        chunk) -- see forward()'s row_start derivation for the identity
        this must stay in sync with.
        """
        self.materialize_mps_non_layer_weights("rope")
        if img_position_ids.dim() != 3 or img_position_ids.shape[0] != 1:
            raise ValueError(
                "img_position_ids must be [1, S, 3], got "
                f"{list(img_position_ids.shape)}"
            )
        seq_len = int(img_position_ids.shape[1])
        ulysses_ws, ulysses_rank = get_ulysses_ctx()
        ring_ws, ring_rank = get_ring_ctx()
        sp_ws = ulysses_ws * ring_ws
        if seq_len % sp_ws:
            raise ValueError(
                f"packed seq_len {seq_len} not divisible by the combined "
                f"sequence-parallel world size {sp_ws} "
                f"(ulysses={ulysses_ws} x ring={ring_ws})"
            )
        local_seq_len = seq_len // sp_ws
        ring_chunk_len = local_seq_len * ulysses_ws
        row_start = ring_rank * ring_chunk_len + ulysses_rank * local_seq_len
        rope_freqs = self.rope(
            img_position_ids[:, row_start : row_start + local_seq_len]
        ).to(device)
        result = (
            _rope_cos_sin_cache(rope_freqs, dtype=_BF16_DTYPE),
            torch.arange(
                local_seq_len,
                device=device,
                dtype=torch.long,
            ),
        )

View on GitHub (pinned to 0132848349)

Solutions

1. Pad the packed sequence length to a multiple of ulysses_ws * ring_ws before calling build_rope_cache
2. Or lower the SP degrees so the product divides the current seq_len
3. Ensure your sequence packer pads to max(MINIMAX_H3_PACKED_SEQUENCE_ALIGNMENT, ulysses*ring)

Example fix

# before
rope = model.build_rope_cache(img_position_ids=pos)  # S=1000, sp_ws=8
# after
sp_ws = ulysses_ws * ring_ws
pad = (-S) % sp_ws
pos = torch.cat([pos, torch.full((1, pad, 3), -1, dtype=pos.dtype)], dim=1)
rope = model.build_rope_cache(img_position_ids=pos)

ulysses_ws, _ = get_ulysses_ctx(); ring_ws, _ = get_ring_ctx()
sp_ws = ulysses_ws * ring_ws
if seq_len % sp_ws:
    pad = (-seq_len) % sp_ws
    img_position_ids = torch.cat([img_position_ids, torch.zeros(1, pad, 3, dtype=img_position_ids.dtype)], dim=1)

def seq_divisible(seq_len: int, sp_ws: int) -> bool:
    return sp_ws >= 1 and seq_len % sp_ws == 0

When it happens

Trigger: Running with ulysses=4, ring=2 (sp_ws=8) but a packed seq_len like 1000 (1000 % 8 != 0); typically happens when the packer pads to a different alignment than the runtime SP degree.

Common situations: Changing ulysses/ring at launch without updating the packing alignment; feeding unpadded or differently-padded sequences in a test/eval harness outside the normal packer; padding removed by an off-by-one.

Related errors

• MiniMax H3 packed sequence alignment {MINIMAX_H3_PACKED_SEQU
• MiniMax H3 TP-local heads {local_heads} must be divisible by
• world_size must be positive and divide global_heads
• MiniMax H3 Ulysses size must be positive.
• MiniMax H3 ring size must be positive.

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/1d2544865a5f46b3. Report an issue: GitHub.