sgl-project/sglang · error · ValueError

world_size must be positive and divide global_heads

Error message

world_size must be positive and divide global_heads

What it means

The packing layout assumes world_size evenly partitions the global heads dimension: output shape is [world_size, rows, global_heads/world_size, 3*head_size]. It raises when world_size < 1 or global_heads % world_size != 0.

Source

Thrown at python/sglang/kernels/ops/diffusion/layout/ulysses_qkv_triton.py:72

def pack_qkv_destination_major(
    q: torch.Tensor,
    k: torch.Tensor,
    v: torch.Tensor,
    world_size: int,
    out: torch.Tensor | None = None,
) -> torch.Tensor:
    """Pack matching ``[rows, global_heads, head_size]`` Q/K/V tensors."""
    if q.dim() != 3 or q.shape != k.shape or q.shape != v.shape:
        raise ValueError("q, k, and v must have the same 3D shape")
    if not (q.is_cuda and k.is_cuda and v.is_cuda):
        raise ValueError("q, k, and v must be CUDA tensors")
    if not (q.device == k.device == v.device and q.dtype == k.dtype == v.dtype):
        raise ValueError("q, k, and v must have the same device and dtype")
    if q.stride(-1) != 1 or k.stride(-1) != 1 or v.stride(-1) != 1:
        raise ValueError("q, k, and v must be contiguous in head_size")
    if world_size < 1 or q.shape[1] % world_size != 0:
        raise ValueError("world_size must be positive and divide global_heads")

    rows, global_heads, head_size = q.shape
    local_heads = global_heads // world_size
    expected_shape = (world_size, rows, local_heads, 3 * head_size)
    if out is not None:
        if not (
            out.shape == expected_shape
            and out.is_contiguous()
            and out.dtype == q.dtype
            and out.device == q.device
        ):
            raise ValueError(
                "out must be a contiguous tensor with the expected shape, "
                "device, and dtype"
            )
        output = out
    else:
        output = torch.empty(

View on GitHub (pinned to 0132848349)

Solutions

1. Use a world_size that divides q.shape[1] (global_heads)
2. Reduce tensor-parallel degree, or repeat/expand heads (model config change) so heads divide evenly
3. Verify the distributed process group initialized correctly before reading world_size
4. Add an assert global_heads % world_size == 0 at pipeline setup time to fail fast with a clear message

Example fix

# before
packed = pack_qkv_destination_major(q, k, v, world_size=8)  # 12 heads
# after
assert q.shape[1] % world_size == 0, f"heads {q.shape[1]} not divisible by {world_size}"
packed = pack_qkv_destination_major(q, k, v, world_size=4)

assert world_size >= 1 and q.shape[1] % world_size == 0

When it happens

Trigger: Passing world_size=0 or a negative value (often from an uninitialized or failed distributed init), or a world size (e.g. 8) that does not divide the model's head count (e.g. 12 heads).

Common situations: Launching Ulysses sequence parallelism with TP degree that doesn't divide attention heads; world_size derived from a process group that failed to initialize and defaulted to 0; small models with few heads run at high parallelism.

Related errors

• MiniMax H3 TP-local heads {local_heads} must be divisible by
• MiniMax H3 packed sequence alignment {MINIMAX_H3_PACKED_SEQU
• packed seq_len {seq_len} not divisible by the combined seque
• q, k, and v must have the same 3D shape
• q, k, and v must be CUDA tensors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/f3e88a57396d7d46. Report an issue: GitHub.