{"record":{"id":"f3e88a57396d7d46","repo":"sgl-project/sglang","slug":"world-size-must-be-positive-and-divide-global-head","errorCode":null,"errorMessage":"world_size must be positive and divide global_heads","messagePattern":"world_size must be positive and divide global_heads","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/kernels/ops/diffusion/layout/ulysses_qkv_triton.py","lineNumber":72,"sourceCode":"\ndef pack_qkv_destination_major(\n    q: torch.Tensor,\n    k: torch.Tensor,\n    v: torch.Tensor,\n    world_size: int,\n    out: torch.Tensor | None = None,\n) -> torch.Tensor:\n    \"\"\"Pack matching ``[rows, global_heads, head_size]`` Q/K/V tensors.\"\"\"\n    if q.dim() != 3 or q.shape != k.shape or q.shape != v.shape:\n        raise ValueError(\"q, k, and v must have the same 3D shape\")\n    if not (q.is_cuda and k.is_cuda and v.is_cuda):\n        raise ValueError(\"q, k, and v must be CUDA tensors\")\n    if not (q.device == k.device == v.device and q.dtype == k.dtype == v.dtype):\n        raise ValueError(\"q, k, and v must have the same device and dtype\")\n    if q.stride(-1) != 1 or k.stride(-1) != 1 or v.stride(-1) != 1:\n        raise ValueError(\"q, k, and v must be contiguous in head_size\")\n    if world_size < 1 or q.shape[1] % world_size != 0:\n        raise ValueError(\"world_size must be positive and divide global_heads\")\n\n    rows, global_heads, head_size = q.shape\n    local_heads = global_heads // world_size\n    expected_shape = (world_size, rows, local_heads, 3 * head_size)\n    if out is not None:\n        if not (\n            out.shape == expected_shape\n            and out.is_contiguous()\n            and out.dtype == q.dtype\n            and out.device == q.device\n        ):\n            raise ValueError(\n                \"out must be a contiguous tensor with the expected shape, \"\n                \"device, and dtype\"\n            )\n        output = out\n    else:\n        output = torch.empty(","sourceCodeStart":54,"sourceCodeEnd":90,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/kernels/ops/diffusion/layout/ulysses_qkv_triton.py#L54-L90","documentation":"The packing layout assumes world_size evenly partitions the global heads dimension: output shape is [world_size, rows, global_heads/world_size, 3*head_size]. It raises when world_size < 1 or global_heads % world_size != 0.","triggerScenarios":"Passing world_size=0 or a negative value (often from an uninitialized or failed distributed init), or a world size (e.g. 8) that does not divide the model's head count (e.g. 12 heads).","commonSituations":"Launching Ulysses sequence parallelism with TP degree that doesn't divide attention heads; world_size derived from a process group that failed to initialize and defaulted to 0; small models with few heads run at high parallelism.","solutions":["Use a world_size that divides q.shape[1] (global_heads)","Reduce tensor-parallel degree, or repeat/expand heads (model config change) so heads divide evenly","Verify the distributed process group initialized correctly before reading world_size","Add an assert global_heads % world_size == 0 at pipeline setup time to fail fast with a clear message"],"exampleFix":"# before\npacked = pack_qkv_destination_major(q, k, v, world_size=8)  # 12 heads\n# after\nassert q.shape[1] % world_size == 0, f\"heads {q.shape[1]} not divisible by {world_size}\"\npacked = pack_qkv_destination_major(q, k, v, world_size=4)","handlingStrategy":"validation","validationCode":"assert world_size >= 1 and q.shape[1] % world_size == 0","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Choose TP degree dividing attention head count","Verify process group init before reading world_size"],"tags":["distributed","world-size","divisibility","ulysses"],"backgroundTag":"distributed-world-size-mismatch","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}