{"record":{"id":"1d2544865a5f46b3","repo":"sgl-project/sglang","slug":"packed-seq-len-seq-len-not-divisible-by-the-comb","errorCode":null,"errorMessage":"packed seq_len {seq_len} not divisible by the combined sequence-parallel world size {sp_ws} (ulysses={ulysses_ws} x ring={ring_ws})","messagePattern":"packed seq_len (.+?) not divisible by the combined sequence-parallel world size (.+?) \\(ulysses=(.+?) x ring=(.+?)\\)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/multimodal_gen/runtime/models/dits/minimax_h3.py","lineNumber":2189,"sourceCode":"        \"\"\"Build request-static RoPE inputs for this rank's row shard.\n\n        Same 2D row split as forward(): ring first (outer, contiguous\n        ring_chunk_len slice), Ulysses second (inner slice within that\n        chunk) -- see forward()'s row_start derivation for the identity\n        this must stay in sync with.\n        \"\"\"\n        self.materialize_mps_non_layer_weights(\"rope\")\n        if img_position_ids.dim() != 3 or img_position_ids.shape[0] != 1:\n            raise ValueError(\n                \"img_position_ids must be [1, S, 3], got \"\n                f\"{list(img_position_ids.shape)}\"\n            )\n        seq_len = int(img_position_ids.shape[1])\n        ulysses_ws, ulysses_rank = get_ulysses_ctx()\n        ring_ws, ring_rank = get_ring_ctx()\n        sp_ws = ulysses_ws * ring_ws\n        if seq_len % sp_ws:\n            raise ValueError(\n                f\"packed seq_len {seq_len} not divisible by the combined \"\n                f\"sequence-parallel world size {sp_ws} \"\n                f\"(ulysses={ulysses_ws} x ring={ring_ws})\"\n            )\n        local_seq_len = seq_len // sp_ws\n        ring_chunk_len = local_seq_len * ulysses_ws\n        row_start = ring_rank * ring_chunk_len + ulysses_rank * local_seq_len\n        rope_freqs = self.rope(\n            img_position_ids[:, row_start : row_start + local_seq_len]\n        ).to(device)\n        result = (\n            _rope_cos_sin_cache(rope_freqs, dtype=_BF16_DTYPE),\n            torch.arange(\n                local_seq_len,\n                device=device,\n                dtype=torch.long,\n            ),\n        )","sourceCodeStart":2171,"sourceCodeEnd":2207,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/multimodal_gen/runtime/models/dits/minimax_h3.py#L2171-L2207","documentation":"build_rope_cache computes local sequence chunks under combined sequence parallelism: the packed seq_len must be divisible by sp_ws = ulysses_ws * ring_ws (read from the distributed context via get_ulysses_ctx/get_ring_ctx). A non-divisible length means the ring chunk / Ulysses slice boundaries would cut through a token position, so it raises showing seq_len and both world sizes.","triggerScenarios":"Running with ulysses=4, ring=2 (sp_ws=8) but a packed seq_len like 1000 (1000 % 8 != 0); typically happens when the packer pads to a different alignment than the runtime SP degree.","commonSituations":"Changing ulysses/ring at launch without updating the packing alignment; feeding unpadded or differently-padded sequences in a test/eval harness outside the normal packer; padding removed by an off-by-one.","solutions":["Pad the packed sequence length to a multiple of ulysses_ws * ring_ws before calling build_rope_cache","Or lower the SP degrees so the product divides the current seq_len","Ensure your sequence packer pads to max(MINIMAX_H3_PACKED_SEQUENCE_ALIGNMENT, ulysses*ring)"],"exampleFix":"# before\nrope = model.build_rope_cache(img_position_ids=pos)  # S=1000, sp_ws=8\n# after\nsp_ws = ulysses_ws * ring_ws\npad = (-S) % sp_ws\npos = torch.cat([pos, torch.full((1, pad, 3), -1, dtype=pos.dtype)], dim=1)\nrope = model.build_rope_cache(img_position_ids=pos)","handlingStrategy":"validation","validationCode":"ulysses_ws, _ = get_ulysses_ctx(); ring_ws, _ = get_ring_ctx()\nsp_ws = ulysses_ws * ring_ws\nif seq_len % sp_ws:\n    pad = (-seq_len) % sp_ws\n    img_position_ids = torch.cat([img_position_ids, torch.zeros(1, pad, 3, dtype=img_position_ids.dtype)], dim=1)","typeGuard":"def seq_divisible(seq_len: int, sp_ws: int) -> bool:\n    return sp_ws >= 1 and seq_len % sp_ws == 0","tryCatchPattern":null,"preventionTips":["Pad packed sequences to max(alignment, ulysses*ring)","Re-check padding whenever SP degrees change at launch"],"tags":["sequence-parallel","ulysses","ring-attention","divisibility","padding"],"backgroundTag":"tensor-shape-mismatch","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}