{"record":{"id":"8478823282da8f58","repo":"sgl-project/sglang","slug":"world-size-world-size-is-not-equal-to-tensor-m","errorCode":null,"errorMessage":"world_size ({world_size}) is not equal to tensor_model_parallel_size ({tensor_model_parallel_size}) x pipeline_model_parallel_size ({pipeline_model_parallel_size})","messagePattern":"world_size \\((.+?)\\) is not equal to tensor_model_parallel_size \\((.+?)\\) x pipeline_model_parallel_size \\((.+?)\\)","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/distributed/parallel_state.py","lineNumber":2415,"sourceCode":"\n    Note that for efficiency, the caller should make sure adjacent ranks\n    are on the same DGX box. For example if we are using 2 DGX-1 boxes\n    with a total of 16 GPUs, rank 0 to 7 belong to the first box and\n    ranks 8 to 15 belong to the second box.\n    \"\"\"\n    # Get world size and rank. Ensure some consistencies.\n    assert torch.distributed.is_initialized()\n    backend = backend or torch.distributed.get_backend(get_world_group().device_group)\n\n    # Joiners construct their local TP/PP layout in global rank space.\n    world_size: int = (\n        tensor_model_parallel_size * pipeline_model_parallel_size\n        if recovered_rank\n        else torch.distributed.get_world_size()\n    )\n\n    if world_size != tensor_model_parallel_size * pipeline_model_parallel_size:\n        raise RuntimeError(\n            f\"world_size ({world_size}) is not equal to \"\n            f\"tensor_model_parallel_size ({tensor_model_parallel_size}) x \"\n            f\"pipeline_model_parallel_size ({pipeline_model_parallel_size})\"\n        )\n    if decode_context_parallel_size < 1:\n        raise RuntimeError(\n            f\"decode_context_parallel_size ({decode_context_parallel_size}) must be >= 1\"\n        )\n    if decode_context_parallel_size > 1 and not (is_hip() or is_cuda()):\n        raise RuntimeError(\n            \"Decode context parallel (decode_context_parallel_size > 1) is \"\n            \"currently only supported on the AMD HIP platform or CUDA platform, but got \"\n            f\"decode_context_parallel_size ({decode_context_parallel_size}) \"\n            \"on a non-HIP or non-CUDA platform.\"\n        )\n    if tensor_model_parallel_size % decode_context_parallel_size != 0:\n        raise RuntimeError(\n            f\"tensor_model_parallel_size ({tensor_model_parallel_size}) must be divisible by \"","sourceCodeStart":2397,"sourceCodeEnd":2433,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/distributed/parallel_state.py#L2397-L2433","documentation":"initialize_model_parallel validates that world_size == tensor_model_parallel_size * pipeline_model_parallel_size (using the torch.distributed world size, or the recovered rank's world size when recovering). TP and PP must exactly tile the total number of ranks; any remainder is a configuration error.","triggerScenarios":"Calling initialize_model_parallel (via _init_parallel_groups or ensure_model_parallel_initialized) with --tp-size and --pp-size whose product differs from the total world size (number of GPUs/processes in the torch.distributed job), e.g. tp=4, pp=2 with 8 world size mismatch (needs 8 = 4*2; fails for tp=4, pp=1 on 8 GPUs when DP is not accounted for).","commonSituations":"Launching on a different GPU count than the script assumes (e.g. 8-GPU launch args on a 4-GPU node); forgetting that data-parallel/attention-EP ranks must be included in world_size; changing --tp-size without adjusting --pp-size or GPU count; recovery paths with a stale world size.","solutions":["Make tp_size * pp_size equal the number of distributed ranks, e.g. 8 GPUs -> --tp 8 --pp 1, or --tp 4 --pp 2","Check torch.distributed.get_world_size() / number of processes (rays/CPU workers excluded appropriately) and adjust launch flags","If you intended data parallelism, configure the DP dimension so that the overall rank count matches the product per replica"],"exampleFix":"# before (8 GPUs)\npython -m sglang.launch_server --model M --tp-size 4 --pp-size 1\n# RuntimeError: world_size (8) is not equal to 4 x 1\n\n# after\npython -m sglang.launch_server --model M --tp-size 8 --pp-size 1\n# or\npython -m sglang.launch_server --model M --tp-size 4 --pp-size 2","handlingStrategy":"validation","validationCode":"import torch\nws = torch.distributed.get_world_size()\nassert ws == tp_size * pp_size, f'{ws} != {tp_size}x{pp_size}; fix --tp/--pp or GPU count'","typeGuard":null,"tryCatchPattern":"try:\n    initialize_model_parallel(tensor_model_parallel_size=tp, pipeline_model_parallel_size=pp)\nexcept RuntimeError as e:\n    if 'world_size' in str(e):\n        raise SystemExit(f'Adjust tp*pp to world size: {e}') from e\n    raise","preventionTips":["Derive tp/pp from detected GPU count in launch scripts instead of hardcoding","Add a pre-launch assert world_size == tp*pp","Watch for leftover/extra processes inflating world_size"],"tags":["parallelism","tensor-parallel","pipeline-parallel","world-size","config-validation","sglang"],"backgroundTag":"parallel-config-validation","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}