{"record":{"id":"798e1b5d3c8bc5d4","repo":"sgl-project/sglang","slug":"world-size-world-size-is-less-than-tensor-para","errorCode":null,"errorMessage":"world_size ({world_size}) is less than tensor_parallel_degree ({tensor_parallel_degree}) x pipeline_parallel_degree ({pipeline_parallel_degree}) xsequence_parallel_degree ({sequence_parallel_degree}) xclassifier_free_guidance_degree ({classifier_free_guidance_degree}) xdata_parallel_degree ({data_parallel_size})","messagePattern":"world_size \\((.+?)\\) is less than tensor_parallel_degree \\((.+?)\\) x pipeline_parallel_degree \\((.+?)\\) xsequence_parallel_degree \\((.+?)\\) xclassifier_free_guidance_degree \\((.+?)\\) xdata_parallel_degree \\((.+?)\\)","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"critical","filePath":"python/sglang/multimodal_gen/runtime/distributed/parallel_state.py","lineNumber":443,"sourceCode":"    if backend is None:\n        from sglang.multimodal_gen.runtime.platforms import current_platform\n\n        backend = current_platform.get_torch_distributed_backend_str()\n    # Get world size and rank. Ensure some consistencies.\n    assert torch.distributed.is_initialized()\n    world_size: int = torch.distributed.get_world_size()\n    backend = backend or torch.distributed.get_backend(get_world_group().device_group)\n\n    dit_parallel_size = (\n        data_parallel_size\n        * classifier_free_guidance_degree\n        * sequence_parallel_degree\n        * pipeline_parallel_degree\n        * tensor_parallel_degree\n    )\n\n    if world_size < dit_parallel_size:\n        raise RuntimeError(\n            f\"world_size ({world_size}) is less than \"\n            f\"tensor_parallel_degree ({tensor_parallel_degree}) x \"\n            f\"pipeline_parallel_degree ({pipeline_parallel_degree}) x\"\n            f\"sequence_parallel_degree ({sequence_parallel_degree}) x\"\n            f\"classifier_free_guidance_degree \"\n            f\"({classifier_free_guidance_degree}) x\"\n            f\"data_parallel_degree ({data_parallel_size})\"\n        )\n\n    rank_generator: RankGenerator = RankGenerator(\n        tensor_parallel_degree,\n        sequence_parallel_degree,\n        pipeline_parallel_degree,\n        classifier_free_guidance_degree,\n        data_parallel_size,\n        \"tp-sp-pp-cfg-dp\",\n    )\n    global _DP","sourceCodeStart":425,"sourceCodeEnd":461,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/multimodal_gen/runtime/distributed/parallel_state.py#L425-L461","documentation":"Raised by initialize_model_parallel when the total requested parallelism (tensor x pipeline x sequence x classifier-free-guidance x data parallel degrees) exceeds the initialized world_size. The product of all parallel degrees must fit within the number of distributed ranks.","triggerScenarios":"Calling maybe_init_distributed_environment_and_model_parallel with e.g. tp=8, pp=2 on a world_size of 8, or setting data_parallel_size such that tp*pp*sp*cfg*dp > world_size.","commonSituations":"Misconfigured CLI/server args (e.g. --tp 8 on 4 GPUs), forgetting that data_parallel_size multiplies the constraint, or changing GPU count without updating parallel degrees in config files.","solutions":["Reduce the parallel degrees (tp/pp/sp/cfg) so their product times data_parallel_size is <= world_size","Or increase world_size (more ranks/GPUs) to at least the product of the degrees","Verify data_parallel_size is not implicitly inflating the requirement; often world_size/tp should equal dp, not require dp extra ranks"],"exampleFix":"# before\n--tensor-parallel-degree 8 --data-parallel-size 2   # world_size=8\n\n# after\n--tensor-parallel-degree 4 --data-parallel-size 2   # 4*2=8 <= 8","handlingStrategy":"validation","validationCode":"def check_degrees(world_size, tp, pp, sp, cfg, dp):\n    need = tp * pp * sp * cfg * dp\n    assert world_size >= need, f\"need {need} ranks, have {world_size}\"\n    return True","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Derive one degree from the others (e.g. dp = world_size // (tp*pp)) instead of over-specifying","Add a startup assertion on the degree product before launching distributed init","Validate config in CI against the actual GPU count"],"tags":["distributed","parallelism","configuration","startup"],"backgroundTag":"parallel-degree-exceeds-world-size","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}