{"record":{"id":"981b8f6510830296","repo":"sgl-project/sglang","slug":"tensor-model-parallel-size-tensor-model-parallel","errorCode":null,"errorMessage":"tensor_model_parallel_size ({tensor_model_parallel_size}) must be divisible by decode_context_parallel_size ({decode_context_parallel_size})","messagePattern":"tensor_model_parallel_size \\((.+?)\\) must be divisible by decode_context_parallel_size \\((.+?)\\)","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/distributed/parallel_state.py","lineNumber":2432,"sourceCode":"    if world_size != tensor_model_parallel_size * pipeline_model_parallel_size:\n        raise RuntimeError(\n            f\"world_size ({world_size}) is not equal to \"\n            f\"tensor_model_parallel_size ({tensor_model_parallel_size}) x \"\n            f\"pipeline_model_parallel_size ({pipeline_model_parallel_size})\"\n        )\n    if decode_context_parallel_size < 1:\n        raise RuntimeError(\n            f\"decode_context_parallel_size ({decode_context_parallel_size}) must be >= 1\"\n        )\n    if decode_context_parallel_size > 1 and not (is_hip() or is_cuda()):\n        raise RuntimeError(\n            \"Decode context parallel (decode_context_parallel_size > 1) is \"\n            \"currently only supported on the AMD HIP platform or CUDA platform, but got \"\n            f\"decode_context_parallel_size ({decode_context_parallel_size}) \"\n            \"on a non-HIP or non-CUDA platform.\"\n        )\n    if tensor_model_parallel_size % decode_context_parallel_size != 0:\n        raise RuntimeError(\n            f\"tensor_model_parallel_size ({tensor_model_parallel_size}) must be divisible by \"\n            f\"decode_context_parallel_size ({decode_context_parallel_size})\"\n        )\n\n    # Build the tensor model-parallel groups.\n    num_tensor_model_parallel_groups: int = world_size // tensor_model_parallel_size\n    global _TP\n    assert _TP is None, \"tensor model parallel group is already initialized\"\n    group_ranks = []\n    for tp_group_idx in range(num_tensor_model_parallel_groups):\n        ranks = list(\n            range(\n                tp_group_idx * tensor_model_parallel_size,\n                (tp_group_idx + 1) * tensor_model_parallel_size,\n            )\n        )\n        group_ranks.append(ranks)\n","sourceCodeStart":2414,"sourceCodeEnd":2450,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/distributed/parallel_state.py#L2414-L2450","documentation":"With decode context parallelism enabled, each TP group is partitioned into DCP subgroups, so tensor_model_parallel_size must be an exact multiple of decode_context_parallel_size. initialize_model_parallel enforces this with a modulo check before building tensor-model-parallel groups.","triggerScenarios":"Calling initialize_model_parallel with tensor_model_parallel_size % decode_context_parallel_size != 0, e.g. tp=4 with decode_context_parallel_size=3.","commonSituations":"Independently tuning --tp-size and --decode-context-parallel-size without keeping divisibility; template configs generated per-node GPU count producing odd pairs; changing tp for a model while forgetting to update the DCP setting.","solutions":["Pick decode_context_parallel_size that divides tensor_model_parallel_size (e.g. tp=8: dcp in {1,2,4,8})","Or raise/lower tp_size to a multiple of the desired DCP size","Add a startup assertion in your launcher: assert tp % dcp == 0"],"exampleFix":"# before\n--tp-size 4 --decode-context-parallel-size 3\n# after\n--tp-size 4 --decode-context-parallel-size 2\n# or\n--tp-size 6 --decode-context-parallel-size 3","handlingStrategy":"validation","validationCode":"assert tp_size % decode_context_parallel_size == 0, \\\n    f'tp={tp_size} must be divisible by dcp={decode_context_parallel_size}'","typeGuard":null,"tryCatchPattern":"try:\n    initialize_model_parallel(tensor_model_parallel_size=tp,\n                             decode_context_parallel_size=dcp)\nexcept RuntimeError as e:\n    if 'divisible' in str(e):\n        dcp = max(d for d in range(1, tp + 1) if tp % d == 0 and d <= dcp)\n        initialize_model_parallel(tensor_model_parallel_size=tp,\n                                 decode_context_parallel_size=dcp)\n    else:\n        raise","preventionTips":["Choose dcp as a power-of-2 divisor of tp (tp//2, tp//4, ...)","Add launcher assertion tp % dcp == 0","Update both flags together when tuning parallelism"],"tags":["parallelism","decode-context-parallel","tensor-parallel","divisibility","config-validation","sglang"],"backgroundTag":"parallel-config-validation","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}