{"record":{"id":"bdfcc0da4ecb5cca","repo":"hiyouga/LlamaFactory","slug":"world-size-helper-get-world-size-must-be-div","errorCode":null,"errorMessage":"world_size ({helper.get_world_size()}) must be divisible by mp_replicate_size ({self.mp_replicate_size}).","messagePattern":"world_size \\((.+?)\\) must be divisible by mp_replicate_size \\((.+?)\\)\\.","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/llamafactory/v1/accelerator/interface.py","lineNumber":76,"sourceCode":"@dataclass\nclass DistributedStrategy:\n    \"\"\"Distributed strategy.\"\"\"\n\n    mp_replicate_size: int = 1\n    \"\"\"Model parallel replicate size, default to 1.\"\"\"\n    mp_shard_size: int | None = None\n    \"\"\"Model parallel shard size, default to world_size // mp_replicate_size.\"\"\"\n    dp_size: int | None = None\n    \"\"\"Data parallel size, default to world_size // cp_size.\"\"\"\n    cp_size: int = 1\n    \"\"\"Context parallel size, default to 1.\"\"\"\n\n    def __post_init__(self) -> None:\n        if not helper.is_distributed():\n            self.mp_shard_size = 1\n        elif self.mp_shard_size is None:\n            if helper.get_world_size() % self.mp_replicate_size != 0:\n                raise ValueError(\n                    f\"world_size ({helper.get_world_size()}) must be divisible by \"\n                    f\"mp_replicate_size ({self.mp_replicate_size}).\"\n                )\n            self.mp_shard_size = helper.get_world_size() // self.mp_replicate_size\n        elif self.mp_replicate_size * self.mp_shard_size != helper.get_world_size():\n            raise ValueError(\n                f\"mp_replicate_size * mp_shard_size must equal to world_size, \"\n                f\"got {self.mp_replicate_size} * {self.mp_shard_size} != {helper.get_world_size()}.\"\n            )\n\n        if not helper.is_distributed():\n            self.dp_size = 1\n        elif self.dp_size is None:\n            if helper.get_world_size() % self.cp_size != 0:\n                raise ValueError(\n                    f\"world_size ({helper.get_world_size()}) must be divisible by cp_size ({self.cp_size}).\"\n                )\n            self.dp_size = helper.get_world_size() // self.cp_size","sourceCodeStart":58,"sourceCodeEnd":94,"githubUrl":"https://github.com/hiyouga/LlamaFactory/blob/f28afaf6355af515454dfb16c97d728307c93897/src/llamafactory/v1/accelerator/interface.py#L58-L94","documentation":"Raised while building the v1 `DistributedConfig` in `__post_init__`: in a distributed run the world size (total ranks from `torchrun`) must be evenly divisible by `mp_replicate_size` so that `mp_shard_size = world_size // mp_replicate_size` is a whole number. This sizes the model-parallel device mesh (replicate x shard).","triggerScenarios":"Launching with `torchrun --nproc_per_node N` (or a Ray/multi-node equivalent) where N (times node count) is not a multiple of the configured `mp_replicate_size`, while `mp_shard_size` is left unset.","commonSituations":"Setting `mp_replicate_size: 3` on 4 GPUs; forgetting that a 2-node x 4-GPU job has world_size 8, not 4; copying a config tuned for 8 GPUs onto a single 4-GPU machine.","solutions":["Set `mp_replicate_size` to a divisor of your total world size (e.g. 1, 2, 4 for 8 ranks)","Or change the launcher so world size matches, e.g. `torchrun --nproc_per_node=8` when `mp_replicate_size: 4`","Double-check multi-node setups: world_size = nproc_per_node * num_nodes; recompute divisibility before launch"],"exampleFix":"# before: 4 GPUs, replicate over 3\n# torchrun --nproc_per_node=4 train.py dist.mp_replicate_size=3\n\n# after\ntorchrun --nproc_per_node=4 train.py dist.mp_replicate_size=2  # 4 % 2 == 0","handlingStrategy":"validation","validationCode":"def check_mesh(world_size: int, mp_replicate_size: int) -> None:\n    if world_size % mp_replicate_size != 0:\n        raise SystemExit(\n            f\"mp_replicate_size={mp_replicate_size} must divide world_size={world_size}\"\n        )\n\n# call with the launcher before trainer construction:\n# check_mesh(int(os.environ[\"WORLD_SIZE\"]), args.mp_replicate_size)","typeGuard":null,"tryCatchPattern":"try:\n    cfg = DistributedConfig(mp_replicate_size=rep)\nexcept ValueError as e:\n    if \"divisible by\" in str(e):\n        # pick the largest divisor of world_size <= rep\n        rep = max(d for d in range(1, rep + 1) if world_size % d == 0)\n        cfg = DistributedConfig(mp_replicate_size=rep)\n    else:\n        raise","preventionTips":["Derive parallel sizes from world_size in launch scripts instead of hardcoding","Assert divisibility in a preflight check before torchrun spawns ranks","Keep a matrix of tested (world_size, mp_replicate_size) pairs per cluster"],"tags":["distributed-training","v1","configuration","device-mesh"],"backgroundTag":null,"analyzedSha":"f28afaf6355af515454dfb16c97d728307c93897","analyzedAt":"2026-08-14T21:57:28.298Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}