{"record":{"id":"700ecfe64cea951d","repo":"hiyouga/LlamaFactory","slug":"total-megatron-bridge-parallel-size-parallel-siz-700ecf","errorCode":null,"errorMessage":"Total Megatron Bridge parallel size ({parallel_size}) must divide `world_size` ({world_size}).","messagePattern":"Total Megatron Bridge parallel size \\((.+?)\\) must divide `world_size` \\((.+?)\\)\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/llamafactory/hparams/parser.py","lineNumber":346,"sourceCode":"    \"\"\"Patch training args to avoid args checking errors and sync MCA settings.\"\"\"\n    training_args.predict_with_generate = False\n    training_args.generation_max_length = data_args.cutoff_len\n    training_args.generation_num_beams = 1\n    training_args.use_mca = True\n    finetuning_args.use_mca = True\n\n\ndef _validate_megatron_bridge_parallel_args(mb_args: MegatronBridgeArguments, world_size: int) -> None:\n    parallel_size = (\n        mb_args.tensor_model_parallel_size\n        * mb_args.pipeline_model_parallel_size\n        * mb_args.context_parallel_size\n        * mb_args.expert_model_parallel_size\n    )\n    if parallel_size > world_size:\n        raise ValueError(f\"Total Megatron Bridge parallel size ({parallel_size}) exceeds `world_size` ({world_size}).\")\n    if world_size % parallel_size != 0:\n        raise ValueError(\n            f\"Total Megatron Bridge parallel size ({parallel_size}) must divide `world_size` ({world_size}).\"\n        )\n\n\ndef _parse_train_mbridge_args(args: dict[str, Any] | list[str] | None = None) -> _TRAIN_MBRIDGE_CLS:\n    parser = HfArgumentParser(_TRAIN_MBRIDGE_ARGS)\n    allow_extra_keys = is_env_enabled(\"ALLOW_EXTRA_ARGS\")\n    model_args, data_args, training_args, finetuning_args, mb_args, generating_args = _parse_args(\n        parser, args, allow_extra_keys=allow_extra_keys\n    )\n    _configure_mbridge_training_args(training_args, data_args, finetuning_args)\n    return model_args, data_args, training_args, finetuning_args, mb_args, generating_args\n\n\ndef _configure_mbridge_training_args(training_args, data_args, finetuning_args) -> None:\n    \"\"\"Patch training args to avoid args checking errors and sync Megatron Bridge settings.\"\"\"\n    training_args.predict_with_generate = False\n    training_args.generation_max_length = data_args.cutoff_len","sourceCodeStart":328,"sourceCodeEnd":364,"githubUrl":"https://github.com/hiyouga/LlamaFactory/blob/f28afaf6355af515454dfb16c97d728307c93897/src/llamafactory/hparams/parser.py#L328-L364","documentation":"Megatron Bridge parallelism must evenly partition the cluster: besides not exceeding world_size, the product TP*PP*CP*EP must divide world_size exactly, because leftover ranks would form data-parallel replicas and any remainder makes the shard assignment impossible. The parser enforces world_size % parallel_size == 0 right after the exceeds check.","triggerScenarios":"USE_MEGATRON_BRIDGE=1 training where TP*PP*CP*EP <= world_size but does not divide it, e.g. product=3 (TP=3) on world_size=4, or product=6 on world_size=8.","commonSituations":"Odd node counts or partial-node launches (e.g. 3 of 4 GPUs visible via CUDA_VISIBLE_DEVICES), MoE configs where expert_model_parallel_size=3 is combined with TP=2 on 8 ranks, or hand-tuned pipeline sizes that don't factor the cluster size.","solutions":["Adjust the parallel sizes so TP*PP*CP*EP is an exact divisor of world_size (e.g. on 8 ranks use 2x2x1x2, 4x2, or 8).","Change world_size to a multiple of the product (add/remove GPUs or change --nproc_per_node).","Check CUDA_VISIBLE_DEVICES / SLURM allocation to confirm world_size is what you think it is, then redo the factoring."],"exampleFix":"# before: 8 ranks, TP=4 PP=1 CP=1 EP=3 -> product 12 > 8; or TP=3 -> 3 does not divide 4\ntensor_model_parallel_size: 3\n\n# after: 8 ranks, TP=4 EP=2 -> product 8 divides 8\ntensor_model_parallel_size: 4\nexpert_model_parallel_size: 2","handlingStrategy":"validation","validationCode":"def divides_world_size(mb: dict, world_size: int) -> bool:\n    p = (mb.get(\"tensor_model_parallel_size\", 1)\n         * mb.get(\"pipeline_model_parallel_size\", 1)\n         * mb.get(\"context_parallel_size\", 1)\n         * mb.get(\"expert_model_parallel_size\", 1))\n    return world_size % p == 0\n\nassert divides_world_size(cfg[\"megatron_bridge\"], world_size), \"parallel size must divide world_size\"","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Prefer power-of-two factorizations of world_size (2x4, 4x2, 8) — they always divide evenly.","Beware CUDA_VISIBLE_DEVICES trimming: verify world_size with torchrun output before assuming the full node count."],"tags":["megatron","distributed","parallelism","config","validation"],"backgroundTag":null,"analyzedSha":"f28afaf6355af515454dfb16c97d728307c93897","analyzedAt":"2026-08-14T21:57:28.298Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}