{"record":{"id":"2d367f99b764a1f3","repo":"hiyouga/LlamaFactory","slug":"total-megatron-bridge-parallel-size-parallel-siz","errorCode":null,"errorMessage":"Total Megatron Bridge parallel size ({parallel_size}) exceeds `world_size` ({world_size}).","messagePattern":"Total Megatron Bridge parallel size \\((.+?)\\) exceeds `world_size` \\((.+?)\\)\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/llamafactory/hparams/parser.py","lineNumber":344,"sourceCode":"\ndef _configure_mca_training_args(training_args, data_args, finetuning_args) -> None:\n    \"\"\"Patch training args to avoid args checking errors and sync MCA settings.\"\"\"\n    training_args.predict_with_generate = False\n    training_args.generation_max_length = data_args.cutoff_len\n    training_args.generation_num_beams = 1\n    training_args.use_mca = True\n    finetuning_args.use_mca = True\n\n\ndef _validate_megatron_bridge_parallel_args(mb_args: MegatronBridgeArguments, world_size: int) -> None:\n    parallel_size = (\n        mb_args.tensor_model_parallel_size\n        * mb_args.pipeline_model_parallel_size\n        * mb_args.context_parallel_size\n        * mb_args.expert_model_parallel_size\n    )\n    if parallel_size > world_size:\n        raise ValueError(f\"Total Megatron Bridge parallel size ({parallel_size}) exceeds `world_size` ({world_size}).\")\n    if world_size % parallel_size != 0:\n        raise ValueError(\n            f\"Total Megatron Bridge parallel size ({parallel_size}) must divide `world_size` ({world_size}).\"\n        )\n\n\ndef _parse_train_mbridge_args(args: dict[str, Any] | list[str] | None = None) -> _TRAIN_MBRIDGE_CLS:\n    parser = HfArgumentParser(_TRAIN_MBRIDGE_ARGS)\n    allow_extra_keys = is_env_enabled(\"ALLOW_EXTRA_ARGS\")\n    model_args, data_args, training_args, finetuning_args, mb_args, generating_args = _parse_args(\n        parser, args, allow_extra_keys=allow_extra_keys\n    )\n    _configure_mbridge_training_args(training_args, data_args, finetuning_args)\n    return model_args, data_args, training_args, finetuning_args, mb_args, generating_args\n\n\ndef _configure_mbridge_training_args(training_args, data_args, finetuning_args) -> None:\n    \"\"\"Patch training args to avoid args checking errors and sync Megatron Bridge settings.\"\"\"","sourceCodeStart":326,"sourceCodeEnd":362,"githubUrl":"https://github.com/hiyouga/LlamaFactory/blob/f28afaf6355af515454dfb16c97d728307c93897/src/llamafactory/hparams/parser.py#L326-L362","documentation":"For Megatron Bridge training (USE_MEGATRON_BRIDGE=1), _validate_megatron_bridge_parallel_args computes the product of tensor_model_parallel_size * pipeline_model_parallel_size * context_parallel_size * expert_model_parallel_size. That product is the number of GPUs each model replica needs, so it cannot exceed world_size (total ranks). If it does, the requested layout cannot fit on the cluster.","triggerScenarios":"Running llamafactory-cli train with USE_MEGATRON_BRIDGE=1 and Megatron Bridge parallel args such that TP*PP*CP*EP > world_size. Example: tensor_model_parallel_size=4, pipeline_model_parallel_size=2 on world_size=4 (product 8 > 4). world_size comes from training_args.world_size (torchrun-launched process count).","commonSituations":"Copying a large-cluster Megatron config (e.g. TP=8) to a smaller dev machine with 4 GPUs; enabling expert_model_parallel_size for a MoE model on top of TP/PP without counting total ranks; launching single-process (world_size=1) while any parallel size is >1.","solutions":["Lower one or more of tensor_model_parallel_size / pipeline_model_parallel_size / context_parallel_size / expert_model_parallel_size so their product fits within world_size.","Increase world_size by launching with more processes (torchrun --nproc_per_node N or llamafactory-cli with FORCE_TORCHRUN=1 and more GPUs).","Verify with a quick calculation: assert TP*PP*CP*EP <= world_size and world_size % (TP*PP*CP*EP) == 0 before launching."],"exampleFix":"# before: 4 GPUs, TP=4 PP=2 -> product 8 > world_size 4\nUSE_MEGATRON_BRIDGE=1 llamafactory-cli train cfg.yaml\n# cfg.yaml: tensor_model_parallel_size: 4, pipeline_model_parallel_size: 2\n\n# after: TP=2 PP=2 -> product 4 == world_size 4\ntensor_model_parallel_size: 2\npipeline_model_parallel_size: 2","handlingStrategy":"validation","validationCode":"def fits_world_size(mb: dict, world_size: int) -> bool:\n    p = (mb.get(\"tensor_model_parallel_size\", 1)\n         * mb.get(\"pipeline_model_parallel_size\", 1)\n         * mb.get(\"context_parallel_size\", 1)\n         * mb.get(\"expert_model_parallel_size\", 1))\n    return p <= world_size\n\nif not fits_world_size(cfg[\"megatron_bridge\"], world_size):\n    raise SystemExit(f\"TP*PP*CP*EP must be <= {world_size}; shrink parallel sizes or add GPUs\")","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Compute TP*PP*CP*EP against the actual torchrun world_size before every launch (world_size = nproc_per_node * num_nodes).","Derive parallel sizes programmatically from GPU count instead of hardcoding them in shared configs."],"tags":["megatron","distributed","parallelism","config","validation"],"backgroundTag":null,"analyzedSha":"f28afaf6355af515454dfb16c97d728307c93897","analyzedAt":"2026-08-14T21:57:28.298Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}