hiyouga/LlamaFactory · error · ValueError

Total Megatron Bridge parallel size ({parallel_size}) must d

Error message

Total Megatron Bridge parallel size ({parallel_size}) must divide `world_size` ({world_size}).

What it means

Megatron Bridge parallelism must evenly partition the cluster: besides not exceeding world_size, the product TP*PP*CP*EP must divide world_size exactly, because leftover ranks would form data-parallel replicas and any remainder makes the shard assignment impossible. The parser enforces world_size % parallel_size == 0 right after the exceeds check.

Source

Thrown at src/llamafactory/hparams/parser.py:346

    """Patch training args to avoid args checking errors and sync MCA settings."""
    training_args.predict_with_generate = False
    training_args.generation_max_length = data_args.cutoff_len
    training_args.generation_num_beams = 1
    training_args.use_mca = True
    finetuning_args.use_mca = True


def _validate_megatron_bridge_parallel_args(mb_args: MegatronBridgeArguments, world_size: int) -> None:
    parallel_size = (
        mb_args.tensor_model_parallel_size
        * mb_args.pipeline_model_parallel_size
        * mb_args.context_parallel_size
        * mb_args.expert_model_parallel_size
    )
    if parallel_size > world_size:
        raise ValueError(f"Total Megatron Bridge parallel size ({parallel_size}) exceeds `world_size` ({world_size}).")
    if world_size % parallel_size != 0:
        raise ValueError(
            f"Total Megatron Bridge parallel size ({parallel_size}) must divide `world_size` ({world_size})."
        )


def _parse_train_mbridge_args(args: dict[str, Any] | list[str] | None = None) -> _TRAIN_MBRIDGE_CLS:
    parser = HfArgumentParser(_TRAIN_MBRIDGE_ARGS)
    allow_extra_keys = is_env_enabled("ALLOW_EXTRA_ARGS")
    model_args, data_args, training_args, finetuning_args, mb_args, generating_args = _parse_args(
        parser, args, allow_extra_keys=allow_extra_keys
    )
    _configure_mbridge_training_args(training_args, data_args, finetuning_args)
    return model_args, data_args, training_args, finetuning_args, mb_args, generating_args


def _configure_mbridge_training_args(training_args, data_args, finetuning_args) -> None:
    """Patch training args to avoid args checking errors and sync Megatron Bridge settings."""
    training_args.predict_with_generate = False
    training_args.generation_max_length = data_args.cutoff_len

View on GitHub (pinned to f28afaf635)

Solutions

1. Adjust the parallel sizes so TP*PP*CP*EP is an exact divisor of world_size (e.g. on 8 ranks use 2x2x1x2, 4x2, or 8).
2. Change world_size to a multiple of the product (add/remove GPUs or change --nproc_per_node).
3. Check CUDA_VISIBLE_DEVICES / SLURM allocation to confirm world_size is what you think it is, then redo the factoring.

Example fix

# before: 8 ranks, TP=4 PP=1 CP=1 EP=3 -> product 12 > 8; or TP=3 -> 3 does not divide 4
tensor_model_parallel_size: 3

# after: 8 ranks, TP=4 EP=2 -> product 8 divides 8
tensor_model_parallel_size: 4
expert_model_parallel_size: 2

def divides_world_size(mb: dict, world_size: int) -> bool:
    p = (mb.get("tensor_model_parallel_size", 1)
         * mb.get("pipeline_model_parallel_size", 1)
         * mb.get("context_parallel_size", 1)
         * mb.get("expert_model_parallel_size", 1))
    return world_size % p == 0

assert divides_world_size(cfg["megatron_bridge"], world_size), "parallel size must divide world_size"

When it happens

Trigger: USE_MEGATRON_BRIDGE=1 training where TP*PP*CP*EP <= world_size but does not divide it, e.g. product=3 (TP=3) on world_size=4, or product=6 on world_size=8.

Common situations: Odd node counts or partial-node launches (e.g. 3 of 4 GPUs visible via CUDA_VISIBLE_DEVICES), MoE configs where expert_model_parallel_size=3 is combined with TP=2 on 8 ranks, or hand-tuned pipeline sizes that don't factor the cluster size.

Related errors

• Total Megatron Bridge parallel size ({parallel_size}) exceed
• `tensor_model_parallel_size` must be >= 1.
• `pipeline_model_parallel_size` must be >= 1.
• Megatron Bridge cannot be used together with MCA or HyperPar
• `expert_model_parallel_size` must be >= 1.

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/700ecfe64cea951d. Report an issue: GitHub.