hiyouga/LlamaFactory · error · ValueError

Total Megatron Bridge parallel size ({parallel_size}) exceed

Error message

Total Megatron Bridge parallel size ({parallel_size}) exceeds `world_size` ({world_size}).

What it means

For Megatron Bridge training (USE_MEGATRON_BRIDGE=1), _validate_megatron_bridge_parallel_args computes the product of tensor_model_parallel_size * pipeline_model_parallel_size * context_parallel_size * expert_model_parallel_size. That product is the number of GPUs each model replica needs, so it cannot exceed world_size (total ranks). If it does, the requested layout cannot fit on the cluster.

Source

Thrown at src/llamafactory/hparams/parser.py:344

def _configure_mca_training_args(training_args, data_args, finetuning_args) -> None:
    """Patch training args to avoid args checking errors and sync MCA settings."""
    training_args.predict_with_generate = False
    training_args.generation_max_length = data_args.cutoff_len
    training_args.generation_num_beams = 1
    training_args.use_mca = True
    finetuning_args.use_mca = True


def _validate_megatron_bridge_parallel_args(mb_args: MegatronBridgeArguments, world_size: int) -> None:
    parallel_size = (
        mb_args.tensor_model_parallel_size
        * mb_args.pipeline_model_parallel_size
        * mb_args.context_parallel_size
        * mb_args.expert_model_parallel_size
    )
    if parallel_size > world_size:
        raise ValueError(f"Total Megatron Bridge parallel size ({parallel_size}) exceeds `world_size` ({world_size}).")
    if world_size % parallel_size != 0:
        raise ValueError(
            f"Total Megatron Bridge parallel size ({parallel_size}) must divide `world_size` ({world_size})."
        )


def _parse_train_mbridge_args(args: dict[str, Any] | list[str] | None = None) -> _TRAIN_MBRIDGE_CLS:
    parser = HfArgumentParser(_TRAIN_MBRIDGE_ARGS)
    allow_extra_keys = is_env_enabled("ALLOW_EXTRA_ARGS")
    model_args, data_args, training_args, finetuning_args, mb_args, generating_args = _parse_args(
        parser, args, allow_extra_keys=allow_extra_keys
    )
    _configure_mbridge_training_args(training_args, data_args, finetuning_args)
    return model_args, data_args, training_args, finetuning_args, mb_args, generating_args


def _configure_mbridge_training_args(training_args, data_args, finetuning_args) -> None:
    """Patch training args to avoid args checking errors and sync Megatron Bridge settings."""

View on GitHub (pinned to f28afaf635)

Solutions

1. Lower one or more of tensor_model_parallel_size / pipeline_model_parallel_size / context_parallel_size / expert_model_parallel_size so their product fits within world_size.
2. Increase world_size by launching with more processes (torchrun --nproc_per_node N or llamafactory-cli with FORCE_TORCHRUN=1 and more GPUs).
3. Verify with a quick calculation: assert TP*PP*CP*EP <= world_size and world_size % (TP*PP*CP*EP) == 0 before launching.

Example fix

# before: 4 GPUs, TP=4 PP=2 -> product 8 > world_size 4
USE_MEGATRON_BRIDGE=1 llamafactory-cli train cfg.yaml
# cfg.yaml: tensor_model_parallel_size: 4, pipeline_model_parallel_size: 2

# after: TP=2 PP=2 -> product 4 == world_size 4
tensor_model_parallel_size: 2
pipeline_model_parallel_size: 2

def fits_world_size(mb: dict, world_size: int) -> bool:
    p = (mb.get("tensor_model_parallel_size", 1)
         * mb.get("pipeline_model_parallel_size", 1)
         * mb.get("context_parallel_size", 1)
         * mb.get("expert_model_parallel_size", 1))
    return p <= world_size

if not fits_world_size(cfg["megatron_bridge"], world_size):
    raise SystemExit(f"TP*PP*CP*EP must be <= {world_size}; shrink parallel sizes or add GPUs")

When it happens

Trigger: Running llamafactory-cli train with USE_MEGATRON_BRIDGE=1 and Megatron Bridge parallel args such that TP*PP*CP*EP > world_size. Example: tensor_model_parallel_size=4, pipeline_model_parallel_size=2 on world_size=4 (product 8 > 4). world_size comes from training_args.world_size (torchrun-launched process count).

Common situations: Copying a large-cluster Megatron config (e.g. TP=8) to a smaller dev machine with 4 GPUs; enabling expert_model_parallel_size for a MoE model on top of TP/PP without counting total ranks; launching single-process (world_size=1) while any parallel size is >1.

Related errors

• Total Megatron Bridge parallel size ({parallel_size}) must d
• `tensor_model_parallel_size` must be >= 1.
• `pipeline_model_parallel_size` must be >= 1.
• Megatron Bridge cannot be used together with MCA or HyperPar
• `expert_model_parallel_size` must be >= 1.

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/2d367f99b764a1f3. Report an issue: GitHub.