hiyouga/LlamaFactory · error · ValueError

`pipeline_model_parallel_size` must be >= 1.

Error message

`pipeline_model_parallel_size` must be >= 1.

What it means

Pipeline parallelism splits transformer layers across stages; at least one stage per rank group is required, so pipeline_model_parallel_size < 1 is invalid. MegatronBridgeArgs.__post_init__ (src/llamafactory/hparams/megatron_bridge_args.py:156) enforces >= 1.

Source

Thrown at src/llamafactory/hparams/megatron_bridge_args.py:156

    export_hf_on_finish: bool = field(
        default=False,
        metadata={"help": "Whether to export the final checkpoint to Hugging Face format after training."},
    )
    extra_config: Optional[str] = field(
        default=None,
        metadata={
            "help": (
                "Optional JSON string or path to a JSON file with extra Megatron Bridge model/training overrides. "
                "Dot-paths are supported (e.g. train.train_iters or checkpoint.save_interval)."
            )
        },
    )

    def __post_init__(self) -> None:
        if self.tensor_model_parallel_size < 1:
            raise ValueError("`tensor_model_parallel_size` must be >= 1.")
        if self.pipeline_model_parallel_size < 1:
            raise ValueError("`pipeline_model_parallel_size` must be >= 1.")
        if self.expert_model_parallel_size < 1:
            raise ValueError("`expert_model_parallel_size` must be >= 1.")
        if self.context_parallel_size < 1:
            raise ValueError("`context_parallel_size` must be >= 1.")
        if self.virtual_pipeline_model_parallel_size is not None and self.virtual_pipeline_model_parallel_size < 1:
            raise ValueError("`virtual_pipeline_model_parallel_size` must be >= 1 when set.")
        if self.sequence_parallel and self.tensor_model_parallel_size <= 1:
            raise ValueError("`sequence_parallel` requires `tensor_model_parallel_size` > 1.")
        if self.recompute_granularity is not None and self.recompute_granularity not in ("full", "selective"):
            raise ValueError("`recompute_granularity` must be 'full' or 'selective'.")
        if self.recompute_method is not None and self.recompute_method not in ("uniform", "block"):
            raise ValueError("`recompute_method` must be 'uniform' or 'block'.")
        if self.recompute_num_layers is not None and self.recompute_num_layers < 1:
            raise ValueError("`recompute_num_layers` must be >= 1 when set.")
        if self.moe_token_dispatcher_type is not None and self.moe_token_dispatcher_type not in (
            "allgather",
            "alltoall",
            "flex",

View on GitHub (pinned to f28afaf635)

Solutions

1. Set pipeline_model_parallel_size: 1 for no pipeline parallelism.
2. For real pipeline parallelism, verify world_size = TP * PP * CP * DP and that the model has enough layers to divide.
3. Audit extra_config JSON overrides, since they can also feed invalid values into these fields.

Example fix

# before (yaml)
pipeline_model_parallel_size: 0

# after (yaml)
pipeline_model_parallel_size: 1

def check_pp_size(pp: int) -> None:
    if pp < 1:
        raise ValueError("pipeline_model_parallel_size must be >= 1; use 1 to disable PP")

def is_valid_parallel_size(v) -> bool:
    return isinstance(v, int) and not isinstance(v, bool) and v >= 1

When it happens

Trigger: A Megatron Bridge config with pipeline_model_parallel_size: 0 or a negative number (including values injected via extra_config overrides that bypass the YAML default).

Common situations: Editing a multi-node Megatron config and zeroing PP to 'turn it off' (should stay 1), or variable substitution producing 0/empty in generated configs.

Related errors

• `tensor_model_parallel_size` must be >= 1.
• Total Megatron Bridge parallel size ({parallel_size}) exceed
• Total Megatron Bridge parallel size ({parallel_size}) must d
• `expert_model_parallel_size` must be >= 1.
• Megatron Bridge cannot be used together with MCA or HyperPar

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/4441f3c2dfc69291. Report an issue: GitHub.