hiyouga/LlamaFactory · error · ValueError

`virtual_pipeline_model_parallel_size` must be >= 1 when set

Error message

`virtual_pipeline_model_parallel_size` must be >= 1 when set.

What it means

Raised in MegatronBridgeArguments.__post_init__ (src/llamafactory/hparams/megatron_bridge_args.py:162) when virtual_pipeline_model_parallel_size is explicitly set (not None) to a value below 1. Virtual pipeline (interleaved) parallelism splits each pipeline stage into multiple model chunks, so the count must be a positive integer. The check runs at argument-construction time, before any training starts.

Source

Thrown at src/llamafactory/hparams/megatron_bridge_args.py:162

        metadata={
            "help": (
                "Optional JSON string or path to a JSON file with extra Megatron Bridge model/training overrides. "
                "Dot-paths are supported (e.g. train.train_iters or checkpoint.save_interval)."
            )
        },
    )

    def __post_init__(self) -> None:
        if self.tensor_model_parallel_size < 1:
            raise ValueError("`tensor_model_parallel_size` must be >= 1.")
        if self.pipeline_model_parallel_size < 1:
            raise ValueError("`pipeline_model_parallel_size` must be >= 1.")
        if self.expert_model_parallel_size < 1:
            raise ValueError("`expert_model_parallel_size` must be >= 1.")
        if self.context_parallel_size < 1:
            raise ValueError("`context_parallel_size` must be >= 1.")
        if self.virtual_pipeline_model_parallel_size is not None and self.virtual_pipeline_model_parallel_size < 1:
            raise ValueError("`virtual_pipeline_model_parallel_size` must be >= 1 when set.")
        if self.sequence_parallel and self.tensor_model_parallel_size <= 1:
            raise ValueError("`sequence_parallel` requires `tensor_model_parallel_size` > 1.")
        if self.recompute_granularity is not None and self.recompute_granularity not in ("full", "selective"):
            raise ValueError("`recompute_granularity` must be 'full' or 'selective'.")
        if self.recompute_method is not None and self.recompute_method not in ("uniform", "block"):
            raise ValueError("`recompute_method` must be 'uniform' or 'block'.")
        if self.recompute_num_layers is not None and self.recompute_num_layers < 1:
            raise ValueError("`recompute_num_layers` must be >= 1 when set.")
        if self.moe_token_dispatcher_type is not None and self.moe_token_dispatcher_type not in (
            "allgather",
            "alltoall",
            "flex",
        ):
            raise ValueError("`moe_token_dispatcher_type` must be 'allgather', 'alltoall', or 'flex'.")

        if isinstance(self.extra_config, str):
            config_str = self.extra_config.strip()
            if config_str.startswith("{"):

View on GitHub (pinned to f28afaf635)

Solutions

1. Remove virtual_pipeline_model_parallel_size from the YAML to disable it (None is the disabled state)
2. Set it to a positive integer (typically 2-8, and it must divide the number of transformer layers per pipeline stage)
3. If templating configs, emit the key only when the value is defined and > 0

Example fix

# before
megatron_bridge_args:
  virtual_pipeline_model_parallel_size: 0

# after
megatron_bridge_args:
  virtual_pipeline_model_parallel_size: 2
# or omit the key entirely

# before building args
vpp = cfg.get('virtual_pipeline_model_parallel_size')
if vpp is not None and vpp < 1:
    del cfg['virtual_pipeline_model_parallel_size']  # or raise with your own message
args = MegatronBridgeArguments(**cfg)

def valid_vpp(v: int | None) -> bool:
    return v is None or (isinstance(v, int) and v >= 1)

When it happens

Trigger: Setting virtual_pipeline_model_parallel_size: 0 (e.g. when templating a YAML config and the variable interpolates to 0) or a negative value in the megatron section of the training config. Leaving it unset (None) does not raise.

Common situations: Auto-generated configs that default numeric fields to 0 instead of omitting them; copying a Megatron-LM reference config into LlamaFactory's schema; users assuming 0 means 'disabled' like other flags.

Related errors

• `sequence_parallel` requires `tensor_model_parallel_size` >
• `tensor_model_parallel_size` must be >= 1.
• `pipeline_model_parallel_size` must be >= 1.
• `expert_model_parallel_size` must be >= 1.
• `recompute_granularity` must be 'full' or 'selective'.

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/6c69ef53b116b76a. Report an issue: GitHub.