hiyouga/LlamaFactory · error · ValueError

`tensor_model_parallel_size` must be >= 1.

Error message

`tensor_model_parallel_size` must be >= 1.

What it means

Megatron Bridge splits each transformer layer's weights across tensor-parallel ranks; a size below 1 is meaningless (zero ranks). MegatronBridgeArgs.__post_init__ (src/llamafactory/hparams/megatron_bridge_args.py:154) validates tensor_model_parallel_size >= 1 at argument-construction time.

Source

Thrown at src/llamafactory/hparams/megatron_bridge_args.py:154

        },
    )
    export_hf_on_finish: bool = field(
        default=False,
        metadata={"help": "Whether to export the final checkpoint to Hugging Face format after training."},
    )
    extra_config: Optional[str] = field(
        default=None,
        metadata={
            "help": (
                "Optional JSON string or path to a JSON file with extra Megatron Bridge model/training overrides. "
                "Dot-paths are supported (e.g. train.train_iters or checkpoint.save_interval)."
            )
        },
    )

    def __post_init__(self) -> None:
        if self.tensor_model_parallel_size < 1:
            raise ValueError("`tensor_model_parallel_size` must be >= 1.")
        if self.pipeline_model_parallel_size < 1:
            raise ValueError("`pipeline_model_parallel_size` must be >= 1.")
        if self.expert_model_parallel_size < 1:
            raise ValueError("`expert_model_parallel_size` must be >= 1.")
        if self.context_parallel_size < 1:
            raise ValueError("`context_parallel_size` must be >= 1.")
        if self.virtual_pipeline_model_parallel_size is not None and self.virtual_pipeline_model_parallel_size < 1:
            raise ValueError("`virtual_pipeline_model_parallel_size` must be >= 1 when set.")
        if self.sequence_parallel and self.tensor_model_parallel_size <= 1:
            raise ValueError("`sequence_parallel` requires `tensor_model_parallel_size` > 1.")
        if self.recompute_granularity is not None and self.recompute_granularity not in ("full", "selective"):
            raise ValueError("`recompute_granularity` must be 'full' or 'selective'.")
        if self.recompute_method is not None and self.recompute_method not in ("uniform", "block"):
            raise ValueError("`recompute_method` must be 'uniform' or 'block'.")
        if self.recompute_num_layers is not None and self.recompute_num_layers < 1:
            raise ValueError("`recompute_num_layers` must be >= 1 when set.")
        if self.moe_token_dispatcher_type is not None and self.moe_token_dispatcher_type not in (
            "allgather",

View on GitHub (pinned to f28afaf635)

Solutions

1. Set tensor_model_parallel_size: 1 to run without tensor parallelism.
2. If you want real TP, ensure the value divides both the attention heads and the world size (e.g. 2 or 4 GPUs).
3. Check templating/env substitution in your config generator so unset values default to 1, not 0.

Example fix

# before (yaml)
tensor_model_parallel_size: 0

# after (yaml)
tensor_model_parallel_size: 1

def check_tp_size(tp: int) -> None:
    if tp < 1:
        raise ValueError("tensor_model_parallel_size must be >= 1; use 1 to disable TP")

def is_valid_parallel_size(v) -> bool:
    return isinstance(v, int) and not isinstance(v, bool) and v >= 1

When it happens

Trigger: A Megatron Bridge training config with tensor_model_parallel_size: 0 (or negative), or the key omitted from a partial override such that an invalid value lands in the dataclass.

Common situations: Hand-editing a megatron YAML and setting 0 to 'disable' TP (the correct way is to leave it at 1); templating scripts that substitute an unset variable as 0.

Related errors

• `pipeline_model_parallel_size` must be >= 1.
• Total Megatron Bridge parallel size ({parallel_size}) exceed
• Total Megatron Bridge parallel size ({parallel_size}) must d
• `expert_model_parallel_size` must be >= 1.
• Megatron Bridge cannot be used together with MCA or HyperPar

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/6c7b5e9839e47a99. Report an issue: GitHub.