hiyouga/LlamaFactory · error · ValueError

`expert_model_parallel_size` must be >= 1.

Error message

`expert_model_parallel_size` must be >= 1.

What it means

Expert (MoE) parallelism distributes Mixture-of-Experts routers/experts across ranks and must have at least one expert-parallel group. MegatronBridgeArgs.__post_init__ (src/llamafactory/hparams/megatron_bridge_args.py:158) rejects expert_model_parallel_size < 1.

Source

Thrown at src/llamafactory/hparams/megatron_bridge_args.py:158

        metadata={"help": "Whether to export the final checkpoint to Hugging Face format after training."},
    )
    extra_config: Optional[str] = field(
        default=None,
        metadata={
            "help": (
                "Optional JSON string or path to a JSON file with extra Megatron Bridge model/training overrides. "
                "Dot-paths are supported (e.g. train.train_iters or checkpoint.save_interval)."
            )
        },
    )

    def __post_init__(self) -> None:
        if self.tensor_model_parallel_size < 1:
            raise ValueError("`tensor_model_parallel_size` must be >= 1.")
        if self.pipeline_model_parallel_size < 1:
            raise ValueError("`pipeline_model_parallel_size` must be >= 1.")
        if self.expert_model_parallel_size < 1:
            raise ValueError("`expert_model_parallel_size` must be >= 1.")
        if self.context_parallel_size < 1:
            raise ValueError("`context_parallel_size` must be >= 1.")
        if self.virtual_pipeline_model_parallel_size is not None and self.virtual_pipeline_model_parallel_size < 1:
            raise ValueError("`virtual_pipeline_model_parallel_size` must be >= 1 when set.")
        if self.sequence_parallel and self.tensor_model_parallel_size <= 1:
            raise ValueError("`sequence_parallel` requires `tensor_model_parallel_size` > 1.")
        if self.recompute_granularity is not None and self.recompute_granularity not in ("full", "selective"):
            raise ValueError("`recompute_granularity` must be 'full' or 'selective'.")
        if self.recompute_method is not None and self.recompute_method not in ("uniform", "block"):
            raise ValueError("`recompute_method` must be 'uniform' or 'block'.")
        if self.recompute_num_layers is not None and self.recompute_num_layers < 1:
            raise ValueError("`recompute_num_layers` must be >= 1 when set.")
        if self.moe_token_dispatcher_type is not None and self.moe_token_dispatcher_type not in (
            "allgather",
            "alltoall",
            "flex",
        ):
            raise ValueError("`moe_token_dispatcher_type` must be 'allgather', 'alltoall', or 'flex'.")

View on GitHub (pinned to f28afaf635)

Solutions

1. Set expert_model_parallel_size: 1 when no expert sharding is wanted.
2. For MoE models, ensure the value divides the number of experts and fits the world size.
3. Prefer deleting the key (letting the dataclass default of 1 apply) over writing 0.

Example fix

# before (yaml)
expert_model_parallel_size: 0

# after (yaml)
expert_model_parallel_size: 1

def check_ep_size(ep: int) -> None:
    if ep < 1:
        raise ValueError("expert_model_parallel_size must be >= 1; use 1 to disable EP")

def is_valid_parallel_size(v) -> bool:
    return isinstance(v, int) and not isinstance(v, bool) and v >= 1

When it happens

Trigger: A Megatron Bridge config with expert_model_parallel_size: 0 or negative, typically on a MoE model (Mixtral, DeepSeek, Qwen-MoE).

Common situations: Adapting a dense-model Megatron config (where the field may be manually zeroed) to an MoE model, or generated configs defaulting missing integers to 0.

Related errors

• `tensor_model_parallel_size` must be >= 1.
• `pipeline_model_parallel_size` must be >= 1.
• Total Megatron Bridge parallel size ({parallel_size}) exceed
• Total Megatron Bridge parallel size ({parallel_size}) must d
• Invalid API key.

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/b3d180ade7ede618. Report an issue: GitHub.