{"record":{"id":"b3d180ade7ede618","repo":"hiyouga/LlamaFactory","slug":"expert-model-parallel-size-must-be-1","errorCode":null,"errorMessage":"`expert_model_parallel_size` must be >= 1.","messagePattern":"`expert_model_parallel_size` must be >= 1\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/llamafactory/hparams/megatron_bridge_args.py","lineNumber":158,"sourceCode":"        metadata={\"help\": \"Whether to export the final checkpoint to Hugging Face format after training.\"},\n    )\n    extra_config: Optional[str] = field(\n        default=None,\n        metadata={\n            \"help\": (\n                \"Optional JSON string or path to a JSON file with extra Megatron Bridge model/training overrides. \"\n                \"Dot-paths are supported (e.g. train.train_iters or checkpoint.save_interval).\"\n            )\n        },\n    )\n\n    def __post_init__(self) -> None:\n        if self.tensor_model_parallel_size < 1:\n            raise ValueError(\"`tensor_model_parallel_size` must be >= 1.\")\n        if self.pipeline_model_parallel_size < 1:\n            raise ValueError(\"`pipeline_model_parallel_size` must be >= 1.\")\n        if self.expert_model_parallel_size < 1:\n            raise ValueError(\"`expert_model_parallel_size` must be >= 1.\")\n        if self.context_parallel_size < 1:\n            raise ValueError(\"`context_parallel_size` must be >= 1.\")\n        if self.virtual_pipeline_model_parallel_size is not None and self.virtual_pipeline_model_parallel_size < 1:\n            raise ValueError(\"`virtual_pipeline_model_parallel_size` must be >= 1 when set.\")\n        if self.sequence_parallel and self.tensor_model_parallel_size <= 1:\n            raise ValueError(\"`sequence_parallel` requires `tensor_model_parallel_size` > 1.\")\n        if self.recompute_granularity is not None and self.recompute_granularity not in (\"full\", \"selective\"):\n            raise ValueError(\"`recompute_granularity` must be 'full' or 'selective'.\")\n        if self.recompute_method is not None and self.recompute_method not in (\"uniform\", \"block\"):\n            raise ValueError(\"`recompute_method` must be 'uniform' or 'block'.\")\n        if self.recompute_num_layers is not None and self.recompute_num_layers < 1:\n            raise ValueError(\"`recompute_num_layers` must be >= 1 when set.\")\n        if self.moe_token_dispatcher_type is not None and self.moe_token_dispatcher_type not in (\n            \"allgather\",\n            \"alltoall\",\n            \"flex\",\n        ):\n            raise ValueError(\"`moe_token_dispatcher_type` must be 'allgather', 'alltoall', or 'flex'.\")","sourceCodeStart":140,"sourceCodeEnd":176,"githubUrl":"https://github.com/hiyouga/LlamaFactory/blob/f28afaf6355af515454dfb16c97d728307c93897/src/llamafactory/hparams/megatron_bridge_args.py#L140-L176","documentation":"Expert (MoE) parallelism distributes Mixture-of-Experts routers/experts across ranks and must have at least one expert-parallel group. MegatronBridgeArgs.__post_init__ (src/llamafactory/hparams/megatron_bridge_args.py:158) rejects expert_model_parallel_size < 1.","triggerScenarios":"A Megatron Bridge config with expert_model_parallel_size: 0 or negative, typically on a MoE model (Mixtral, DeepSeek, Qwen-MoE).","commonSituations":"Adapting a dense-model Megatron config (where the field may be manually zeroed) to an MoE model, or generated configs defaulting missing integers to 0.","solutions":["Set expert_model_parallel_size: 1 when no expert sharding is wanted.","For MoE models, ensure the value divides the number of experts and fits the world size.","Prefer deleting the key (letting the dataclass default of 1 apply) over writing 0."],"exampleFix":"# before (yaml)\nexpert_model_parallel_size: 0\n\n# after (yaml)\nexpert_model_parallel_size: 1","handlingStrategy":"validation","validationCode":"def check_ep_size(ep: int) -> None:\n    if ep < 1:\n        raise ValueError(\"expert_model_parallel_size must be >= 1; use 1 to disable EP\")","typeGuard":"def is_valid_parallel_size(v) -> bool:\n    return isinstance(v, int) and not isinstance(v, bool) and v >= 1","tryCatchPattern":null,"preventionTips":["For dense models just omit the field; the default of 1 is correct.","For MoE models, choose EP as a divisor of the expert count."],"tags":["config","megatron","moe","parallelism"],"backgroundTag":null,"analyzedSha":"f28afaf6355af515454dfb16c97d728307c93897","analyzedAt":"2026-08-14T21:57:28.298Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}