{"record":{"id":"6c7b5e9839e47a99","repo":"hiyouga/LlamaFactory","slug":"tensor-model-parallel-size-must-be-1","errorCode":null,"errorMessage":"`tensor_model_parallel_size` must be >= 1.","messagePattern":"`tensor_model_parallel_size` must be >= 1\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/llamafactory/hparams/megatron_bridge_args.py","lineNumber":154,"sourceCode":"        },\n    )\n    export_hf_on_finish: bool = field(\n        default=False,\n        metadata={\"help\": \"Whether to export the final checkpoint to Hugging Face format after training.\"},\n    )\n    extra_config: Optional[str] = field(\n        default=None,\n        metadata={\n            \"help\": (\n                \"Optional JSON string or path to a JSON file with extra Megatron Bridge model/training overrides. \"\n                \"Dot-paths are supported (e.g. train.train_iters or checkpoint.save_interval).\"\n            )\n        },\n    )\n\n    def __post_init__(self) -> None:\n        if self.tensor_model_parallel_size < 1:\n            raise ValueError(\"`tensor_model_parallel_size` must be >= 1.\")\n        if self.pipeline_model_parallel_size < 1:\n            raise ValueError(\"`pipeline_model_parallel_size` must be >= 1.\")\n        if self.expert_model_parallel_size < 1:\n            raise ValueError(\"`expert_model_parallel_size` must be >= 1.\")\n        if self.context_parallel_size < 1:\n            raise ValueError(\"`context_parallel_size` must be >= 1.\")\n        if self.virtual_pipeline_model_parallel_size is not None and self.virtual_pipeline_model_parallel_size < 1:\n            raise ValueError(\"`virtual_pipeline_model_parallel_size` must be >= 1 when set.\")\n        if self.sequence_parallel and self.tensor_model_parallel_size <= 1:\n            raise ValueError(\"`sequence_parallel` requires `tensor_model_parallel_size` > 1.\")\n        if self.recompute_granularity is not None and self.recompute_granularity not in (\"full\", \"selective\"):\n            raise ValueError(\"`recompute_granularity` must be 'full' or 'selective'.\")\n        if self.recompute_method is not None and self.recompute_method not in (\"uniform\", \"block\"):\n            raise ValueError(\"`recompute_method` must be 'uniform' or 'block'.\")\n        if self.recompute_num_layers is not None and self.recompute_num_layers < 1:\n            raise ValueError(\"`recompute_num_layers` must be >= 1 when set.\")\n        if self.moe_token_dispatcher_type is not None and self.moe_token_dispatcher_type not in (\n            \"allgather\",","sourceCodeStart":136,"sourceCodeEnd":172,"githubUrl":"https://github.com/hiyouga/LlamaFactory/blob/f28afaf6355af515454dfb16c97d728307c93897/src/llamafactory/hparams/megatron_bridge_args.py#L136-L172","documentation":"Megatron Bridge splits each transformer layer's weights across tensor-parallel ranks; a size below 1 is meaningless (zero ranks). MegatronBridgeArgs.__post_init__ (src/llamafactory/hparams/megatron_bridge_args.py:154) validates tensor_model_parallel_size >= 1 at argument-construction time.","triggerScenarios":"A Megatron Bridge training config with tensor_model_parallel_size: 0 (or negative), or the key omitted from a partial override such that an invalid value lands in the dataclass.","commonSituations":"Hand-editing a megatron YAML and setting 0 to 'disable' TP (the correct way is to leave it at 1); templating scripts that substitute an unset variable as 0.","solutions":["Set tensor_model_parallel_size: 1 to run without tensor parallelism.","If you want real TP, ensure the value divides both the attention heads and the world size (e.g. 2 or 4 GPUs).","Check templating/env substitution in your config generator so unset values default to 1, not 0."],"exampleFix":"# before (yaml)\ntensor_model_parallel_size: 0\n\n# after (yaml)\ntensor_model_parallel_size: 1","handlingStrategy":"validation","validationCode":"def check_tp_size(tp: int) -> None:\n    if tp < 1:\n        raise ValueError(\"tensor_model_parallel_size must be >= 1; use 1 to disable TP\")","typeGuard":"def is_valid_parallel_size(v) -> bool:\n    return isinstance(v, int) and not isinstance(v, bool) and v >= 1","tryCatchPattern":null,"preventionTips":["Express 'disabled' as 1, never 0, for all Megatron parallel sizes.","Validate TP*PP*CP <= world_size in a pre-launch script before torchrun starts."],"tags":["config","megatron","parallelism","distributed"],"backgroundTag":null,"analyzedSha":"f28afaf6355af515454dfb16c97d728307c93897","analyzedAt":"2026-08-14T21:57:28.298Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}