{"record":{"id":"ea43f647d2569a38","repo":"hiyouga/LlamaFactory","slug":"sequence-parallel-requires-tensor-model-paralle","errorCode":null,"errorMessage":"`sequence_parallel` requires `tensor_model_parallel_size` > 1.","messagePattern":"`sequence_parallel` requires `tensor_model_parallel_size` > 1\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/llamafactory/hparams/megatron_bridge_args.py","lineNumber":164,"sourceCode":"                \"Optional JSON string or path to a JSON file with extra Megatron Bridge model/training overrides. \"\n                \"Dot-paths are supported (e.g. train.train_iters or checkpoint.save_interval).\"\n            )\n        },\n    )\n\n    def __post_init__(self) -> None:\n        if self.tensor_model_parallel_size < 1:\n            raise ValueError(\"`tensor_model_parallel_size` must be >= 1.\")\n        if self.pipeline_model_parallel_size < 1:\n            raise ValueError(\"`pipeline_model_parallel_size` must be >= 1.\")\n        if self.expert_model_parallel_size < 1:\n            raise ValueError(\"`expert_model_parallel_size` must be >= 1.\")\n        if self.context_parallel_size < 1:\n            raise ValueError(\"`context_parallel_size` must be >= 1.\")\n        if self.virtual_pipeline_model_parallel_size is not None and self.virtual_pipeline_model_parallel_size < 1:\n            raise ValueError(\"`virtual_pipeline_model_parallel_size` must be >= 1 when set.\")\n        if self.sequence_parallel and self.tensor_model_parallel_size <= 1:\n            raise ValueError(\"`sequence_parallel` requires `tensor_model_parallel_size` > 1.\")\n        if self.recompute_granularity is not None and self.recompute_granularity not in (\"full\", \"selective\"):\n            raise ValueError(\"`recompute_granularity` must be 'full' or 'selective'.\")\n        if self.recompute_method is not None and self.recompute_method not in (\"uniform\", \"block\"):\n            raise ValueError(\"`recompute_method` must be 'uniform' or 'block'.\")\n        if self.recompute_num_layers is not None and self.recompute_num_layers < 1:\n            raise ValueError(\"`recompute_num_layers` must be >= 1 when set.\")\n        if self.moe_token_dispatcher_type is not None and self.moe_token_dispatcher_type not in (\n            \"allgather\",\n            \"alltoall\",\n            \"flex\",\n        ):\n            raise ValueError(\"`moe_token_dispatcher_type` must be 'allgather', 'alltoall', or 'flex'.\")\n\n        if isinstance(self.extra_config, str):\n            config_str = self.extra_config.strip()\n            if config_str.startswith(\"{\"):\n                self.extra_config = _convert_str_dict(json.loads(config_str))\n            else:","sourceCodeStart":146,"sourceCodeEnd":182,"githubUrl":"https://github.com/hiyouga/LlamaFactory/blob/f28afaf6355af515454dfb16c97d728307c93897/src/llamafactory/hparams/megatron_bridge_args.py#L146-L182","documentation":"Raised in MegatronBridgeArguments.__post_init__ (megatron_bridge_args.py:164) when sequence_parallel is enabled but tensor_model_parallel_size <= 1. Sequence parallelism shards layer-norm and dropout activations across the tensor-parallel group, so it is meaningless without actual tensor parallelism; Megatron-core asserts the same invariant. The check fires immediately during config parsing.","triggerScenarios":"A config with sequence_parallel: true and tensor_model_parallel_size: 1 (the default), or a multi-node config where TP was reduced to 1 while the sequence_parallel flag was left over from a TP>1 run.","commonSituations":"Reusing a large-cluster config on a single GPU / single-group run; YAML includes that set sequence_parallel globally while per-experiment TP varies; downscaling a TP=8 recipe to TP=2 then to TP=1 without pruning dependent flags.","solutions":["Set tensor_model_parallel_size to at least 2 in the same config","If running on one GPU, set sequence_parallel: false (or remove it)","Audit shared YAML anchors so sequence_parallel is only applied together with TP > 1"],"exampleFix":"# before\nsequence_parallel: true\ntensor_model_parallel_size: 1\n\n# after\ntensor_model_parallel_size: 2\nsequence_parallel: true","handlingStrategy":"validation","validationCode":"tp = cfg.get('tensor_model_parallel_size', 1)\nif cfg.get('sequence_parallel') and tp <= 1:\n    cfg['sequence_parallel'] = False\nargs = MegatronBridgeArguments(**cfg)","typeGuard":"def sequence_parallel_safe(cfg: dict) -> bool:\n    return not cfg.get('sequence_parallel') or cfg.get('tensor_model_parallel_size', 1) > 1","tryCatchPattern":null,"preventionTips":["Couple sequence_parallel with tensor_model_parallel_size in one YAML block","When downscaling TP, grep the config for dependent flags (sequence_parallel, context_parallel)"],"tags":["megatron","sequence-parallel","config-validation","hparams"],"backgroundTag":null,"analyzedSha":"f28afaf6355af515454dfb16c97d728307c93897","analyzedAt":"2026-08-14T21:57:28.298Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}