hiyouga/LlamaFactory · error · ValueError

`sequence_parallel` requires `tensor_model_parallel_size` >

Error message

`sequence_parallel` requires `tensor_model_parallel_size` > 1.

What it means

Raised in MegatronBridgeArguments.__post_init__ (megatron_bridge_args.py:164) when sequence_parallel is enabled but tensor_model_parallel_size <= 1. Sequence parallelism shards layer-norm and dropout activations across the tensor-parallel group, so it is meaningless without actual tensor parallelism; Megatron-core asserts the same invariant. The check fires immediately during config parsing.

Source

Thrown at src/llamafactory/hparams/megatron_bridge_args.py:164

                "Optional JSON string or path to a JSON file with extra Megatron Bridge model/training overrides. "
                "Dot-paths are supported (e.g. train.train_iters or checkpoint.save_interval)."
            )
        },
    )

    def __post_init__(self) -> None:
        if self.tensor_model_parallel_size < 1:
            raise ValueError("`tensor_model_parallel_size` must be >= 1.")
        if self.pipeline_model_parallel_size < 1:
            raise ValueError("`pipeline_model_parallel_size` must be >= 1.")
        if self.expert_model_parallel_size < 1:
            raise ValueError("`expert_model_parallel_size` must be >= 1.")
        if self.context_parallel_size < 1:
            raise ValueError("`context_parallel_size` must be >= 1.")
        if self.virtual_pipeline_model_parallel_size is not None and self.virtual_pipeline_model_parallel_size < 1:
            raise ValueError("`virtual_pipeline_model_parallel_size` must be >= 1 when set.")
        if self.sequence_parallel and self.tensor_model_parallel_size <= 1:
            raise ValueError("`sequence_parallel` requires `tensor_model_parallel_size` > 1.")
        if self.recompute_granularity is not None and self.recompute_granularity not in ("full", "selective"):
            raise ValueError("`recompute_granularity` must be 'full' or 'selective'.")
        if self.recompute_method is not None and self.recompute_method not in ("uniform", "block"):
            raise ValueError("`recompute_method` must be 'uniform' or 'block'.")
        if self.recompute_num_layers is not None and self.recompute_num_layers < 1:
            raise ValueError("`recompute_num_layers` must be >= 1 when set.")
        if self.moe_token_dispatcher_type is not None and self.moe_token_dispatcher_type not in (
            "allgather",
            "alltoall",
            "flex",
        ):
            raise ValueError("`moe_token_dispatcher_type` must be 'allgather', 'alltoall', or 'flex'.")

        if isinstance(self.extra_config, str):
            config_str = self.extra_config.strip()
            if config_str.startswith("{"):
                self.extra_config = _convert_str_dict(json.loads(config_str))
            else:

View on GitHub (pinned to f28afaf635)

Solutions

1. Set tensor_model_parallel_size to at least 2 in the same config
2. If running on one GPU, set sequence_parallel: false (or remove it)
3. Audit shared YAML anchors so sequence_parallel is only applied together with TP > 1

Example fix

# before
sequence_parallel: true
tensor_model_parallel_size: 1

# after
tensor_model_parallel_size: 2
sequence_parallel: true

tp = cfg.get('tensor_model_parallel_size', 1)
if cfg.get('sequence_parallel') and tp <= 1:
    cfg['sequence_parallel'] = False
args = MegatronBridgeArguments(**cfg)

def sequence_parallel_safe(cfg: dict) -> bool:
    return not cfg.get('sequence_parallel') or cfg.get('tensor_model_parallel_size', 1) > 1

When it happens

Trigger: A config with sequence_parallel: true and tensor_model_parallel_size: 1 (the default), or a multi-node config where TP was reduced to 1 while the sequence_parallel flag was left over from a TP>1 run.

Common situations: Reusing a large-cluster config on a single GPU / single-group run; YAML includes that set sequence_parallel globally while per-experiment TP varies; downscaling a TP=8 recipe to TP=2 then to TP=1 without pruning dependent flags.

Related errors

• `virtual_pipeline_model_parallel_size` must be >= 1 when set
• `recompute_granularity` must be 'full' or 'selective'.
• `recompute_method` must be 'uniform' or 'block'.
• `recompute_num_layers` must be >= 1 when set.
• `moe_token_dispatcher_type` must be 'allgather', 'alltoall',

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/ea43f647d2569a38. Report an issue: GitHub.