hiyouga/LlamaFactory · error · ValueError

`context_parallel_size` must be >= 1.

Error message

`context_parallel_size` must be >= 1.

What it means

Context parallelism shards the sequence dimension across ranks for long-context training; fewer than one context-parallel rank is invalid. MegatronBridgeArgs.__post_init__ (src/llamafactory/hparams/megatron_bridge_args.py:160) requires context_parallel_size >= 1.

Source

Thrown at src/llamafactory/hparams/megatron_bridge_args.py:160

    extra_config: Optional[str] = field(
        default=None,
        metadata={
            "help": (
                "Optional JSON string or path to a JSON file with extra Megatron Bridge model/training overrides. "
                "Dot-paths are supported (e.g. train.train_iters or checkpoint.save_interval)."
            )
        },
    )

    def __post_init__(self) -> None:
        if self.tensor_model_parallel_size < 1:
            raise ValueError("`tensor_model_parallel_size` must be >= 1.")
        if self.pipeline_model_parallel_size < 1:
            raise ValueError("`pipeline_model_parallel_size` must be >= 1.")
        if self.expert_model_parallel_size < 1:
            raise ValueError("`expert_model_parallel_size` must be >= 1.")
        if self.context_parallel_size < 1:
            raise ValueError("`context_parallel_size` must be >= 1.")
        if self.virtual_pipeline_model_parallel_size is not None and self.virtual_pipeline_model_parallel_size < 1:
            raise ValueError("`virtual_pipeline_model_parallel_size` must be >= 1 when set.")
        if self.sequence_parallel and self.tensor_model_parallel_size <= 1:
            raise ValueError("`sequence_parallel` requires `tensor_model_parallel_size` > 1.")
        if self.recompute_granularity is not None and self.recompute_granularity not in ("full", "selective"):
            raise ValueError("`recompute_granularity` must be 'full' or 'selective'.")
        if self.recompute_method is not None and self.recompute_method not in ("uniform", "block"):
            raise ValueError("`recompute_method` must be 'uniform' or 'block'.")
        if self.recompute_num_layers is not None and self.recompute_num_layers < 1:
            raise ValueError("`recompute_num_layers` must be >= 1 when set.")
        if self.moe_token_dispatcher_type is not None and self.moe_token_dispatcher_type not in (
            "allgather",
            "alltoall",
            "flex",
        ):
            raise ValueError("`moe_token_dispatcher_type` must be 'allgather', 'alltoall', or 'flex'.")

        if isinstance(self.extra_config, str):

View on GitHub (pinned to f28afaf635)

Solutions

1. Set context_parallel_size: 1 to disable context parallelism.
2. When enabling CP for long sequences, ensure sequence length and world size are divisible appropriately and pair with recompute settings.
3. Remove the key entirely to fall back to the default of 1 instead of writing 0.

Example fix

# before (yaml)
context_parallel_size: 0

# after (yaml)
context_parallel_size: 1

def check_cp_size(cp: int) -> None:
    if cp < 1:
        raise ValueError("context_parallel_size must be >= 1; use 1 to disable CP")

def is_valid_parallel_size(v) -> bool:
    return isinstance(v, int) and not isinstance(v, bool) and v >= 1

When it happens

Trigger: A Megatron Bridge config with context_parallel_size: 0 or negative — often hand-set to disable CP, or produced by a config template where the field is conditionally populated.

Common situations: Long-context (32k+) training configs where users toggle CP on/off by editing numbers, and scripts that emit 0 for 'off'.

Related errors

• Invalid API key.
• `tensor_model_parallel_size` must be >= 1.
• `pipeline_model_parallel_size` must be >= 1.
• `expert_model_parallel_size` must be >= 1.
• Total Megatron Bridge parallel size ({parallel_size}) exceed

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/5448abd7a54f13c4. Report an issue: GitHub.