hiyouga/LlamaFactory · error · NotImplementedError

RM trainer currently only supports cp_size == 1.

Error message

RM trainer currently only supports cp_size == 1.

What it means

NotImplementedError from RMTrainer.__init__ (rm_trainer.py:80) when args.cp_size > 1. The v1 reward-model trainer has no context-parallelism support; the guard fires before super().__init__() and before DDP/FSDP wrapping in _shard_model.

Source

Thrown at src/llamafactory/v1/trainers/rm_trainer.py:80

        std = 1.0 / (hidden_size * 10)
        with torch.no_grad():
            score.weight.normal_(mean=0.0, std=std)
            if score.bias is not None:
                score.bias.zero_()
        logger.info_rank0(f"Initialized score head with Gaussian (std={std:.6f}): {score.weight.shape}")


class RMTrainer(BaseTrainer):
    def __init__(
        self,
        args: TrainingArguments,
        model: HFModel,
        renderer,
        train_dataset,
        callbacks=None,
    ) -> None:
        if args.cp_size > 1:
            raise NotImplementedError("RM trainer currently only supports cp_size == 1.")

        super().__init__(args, model, renderer, train_dataset, callbacks)

    def _shard_model(self) -> None:
        if self.args.dist_config is None:
            if DistributedInterface().get_world_size(Dim.DP) > 1:
                from torch.nn.parallel import DistributedDataParallel as DDP

                device_ids = None if self.device.type == "cpu" else [self.device.index]
                self.model = DDP(self.model, device_ids=device_ids, find_unused_parameters=True)
        else:
            super()._shard_model()

    @property
    def _unwrapped_model(self):
        """Access the underlying model, unwrapping DDP/FSDP wrappers if present."""
        model = self.model
        if hasattr(model, "module"):

View on GitHub (pinned to f28afaf635)

Solutions

1. Set cp_size: 1 for RM training.
2. Fit long sequences by raising memory headroom (activation checkpointing, smaller per-device batch, sequence truncation via cutoff_len) instead of CP.
3. If CP is mandatory, use the v0 RM trainer.

Example fix

# before
training_args:
  cp_size: 4

# after
training_args:
  cp_size: 1

if args.cp_size > 1:
    raise SystemExit("v1 RMTrainer does not support cp_size > 1")

When it happens

Trigger: Running RM training (stage rm) with cp_size > 1 in the v1 training arguments.

Common situations: Long preference sequences tempt users to enable CP to shard sequence length across GPUs; a cluster template sets cp_size globally and is reused for RM runs.

Related errors

• DPO trainer currently only supports cp_size == 1.
• SGLang engine does not support `get_scores`.
• vLLM engine does not support `get_scores`.
• Other sequence parallel modes are to be implemented.
• Invalid role

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/237078ab05647993. Report an issue: GitHub.