hiyouga/LlamaFactory · error · NotImplementedError

DPO trainer currently only supports cp_size == 1.

Error message

DPO trainer currently only supports cp_size == 1.

What it means

NotImplementedError from DPOTrainer.__init__ (dpo_trainer.py:98) when args.cp_size > 1. Context parallelism is not yet implemented for the v1 DPO trainer; the check runs before super().__init__ so it fails fast, before model sharding or dataloader creation.

Source

Thrown at src/llamafactory/v1/trainers/dpo_trainer.py:98

    sample_keys = sorted(sample.keys())
    raise ValueError(
        "DPO training requires pair-format samples containing chosen/rejected responses. "
        f"First sample from dataset '{dataset_name}' has keys: {sample_keys}. "
        "Please use pair data (e.g. a dataset with chosen_messages/rejected_messages)."
    )


class DPOTrainer(BaseTrainer):
    def __init__(
        self,
        args: TrainingArguments,
        model: HFModel,
        renderer,
        train_dataset,
        callbacks=None,
    ) -> None:
        if args.cp_size > 1:
            raise NotImplementedError("DPO trainer currently only supports cp_size == 1.")

        self.pref_loss = args.pref_loss
        self.pref_beta = args.pref_beta
        self.pref_ftx = args.pref_ftx
        self.simpo_gamma = args.simpo_gamma
        self.ld_alpha = args.ld_alpha
        self.dpo_label_smoothing = args.dpo_label_smoothing

        # ref_model must be created AFTER super().__init__() because FSDP2 with
        # init_on_meta materialises the model during _shard_model().  We defer
        # creation to _init_ref_model() below.
        self.ref_model = None

        super().__init__(args, model, renderer, train_dataset, callbacks)

        if self.pref_loss == "sigmoid":
            self._init_ref_model()

View on GitHub (pinned to f28afaf635)

Solutions

1. Set cp_size: 1 in the training config for DPO runs.
2. To fit long sequences, reduce them via cutoff_len, or increase per-GPU memory techniques (activation checkpointing, zero-stage) instead of CP.
3. If CP is a hard requirement, fall back to the v0 DPO implementation or implement CP support in the v1 trainer.

Example fix

# before
training_args:
  cp_size: 2

# after
training_args:
  cp_size: 1

if args.cp_size > 1:
    raise SystemExit("v1 DPOTrainer does not support cp_size > 1; set cp_size=1 or use v0")

When it happens

Trigger: Launching DPO training with cp_size set to a value greater than 1 in the v1 training arguments (e.g. to fit long sequences across GPUs).

Common situations: Users with very long preference sequences try to reuse a cp_size>1 SFT config for DPO; or a shared yaml defaults cp_size > 1 and is reused for a DPO run.

Related errors

• RM trainer currently only supports cp_size == 1.
• Other sequence parallel modes are to be implemented.
• `num_layers` {num_layers} should be divisible by `num_expand
• bf16 and fp16 cannot be both True.
• Invalid role

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/0a9ebfc1fba378a2. Report an issue: GitHub.