hiyouga/LlamaFactory · error · ValueError

DPO training requires pair-format samples containing chosen/

Error message

DPO training requires pair-format samples containing chosen/rejected responses. First sample from dataset '{dataset_name}' has keys: {sample_keys}. Please use pair data (e.g. a dataset with chosen_messages/rejected_messages).

What it means

ValueError from _validate_dpo_dataset_format (dpo_trainer.py:81) when the first sample lacks both 'chosen_messages' and 'rejected_messages' keys. DPO needs preference pairs; the message includes the offending dataset name and the sample's actual sorted keys so the mismatch is immediately visible. It is a format check on sample[0], so a single malformed leading row also triggers it.

Source

Thrown at src/llamafactory/v1/trainers/dpo_trainer.py:81

    rejected_logratios = policy_rejected_logps - ref_rejected_logps
    logits = chosen_logratios - rejected_logratios
    return -F.logsigmoid(beta * logits) * (1 - label_smoothing) - F.logsigmoid(-beta * logits) * label_smoothing


def _validate_dpo_dataset_format(train_dataset: DataEngine, dataset_path: str) -> None:
    if train_dataset.streaming:
        return

    if len(train_dataset) == 0:
        raise ValueError(f"DPO training dataset is empty: {dataset_path}")

    sample = train_dataset[0]
    if "chosen_messages" in sample and "rejected_messages" in sample:
        return

    dataset_name = sample.get("_dataset_name", "unknown")
    sample_keys = sorted(sample.keys())
    raise ValueError(
        "DPO training requires pair-format samples containing chosen/rejected responses. "
        f"First sample from dataset '{dataset_name}' has keys: {sample_keys}. "
        "Please use pair data (e.g. a dataset with chosen_messages/rejected_messages)."
    )


class DPOTrainer(BaseTrainer):
    def __init__(
        self,
        args: TrainingArguments,
        model: HFModel,
        renderer,
        train_dataset,
        callbacks=None,
    ) -> None:
        if args.cp_size > 1:
            raise NotImplementedError("DPO trainer currently only supports cp_size == 1.")

View on GitHub (pinned to f28afaf635)

Solutions

1. Switch to a pair dataset that yields chosen_messages/rejected_messages (e.g. the ranking subset used in LlamaFactory examples).
2. If the raw data has chosen/rejected text columns, apply a pair converter so samples are emitted with chosen_messages/rejected_messages.
3. Print train_dataset[0] and compare its keys against the required pair keys to find naming mismatches.
4. Verify the dataset_info.json 'format'/'columns' mapping produces the *_messages fields.

Example fix

# before: SFT dataset
{
  "messages": [{"role": "user", ...}, {"role": "assistant", ...}]
}

# after: DPO pair dataset
{
  "chosen_messages": [{"role": "user", ...}, {"role": "assistant", ...}],
  "rejected_messages": [{"role": "user", ...}, {"role": "assistant", ...}]
}

sample = train_dataset[0]
assert "chosen_messages" in sample and "rejected_messages" in sample, f"not pair data; keys={sorted(sample)}"

def is_pair_sample(sample: dict) -> bool:
    return "chosen_messages" in sample and "rejected_messages" in sample

When it happens

Trigger: Feeding an SFT-style dataset (only 'messages' or instruction/output columns) to DPOTrainer; using a dataset whose pair columns are named differently and were not converted to chosen_messages/rejected_messages by the data converter.

Common situations: Copy-pasting an SFT yaml for DPO without changing the dataset; ranking data stored as separate chosen/rejected string columns without converter='pair'; a converter that silently skips conversion because column names don't match.

Related errors

• DPO training dataset is empty: {dataset_path}
• RM training requires pair-format samples containing chosen/r
• RM training dataset is empty: {dataset_path}
• Unknown mixing strategy: {data_args.mix_strategy}.
• Cannot specify `val_size` if `eval_dataset` is not None.

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/d1adaeb23cc1a0e2. Report an issue: GitHub.