hiyouga/LlamaFactory · error · ValueError

RM training requires pair-format samples containing chosen/r

Error message

RM training requires pair-format samples containing chosen/rejected responses. First sample from dataset '{dataset_name}' has keys: {sample_keys}. Please use pair data (e.g. a dataset with chosen_messages/rejected_messages, or set converter='pair' for raw chosen/rejected fields).

What it means

ValueError from _validate_rm_dataset_format (rm_trainer.py:43) when the first sample lacks chosen_messages/rejected_messages. RM training needs preference pairs; the message names the dataset and lists the sample's actual keys. The hint about converter='pair' is specific to RM: raw chosen/rejected columns can be adapted via a pair converter.

Source

Thrown at src/llamafactory/v1/trainers/rm_trainer.py:43

from ..utils import logging
from ..utils.types import BatchInput, HFModel, Tensor


logger = logging.get_logger(__name__)


def _validate_rm_dataset_format(train_dataset: DataEngine, dataset_path: str) -> None:
    """Validate RM dataset format early for clearer error messages."""
    if len(train_dataset) == 0:
        raise ValueError(f"RM training dataset is empty: {dataset_path}")

    sample = train_dataset[0]
    if "chosen_messages" in sample and "rejected_messages" in sample:
        return

    dataset_name = sample.get("_dataset_name", "unknown")
    sample_keys = sorted(sample.keys())
    raise ValueError(
        "RM training requires pair-format samples containing chosen/rejected responses. "
        f"First sample from dataset '{dataset_name}' has keys: {sample_keys}. "
        "Please use pair data (e.g. a dataset with chosen_messages/rejected_messages, "
        "or set converter='pair' for raw chosen/rejected fields)."
    )


def _init_score_head(model: HFModel) -> None:
    """Initialize the score head for RM training with small Gaussian weights.

    Uses Gaussian initialization so that different parameters have distinct values,
    providing better gradient flow than zero initialization while keeping initial
    scores small enough that the starting loss is close to ln(2).
    """
    unwrapped = model.module if hasattr(model, "module") else model
    score = getattr(unwrapped, "score", None)
    if score is not None and hasattr(score, "weight"):
        hidden_size = score.weight.shape[-1]

View on GitHub (pinned to f28afaf635)

Solutions

1. Use a preference-pair dataset that yields chosen_messages/rejected_messages.
2. For datasets with raw chosen/rejected fields, set converter: pair (or ranking format) in dataset_info.json so the loader emits the *_messages keys.
3. Print train_dataset[0] and align column names via the dataset_info mapping until the pair keys appear.

Example fix

# before (dataset_info.json)
{"my_rm": {"file_name": "prefs.json", "format": "alpaca"}}

# after
{"my_rm": {"file_name": "prefs.json", "format": "alpaca", "converter": "pair"}}

sample = train_dataset[0]
if not ("chosen_messages" in sample and "rejected_messages" in sample):
    raise SystemExit(f"RM needs pair data; keys={sorted(sample)}; add converter='pair' in dataset_info")

def is_pair_sample(sample: dict) -> bool:
    return "chosen_messages" in sample and "rejected_messages" in sample

When it happens

Trigger: Pointing RMTrainer at an SFT/chat dataset (single response), or at a dataset with raw 'chosen'/'rejected' string columns that was not loaded with converter='pair'.

Common situations: Reusing a chat dataset for reward modeling; a hub dataset with preference columns under different names; forgetting the converter option in dataset_info.json.

Related errors

• DPO training requires pair-format samples containing chosen/
• RM training dataset is empty: {dataset_path}
• DPO training dataset is empty: {dataset_path}
• Unknown mixing strategy: {data_args.mix_strategy}.
• Cannot specify `val_size` if `eval_dataset` is not None.

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/6d69b93e1d1be79f. Report an issue: GitHub.