{"record":{"id":"6d69b93e1d1be79f","repo":"hiyouga/LlamaFactory","slug":"rm-training-requires-pair-format-samples-containin","errorCode":null,"errorMessage":"RM training requires pair-format samples containing chosen/rejected responses. First sample from dataset '{dataset_name}' has keys: {sample_keys}. Please use pair data (e.g. a dataset with chosen_messages/rejected_messages, or set converter='pair' for raw chosen/rejected fields).","messagePattern":"RM training requires pair-format samples containing chosen/rejected responses\\. First sample from dataset '(.+?)' has keys: (.+?)\\. Please use pair data \\(e\\.g\\. a dataset with chosen_messages/rejected_messages, or set converter='pair' for raw chosen/rejected fields\\)\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/llamafactory/v1/trainers/rm_trainer.py","lineNumber":43,"sourceCode":"from ..utils import logging\nfrom ..utils.types import BatchInput, HFModel, Tensor\n\n\nlogger = logging.get_logger(__name__)\n\n\ndef _validate_rm_dataset_format(train_dataset: DataEngine, dataset_path: str) -> None:\n    \"\"\"Validate RM dataset format early for clearer error messages.\"\"\"\n    if len(train_dataset) == 0:\n        raise ValueError(f\"RM training dataset is empty: {dataset_path}\")\n\n    sample = train_dataset[0]\n    if \"chosen_messages\" in sample and \"rejected_messages\" in sample:\n        return\n\n    dataset_name = sample.get(\"_dataset_name\", \"unknown\")\n    sample_keys = sorted(sample.keys())\n    raise ValueError(\n        \"RM training requires pair-format samples containing chosen/rejected responses. \"\n        f\"First sample from dataset '{dataset_name}' has keys: {sample_keys}. \"\n        \"Please use pair data (e.g. a dataset with chosen_messages/rejected_messages, \"\n        \"or set converter='pair' for raw chosen/rejected fields).\"\n    )\n\n\ndef _init_score_head(model: HFModel) -> None:\n    \"\"\"Initialize the score head for RM training with small Gaussian weights.\n\n    Uses Gaussian initialization so that different parameters have distinct values,\n    providing better gradient flow than zero initialization while keeping initial\n    scores small enough that the starting loss is close to ln(2).\n    \"\"\"\n    unwrapped = model.module if hasattr(model, \"module\") else model\n    score = getattr(unwrapped, \"score\", None)\n    if score is not None and hasattr(score, \"weight\"):\n        hidden_size = score.weight.shape[-1]","sourceCodeStart":25,"sourceCodeEnd":61,"githubUrl":"https://github.com/hiyouga/LlamaFactory/blob/f28afaf6355af515454dfb16c97d728307c93897/src/llamafactory/v1/trainers/rm_trainer.py#L25-L61","documentation":"ValueError from _validate_rm_dataset_format (rm_trainer.py:43) when the first sample lacks chosen_messages/rejected_messages. RM training needs preference pairs; the message names the dataset and lists the sample's actual keys. The hint about converter='pair' is specific to RM: raw chosen/rejected columns can be adapted via a pair converter.","triggerScenarios":"Pointing RMTrainer at an SFT/chat dataset (single response), or at a dataset with raw 'chosen'/'rejected' string columns that was not loaded with converter='pair'.","commonSituations":"Reusing a chat dataset for reward modeling; a hub dataset with preference columns under different names; forgetting the converter option in dataset_info.json.","solutions":["Use a preference-pair dataset that yields chosen_messages/rejected_messages.","For datasets with raw chosen/rejected fields, set converter: pair (or ranking format) in dataset_info.json so the loader emits the *_messages keys.","Print train_dataset[0] and align column names via the dataset_info mapping until the pair keys appear."],"exampleFix":"# before (dataset_info.json)\n{\"my_rm\": {\"file_name\": \"prefs.json\", \"format\": \"alpaca\"}}\n\n# after\n{\"my_rm\": {\"file_name\": \"prefs.json\", \"format\": \"alpaca\", \"converter\": \"pair\"}}","handlingStrategy":"validation","validationCode":"sample = train_dataset[0]\nif not (\"chosen_messages\" in sample and \"rejected_messages\" in sample):\n    raise SystemExit(f\"RM needs pair data; keys={sorted(sample)}; add converter='pair' in dataset_info\")","typeGuard":"def is_pair_sample(sample: dict) -> bool:\n    return \"chosen_messages\" in sample and \"rejected_messages\" in sample","tryCatchPattern":null,"preventionTips":["Use converter: pair for raw chosen/rejected datasets.","Smoke-test the first sample's keys before long training runs."],"tags":["data","dataset","reward-model","preference-data","validation"],"backgroundTag":null,"analyzedSha":"f28afaf6355af515454dfb16c97d728307c93897","analyzedAt":"2026-08-14T21:57:28.298Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}