{"record":{"id":"59b6c7f58538c649","repo":"hiyouga/LlamaFactory","slug":"rm-training-requires-pair-data-with-token-type-ids","errorCode":null,"errorMessage":"RM training requires pair data with token_type_ids. Ensure the dataset has chosen_messages/rejected_messages.","messagePattern":"RM training requires pair data with token_type_ids\\. Ensure the dataset has chosen_messages/rejected_messages\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/llamafactory/v1/trainers/rm_trainer.py","lineNumber":107,"sourceCode":"                device_ids = None if self.device.type == \"cpu\" else [self.device.index]\n                self.model = DDP(self.model, device_ids=device_ids, find_unused_parameters=True)\n        else:\n            super()._shard_model()\n\n    @property\n    def _unwrapped_model(self):\n        \"\"\"Access the underlying model, unwrapping DDP/FSDP wrappers if present.\"\"\"\n        model = self.model\n        if hasattr(model, \"module\"):\n            model = model.module\n        return model\n\n    def compute_loss(self, batch: BatchInput) -> Tensor:\n        input_ids = batch[\"input_ids\"].to(self.device, non_blocking=True)\n\n        token_type_ids = batch.get(\"token_type_ids\")\n        if token_type_ids is None:\n            raise ValueError(\n                \"RM training requires pair data with token_type_ids. \"\n                \"Ensure the dataset has chosen_messages/rejected_messages.\"\n            )\n        token_type_ids = token_type_ids.to(self.device, non_blocking=True)\n\n        # Use token_type_ids as document-index attention mask (values: 1=chosen, 2=rejected, 0=padding).\n        # Transformers v5 models natively support this format in _update_causal_mask,\n        # constructing the correct block-diagonal causal mask internally for all attention backends.\n        model_attention_mask = token_type_ids\n\n        # Build position_ids that reset at each document boundary.\n        batch_size, seq_len = token_type_ids.shape\n        arange = torch.arange(seq_len, device=self.device).unsqueeze(0).expand(batch_size, -1)\n        chosen_mask = token_type_ids == 1\n        rejected_mask = token_type_ids == 2\n        chosen_lens = chosen_mask.sum(dim=1, keepdim=True)\n        position_ids = torch.zeros_like(token_type_ids)\n        position_ids[chosen_mask] = arange[chosen_mask]","sourceCodeStart":89,"sourceCodeEnd":125,"githubUrl":"https://github.com/hiyouga/LlamaFactory/blob/f28afaf6355af515454dfb16c97d728307c93897/src/llamafactory/v1/trainers/rm_trainer.py#L89-L125","documentation":"ValueError in RMTrainer.compute_loss (rm_trainer.py:107) when the batch carries no token_type_ids. RM packs chosen and rejected into one sequence and relies on token_type_ids (1=chosen, 2=rejected, 0=padding) to build the block-diagonal attention mask and locate each response's last token. Without it the loss is uncomputable, so the trainer refuses the batch.","triggerScenarios":"The data processor did not emit token_type_ids: non-pair dataset slipped past validation (e.g. a custom collator), a batch assembled without the pair processor, or a Manually constructed BatchInput missing the field. Note batch.get() returning None also covers an explicit None value.","commonSituations":"Custom collators or data pipelines that drop token_type_ids; using a processor stage that does not generate pair token types; mixing an SFT data pipeline with RMTrainer.","solutions":["Use the RM/pair data processor so batches contain token_type_ids alongside input_ids.","If building batches manually, populate token_type_ids with 1 on chosen-response tokens, 2 on rejected-response tokens, 0 on padding before calling compute_loss.","Ensure the dataset passed pair-format validation (chosen_messages/rejected_messages) so the processor marks token types."],"exampleFix":"# before (custom batch)\nbatch = {\"input_ids\": ids, \"attention_mask\": mask}\n\n# after\nbatch = {\"input_ids\": ids, \"token_type_ids\": token_types}  # 1=chosen, 2=rejected, 0=pad","handlingStrategy":"type-guard","validationCode":"assert batch.get(\"token_type_ids\") is not None, \"RM batch missing token_type_ids (1=chosen, 2=rejected, 0=pad)\"","typeGuard":"def is_rm_batch(batch) -> bool:\n    tt = batch.get(\"token_type_ids\")\n    return tt is not None and bool(((tt == 1) | (tt == 2)).any())","tryCatchPattern":null,"preventionTips":["Use the RM pair data processor end-to-end; do not substitute custom collators without token_type_ids.","Assert token_type_ids presence in the collator's output during development."],"tags":["reward-model","data","batch","token-type-ids"],"backgroundTag":null,"analyzedSha":"f28afaf6355af515454dfb16c97d728307c93897","analyzedAt":"2026-08-14T21:57:28.298Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}