{"record":{"id":"dd4b97e831ed67f2","repo":"hiyouga/LlamaFactory","slug":"no-valid-rm-pairs-found-in-this-micro-batch-this","errorCode":null,"errorMessage":"No valid RM pairs found in this micro-batch. This is usually caused by cutoff_len being too small and truncating chosen/rejected tokens.","messagePattern":"No valid RM pairs found in this micro-batch\\. This is usually caused by cutoff_len being too small and truncating chosen/rejected tokens\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/llamafactory/v1/trainers/rm_trainer.py","lineNumber":143,"sourceCode":"        position_ids[chosen_mask] = arange[chosen_mask]\n        position_ids[rejected_mask] = (arange - chosen_lens)[rejected_mask]\n\n        model_output = self.model(\n            input_ids=input_ids,\n            attention_mask=model_attention_mask,\n            position_ids=position_ids,\n            use_cache=False,\n            return_dict=True,\n        )\n\n        rewards = model_output.logits.float().squeeze(-1)\n\n        chosen_mask = token_type_ids == 1\n        rejected_mask = token_type_ids == 2\n\n        valid_pair_mask = chosen_mask.any(dim=-1) & rejected_mask.any(dim=-1)\n        if not torch.any(valid_pair_mask):\n            raise ValueError(\n                \"No valid RM pairs found in this micro-batch. \"\n                \"This is usually caused by cutoff_len being too small and truncating chosen/rejected tokens.\"\n            )\n\n        rewards = rewards[valid_pair_mask]\n        chosen_mask = chosen_mask[valid_pair_mask]\n        rejected_mask = rejected_mask[valid_pair_mask]\n\n        seq_len = rewards.size(-1)\n        position_index = torch.arange(seq_len, device=self.device).unsqueeze(0)\n        chosen_last_idx = (position_index * chosen_mask.long()).max(dim=-1).values\n        rejected_last_idx = (position_index * rejected_mask.long()).max(dim=-1).values\n\n        chosen_scores = rewards.gather(dim=1, index=chosen_last_idx.unsqueeze(-1)).squeeze(-1)\n        rejected_scores = rewards.gather(dim=1, index=rejected_last_idx.unsqueeze(-1)).squeeze(-1)\n        return -F.logsigmoid(chosen_scores - rejected_scores).mean()\n\n","sourceCodeStart":125,"sourceCodeEnd":161,"githubUrl":"https://github.com/hiyouga/LlamaFactory/blob/f28afaf6355af515454dfb16c97d728307c93897/src/llamafactory/v1/trainers/rm_trainer.py#L125-L161","documentation":"ValueError in RMTrainer.compute_loss (rm_trainer.py:143) when no sequence in the micro-batch contains both chosen tokens (token_type_ids==1) and rejected tokens (token_type_ids==2). Truncation to cutoff_len removed the tail of each pair, so entire documents lost their labels and valid_pair_mask is all False.","triggerScenarios":"cutoff_len shorter than prompt+chosen or prompt+rejected, so truncation cuts the response tokens entirely; or token_type_ids populated incorrectly (all zeros) so masks match nothing.","commonSituations":"Long prompts with short cutoff_len (e.g. 512 for long-context preference data); packing/truncation applied from the wrong end; debugging environments with tiny cutoff values.","solutions":["Increase cutoff_len so prompt + each response fits (compare against the 95th percentile of tokenized pair lengths).","If memory forces a small cutoff, filter out dataset pairs whose tokenized length exceeds it during preprocessing.","Verify token_type_ids actually contain 1s and 2s (print batch['token_type_ids'].unique()) to rule out an encoding bug."],"exampleFix":"# before\ndata_args:\n  cutoff_len: 512   # truncates responses of long preference pairs\n\n# after\ndata_args:\n  cutoff_len: 4096","handlingStrategy":"validation","validationCode":"chosen = (token_type_ids == 1).any(-1)\nrejected = (token_type_ids == 2).any(-1)\nassert (chosen & rejected).any(), \"no valid pair survives truncation; raise cutoff_len or filter long pairs\"\n# dataset-level: filter pairs longer than cutoff_len before training","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Compute tokenized pair-length percentiles and set cutoff_len above p95.","Pre-filter over-length pairs during data prep instead of relying on truncation.","Check token_type_ids.unique() in a debug step to confirm labels survive truncation."],"tags":["reward-model","truncation","cutoff-len","data"],"backgroundTag":null,"analyzedSha":"f28afaf6355af515454dfb16c97d728307c93897","analyzedAt":"2026-08-14T21:57:28.298Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}