hiyouga/LlamaFactory · error · ValueError

No valid RM pairs found in this micro-batch. This is usually

Error message

No valid RM pairs found in this micro-batch. This is usually caused by cutoff_len being too small and truncating chosen/rejected tokens.

What it means

ValueError in RMTrainer.compute_loss (rm_trainer.py:143) when no sequence in the micro-batch contains both chosen tokens (token_type_ids==1) and rejected tokens (token_type_ids==2). Truncation to cutoff_len removed the tail of each pair, so entire documents lost their labels and valid_pair_mask is all False.

Source

Thrown at src/llamafactory/v1/trainers/rm_trainer.py:143

        position_ids[chosen_mask] = arange[chosen_mask]
        position_ids[rejected_mask] = (arange - chosen_lens)[rejected_mask]

        model_output = self.model(
            input_ids=input_ids,
            attention_mask=model_attention_mask,
            position_ids=position_ids,
            use_cache=False,
            return_dict=True,
        )

        rewards = model_output.logits.float().squeeze(-1)

        chosen_mask = token_type_ids == 1
        rejected_mask = token_type_ids == 2

        valid_pair_mask = chosen_mask.any(dim=-1) & rejected_mask.any(dim=-1)
        if not torch.any(valid_pair_mask):
            raise ValueError(
                "No valid RM pairs found in this micro-batch. "
                "This is usually caused by cutoff_len being too small and truncating chosen/rejected tokens."
            )

        rewards = rewards[valid_pair_mask]
        chosen_mask = chosen_mask[valid_pair_mask]
        rejected_mask = rejected_mask[valid_pair_mask]

        seq_len = rewards.size(-1)
        position_index = torch.arange(seq_len, device=self.device).unsqueeze(0)
        chosen_last_idx = (position_index * chosen_mask.long()).max(dim=-1).values
        rejected_last_idx = (position_index * rejected_mask.long()).max(dim=-1).values

        chosen_scores = rewards.gather(dim=1, index=chosen_last_idx.unsqueeze(-1)).squeeze(-1)
        rejected_scores = rewards.gather(dim=1, index=rejected_last_idx.unsqueeze(-1)).squeeze(-1)
        return -F.logsigmoid(chosen_scores - rejected_scores).mean()

View on GitHub (pinned to f28afaf635)

Solutions

Increase cutoff_len so prompt + each response fits (compare against the 95th percentile of tokenized pair lengths).
If memory forces a small cutoff, filter out dataset pairs whose tokenized length exceeds it during preprocessing.
Verify token_type_ids actually contain 1s and 2s (print batch['token_type_ids'].unique()) to rule out an encoding bug.

Example fix

# before
data_args:
  cutoff_len: 512   # truncates responses of long preference pairs

# after
data_args:
  cutoff_len: 4096

Defensive patterns

Strategy: validation

Validate before calling

chosen = (token_type_ids == 1).any(-1)
rejected = (token_type_ids == 2).any(-1)
assert (chosen & rejected).any(), "no valid pair survives truncation; raise cutoff_len or filter long pairs"
# dataset-level: filter pairs longer than cutoff_len before training

Prevention

Compute tokenized pair-length percentiles and set cutoff_len above p95.
Pre-filter over-length pairs during data prep instead of relying on truncation.
Check token_type_ids.unique() in a debug step to confirm labels survive truncation.

When it happens

Trigger: cutoff_len shorter than prompt+chosen or prompt+rejected, so truncation cuts the response tokens entirely; or token_type_ids populated incorrectly (all zeros) so masks match nothing.

Common situations: Long prompts with short cutoff_len (e.g. 512 for long-context preference data); packing/truncation applied from the wrong end; debugging environments with tiny cutoff values.

Related errors

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/dd4b97e831ed67f2. Report an issue: GitHub.