huggingface/transformers · error · ValueError

Predictions and labels have mismatched lengths {len(preds)}

Error message

Predictions and labels have mismatched lengths {len(preds)} and {len(labels)}

What it means

Raised by the deprecated xnli_compute_metrics helper when len(preds) != len(labels). Accuracy is computed element-wise, so mismatched lengths mean the predictions and references do not correspond (wrong shard, off-by-one batching, or a shuffled ordering) and the metric would be meaningless; the function refuses rather than returning a wrong number.

Source

Thrown at src/transformers/data/metrics/__init__.py:94

    elif task_name == "mnli-mm":
        return {"mnli-mm/acc": simple_accuracy(preds, labels)}
    elif task_name == "qnli":
        return {"acc": simple_accuracy(preds, labels)}
    elif task_name == "rte":
        return {"acc": simple_accuracy(preds, labels)}
    elif task_name == "wnli":
        return {"acc": simple_accuracy(preds, labels)}
    elif task_name == "hans":
        return {"acc": simple_accuracy(preds, labels)}
    else:
        raise KeyError(task_name)


def xnli_compute_metrics(task_name, preds, labels):
    warnings.warn(DEPRECATION_WARNING, FutureWarning)
    requires_backends(xnli_compute_metrics, "sklearn")
    if len(preds) != len(labels):
        raise ValueError(f"Predictions and labels have mismatched lengths {len(preds)} and {len(labels)}")
    if task_name == "xnli":
        return {"acc": simple_accuracy(preds, labels)}
    else:
        raise KeyError(task_name)

View on GitHub (pinned to a597f97485)

Solutions

1. Re-run evaluation ensuring every batch contributes exactly one prediction per example (drop_last=False, correct sharding).
2. Verify dataset versions/order: labels from the same split and revision the model was evaluated on.
3. Align lengths explicitly before scoring, e.g. truncate to min length only after confirming the tail is padding-only.

Example fix

# before
metrics = xnli_compute_metrics('xnli', preds, labels)  # len(preds)=1002, len(labels)=1000

# after
assert len(preds) == len(labels), (len(preds), len(labels))
metrics = xnli_compute_metrics('xnli', preds, labels)

assert len(preds) == len(labels), f'preds {len(preds)} != labels {len(labels)}'
metrics = xnli_compute_metrics('xnli', preds, labels)

try:
    metrics = xnli_compute_metrics(task, preds, labels)
except ValueError as e:
    if 'mismatched lengths' in str(e):
        n = min(len(preds), len(labels))
        metrics = xnli_compute_metrics(task, preds[:n], labels[:n])  # only after confirming shard alignment
    else:
        raise

When it happens

Trigger: Calling xnli_compute_metrics('xnli', preds, labels) where predictions were gathered over a subset/reshard of the evaluation set, or where an extra dummy batch's predictions were appended.

Common situations: Multi-GPU evaluation where predictions are gathered but the last partial batch is handled inconsistently; comparing predictions from a checkpoint evaluated on a different dataset version; evaluation loops that drop the label tensor of the final batch.

Related errors

• No valid predictions
• `num_head` was provided as a list of length {len(num_heads)}
• `head_dim` was provided as a list of length {len(num_heads)}
• The `layer_types` entries must be in {ALLOWED_LAYER_TYPES}
• `num_hidden_layers` ({num_hidden_layers}) must be equal to t

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/0da2f9c3272f7821. Report an issue: GitHub.