{"record":{"id":"0da2f9c3272f7821","repo":"huggingface/transformers","slug":"predictions-and-labels-have-mismatched-lengths-le","errorCode":null,"errorMessage":"Predictions and labels have mismatched lengths {len(preds)} and {len(labels)}","messagePattern":"Predictions and labels have mismatched lengths (.+?) and (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/transformers/data/metrics/__init__.py","lineNumber":94,"sourceCode":"    elif task_name == \"mnli-mm\":\n        return {\"mnli-mm/acc\": simple_accuracy(preds, labels)}\n    elif task_name == \"qnli\":\n        return {\"acc\": simple_accuracy(preds, labels)}\n    elif task_name == \"rte\":\n        return {\"acc\": simple_accuracy(preds, labels)}\n    elif task_name == \"wnli\":\n        return {\"acc\": simple_accuracy(preds, labels)}\n    elif task_name == \"hans\":\n        return {\"acc\": simple_accuracy(preds, labels)}\n    else:\n        raise KeyError(task_name)\n\n\ndef xnli_compute_metrics(task_name, preds, labels):\n    warnings.warn(DEPRECATION_WARNING, FutureWarning)\n    requires_backends(xnli_compute_metrics, \"sklearn\")\n    if len(preds) != len(labels):\n        raise ValueError(f\"Predictions and labels have mismatched lengths {len(preds)} and {len(labels)}\")\n    if task_name == \"xnli\":\n        return {\"acc\": simple_accuracy(preds, labels)}\n    else:\n        raise KeyError(task_name)\n","sourceCodeStart":76,"sourceCodeEnd":99,"githubUrl":"https://github.com/huggingface/transformers/blob/a597f974857b3d92939971296bc0deb93d33d780/src/transformers/data/metrics/__init__.py#L76-L99","documentation":"Raised by the deprecated xnli_compute_metrics helper when len(preds) != len(labels). Accuracy is computed element-wise, so mismatched lengths mean the predictions and references do not correspond (wrong shard, off-by-one batching, or a shuffled ordering) and the metric would be meaningless; the function refuses rather than returning a wrong number.","triggerScenarios":"Calling xnli_compute_metrics('xnli', preds, labels) where predictions were gathered over a subset/reshard of the evaluation set, or where an extra dummy batch's predictions were appended.","commonSituations":"Multi-GPU evaluation where predictions are gathered but the last partial batch is handled inconsistently; comparing predictions from a checkpoint evaluated on a different dataset version; evaluation loops that drop the label tensor of the final batch.","solutions":["Re-run evaluation ensuring every batch contributes exactly one prediction per example (drop_last=False, correct sharding).","Verify dataset versions/order: labels from the same split and revision the model was evaluated on.","Align lengths explicitly before scoring, e.g. truncate to min length only after confirming the tail is padding-only."],"exampleFix":"# before\nmetrics = xnli_compute_metrics('xnli', preds, labels)  # len(preds)=1002, len(labels)=1000\n\n# after\nassert len(preds) == len(labels), (len(preds), len(labels))\nmetrics = xnli_compute_metrics('xnli', preds, labels)","handlingStrategy":"validation","validationCode":"assert len(preds) == len(labels), f'preds {len(preds)} != labels {len(labels)}'\nmetrics = xnli_compute_metrics('xnli', preds, labels)","typeGuard":null,"tryCatchPattern":"try:\n    metrics = xnli_compute_metrics(task, preds, labels)\nexcept ValueError as e:\n    if 'mismatched lengths' in str(e):\n        n = min(len(preds), len(labels))\n        metrics = xnli_compute_metrics(task, preds[:n], labels[:n])  # only after confirming shard alignment\n    else:\n        raise","preventionTips":["Assert equal lengths before every metric computation.","In distributed evaluation, gather predictions with the same all_gather used for labels so shards stay aligned.","Log dataset sizes before eval to catch version/order drift."],"tags":["metrics","xnli","evaluation","shape-mismatch","deprecated"],"backgroundTag":null,"analyzedSha":"a597f974857b3d92939971296bc0deb93d33d780","analyzedAt":"2026-08-14T18:24:08.354Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}