{"record":{"id":"798505a1526ac2be","repo":"huggingface/transformers","slug":"error-with-input-length-len-input-ids-vs-batch","errorCode":null,"errorMessage":"Error with input length {len(input_ids)} vs {batch_length}","messagePattern":"Error with input length (.+?) vs (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/transformers/data/processors/utils.py","lineNumber":292,"sourceCode":"        features = []\n        for ex_index, (input_ids, example) in enumerate(zip(all_input_ids, self.examples)):\n            if ex_index % 10000 == 0:\n                logger.info(f\"Writing example {ex_index}/{len(self.examples)}\")\n            # The mask has 1 for real tokens and 0 for padding tokens. Only real\n            # tokens are attended to.\n            attention_mask = [1 if mask_padding_with_zero else 0] * len(input_ids)\n\n            # Zero-pad up to the sequence length.\n            padding_length = batch_length - len(input_ids)\n            if pad_on_left:\n                input_ids = ([pad_token] * padding_length) + input_ids\n                attention_mask = ([0 if mask_padding_with_zero else 1] * padding_length) + attention_mask\n            else:\n                input_ids = input_ids + ([pad_token] * padding_length)\n                attention_mask = attention_mask + ([0 if mask_padding_with_zero else 1] * padding_length)\n\n            if len(input_ids) != batch_length:\n                raise ValueError(f\"Error with input length {len(input_ids)} vs {batch_length}\")\n            if len(attention_mask) != batch_length:\n                raise ValueError(f\"Error with input length {len(attention_mask)} vs {batch_length}\")\n\n            if self.mode == \"classification\":\n                label = label_map[example.label]\n            elif self.mode == \"regression\":\n                label = float(example.label)\n            else:\n                raise ValueError(self.mode)\n\n            if ex_index < 5 and self.verbose:\n                logger.info(\"*** Example ***\")\n                logger.info(f\"guid: {example.guid}\")\n                logger.info(f\"input_ids: {' '.join([str(x) for x in input_ids])}\")\n                logger.info(f\"attention_mask: {' '.join([str(x) for x in attention_mask])}\")\n                logger.info(f\"label: {example.label} (id = {label})\")\n\n            features.append(InputFeatures(input_ids=input_ids, attention_mask=attention_mask, label=label))","sourceCodeStart":274,"sourceCodeEnd":310,"githubUrl":"https://github.com/huggingface/transformers/blob/a597f974857b3d92939971296bc0deb93d33d780/src/transformers/data/processors/utils.py#L274-L310","documentation":"While padding tokenized inputs to batch_length (the max sequence length in the batch or max_length), the featurizer computes padding_length = batch_length - len(input_ids) and pads. If input_ids is LONGER than batch_length, padding_length is negative, the 'padding' shrinks nothing (slicing a negative count of pad tokens appends/removes nothing with * negative -> empty list), and the final length check fires. In practice this means an input exceeded max_seq_length and no truncation strategy was applied.","triggerScenarios":"Tokenizing examples longer than max_length with no truncation (the legacy featurizer's tokenizer settings do not truncate); setting max_length smaller than some sequences in the batch; enabling pad_on_left with an over-long sequence (same arithmetic).","commonSituations":"Reusing a max_seq_length tuned for short texts (e.g. 128) on longer documents; a few outlier documents in the dataset; switching a tokenizer/pipeline that previously truncated automatically to the low-level featurizer API which does not.","solutions":["Enable truncation when tokenizing so inputs are cut to max_length (e.g. tokenizer(..., truncation=True, max_length=max_seq_length)).","Raise max_seq_length / batch_length above the longest sequence in your data (check max(len(tokenizer.encode(t)) for t in texts)).","Pre-filter or split over-long examples before featurization."],"exampleFix":"# before\nfeatures = featurizer.get_features(texts, max_length=128)  # some texts longer\n\n# after\nfeatures = featurizer.get_features(texts, max_length=128, truncation=True)\n# or: raise the limit after measuring the longest sequence","handlingStrategy":"validation","validationCode":"max_len = max(len(tokenizer.encode(t)) for t in texts)\nif max_len > max_seq_length:\n    raise ValueError(f\"Longest example {max_len} > max_seq_length {max_seq_length}; enable truncation or raise the limit\")","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Always tokenize with truncation=True and an explicit max_length.","Measure the longest tokenized example before choosing max_seq_length.","Log distribution of token lengths during preprocessing to catch outliers."],"tags":["tokenization","padding","max-length","truncation"],"backgroundTag":null,"analyzedSha":"a597f974857b3d92939971296bc0deb93d33d780","analyzedAt":"2026-08-14T18:24:08.354Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}