{"record":{"id":"6e48340ac030544d","repo":"huggingface/transformers","slug":"error-with-input-length-len-attention-mask-vs","errorCode":null,"errorMessage":"Error with input length {len(attention_mask)} vs {batch_length}","messagePattern":"Error with input length (.+?) vs (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/transformers/data/processors/utils.py","lineNumber":294,"sourceCode":"            if ex_index % 10000 == 0:\n                logger.info(f\"Writing example {ex_index}/{len(self.examples)}\")\n            # The mask has 1 for real tokens and 0 for padding tokens. Only real\n            # tokens are attended to.\n            attention_mask = [1 if mask_padding_with_zero else 0] * len(input_ids)\n\n            # Zero-pad up to the sequence length.\n            padding_length = batch_length - len(input_ids)\n            if pad_on_left:\n                input_ids = ([pad_token] * padding_length) + input_ids\n                attention_mask = ([0 if mask_padding_with_zero else 1] * padding_length) + attention_mask\n            else:\n                input_ids = input_ids + ([pad_token] * padding_length)\n                attention_mask = attention_mask + ([0 if mask_padding_with_zero else 1] * padding_length)\n\n            if len(input_ids) != batch_length:\n                raise ValueError(f\"Error with input length {len(input_ids)} vs {batch_length}\")\n            if len(attention_mask) != batch_length:\n                raise ValueError(f\"Error with input length {len(attention_mask)} vs {batch_length}\")\n\n            if self.mode == \"classification\":\n                label = label_map[example.label]\n            elif self.mode == \"regression\":\n                label = float(example.label)\n            else:\n                raise ValueError(self.mode)\n\n            if ex_index < 5 and self.verbose:\n                logger.info(\"*** Example ***\")\n                logger.info(f\"guid: {example.guid}\")\n                logger.info(f\"input_ids: {' '.join([str(x) for x in input_ids])}\")\n                logger.info(f\"attention_mask: {' '.join([str(x) for x in attention_mask])}\")\n                logger.info(f\"label: {example.label} (id = {label})\")\n\n            features.append(InputFeatures(input_ids=input_ids, attention_mask=attention_mask, label=label))\n\n        if return_tensors is None:","sourceCodeStart":276,"sourceCodeEnd":312,"githubUrl":"https://github.com/huggingface/transformers/blob/a597f974857b3d92939971296bc0deb93d33d780/src/transformers/data/processors/utils.py#L276-L312","documentation":"The attention_mask is built as a list of the same length as input_ids and padded identically, so this length check fails exactly under the same conditions as the input_ids check: an input longer than batch_length makes padding_length negative and the padded attention_mask ends up not matching batch_length. Seeing this message means the example length exceeded max_seq_length without truncation.","triggerScenarios":"Same root cause as the input_ids check: max_length/batch_length smaller than a tokenized example and no truncation; mixing pad_on_left settings between runs while reusing cached lengths.","commonSituations":"Long outlier documents exceeding max_seq_length; lowering max_seq_length for memory reasons without adding truncation; datasets with mixed-length domains (tweets vs articles) sharing one limit.","solutions":["Turn on truncation at tokenization time so len(input_ids) never exceeds max_length.","Increase max_seq_length to at least the longest tokenized example.","Measure token lengths first: max(map(len, tokenizer(list_of_texts)['input_ids'])) to pick a safe limit."],"exampleFix":"# before\nfeatures = featurizer.get_features(texts, max_length=64)  # docs exceed 64\n\n# after\nfeatures = featurizer.get_features(texts, max_length=64, truncation=True)","handlingStrategy":"validation","validationCode":"lengths = [len(ids) for ids in tokenizer(texts, add_special_tokens=True)[\"input_ids\"]]\nassert max(lengths) <= max_seq_length, f\"max token length {max(lengths)} exceeds {max_seq_length}\"","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Treat any padding-length error as an over-long input: truncate first.","Keep one source of truth for max_length shared by tokenization and featurization.","Filter or split documents longer than the limit during preprocessing."],"tags":["tokenization","padding","attention-mask","max-length"],"backgroundTag":null,"analyzedSha":"a597f974857b3d92939971296bc0deb93d33d780","analyzedAt":"2026-08-14T18:24:08.354Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}