{"record":{"id":"b9c089781b85ba64","repo":"hankcs/HanLP","slug":"input-tokens-input-tokens-exceed-the-max-sequenc","errorCode":null,"errorMessage":"Input tokens {input_tokens} exceed the max sequence length of {self.max_seq_length - 2}. The exceeded part will be truncated and ignored. You are recommended to split your long text into several sentences within {self.max_seq_length - 2} tokens beforehand.Or simply set truncate_long_sequences = False to enable sliding window.","messagePattern":"Input tokens (.+?) exceed the max sequence length of (.+?)\\. The exceeded part will be truncated and ignored\\. You are recommended to split your long text into several sentences within (.+?) tokens beforehand\\.Or simply set truncate_long_sequences = False to enable sliding window\\.","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"hanlp/transform/transformer_tokenizer.py","lineNumber":500,"sourceCode":"                                             sep_token_extra=self.sep_token_extra,\n                                             # roberta uses an extra separator b/w pairs of sentences, cf. github.com/pytorch/fairseq/commit/1684e166e3da03f5b600dbb7855cb98ddfcd0805\n                                             pad_on_left=self.pad_on_left,\n                                             # pad on the left for xlnet\n                                             pad_token_id=self.pad_token_id,\n                                             pad_token_segment_id=self.pad_token_segment_id,\n                                             pad_token_label_id=0,\n                                             do_padding=self.do_padding)\n        if len(input_ids) > self.max_seq_length:\n            if self.truncate_long_sequences:\n                # raise SequenceTooLong(\n                #     f'Input tokens {input_tokens} exceed the max sequence length of {self.max_seq_length - 2}. '\n                #     f'For sequence tasks, truncate_long_sequences = True is not supported.'\n                #     f'You are recommended to split your long text into several sentences within '\n                #     f'{self.max_seq_length - 2} tokens beforehand. '\n                #     f'Or simply set truncate_long_sequences = False to enable sliding window.')\n                input_ids = input_ids[:self.max_seq_length]\n                prefix_mask = prefix_mask[:self.max_seq_length]\n                warnings.warn(\n                    f'Input tokens {input_tokens} exceed the max sequence length of {self.max_seq_length - 2}. '\n                    f'The exceeded part will be truncated and ignored. '\n                    f'You are recommended to split your long text into several sentences within '\n                    f'{self.max_seq_length - 2} tokens beforehand.'\n                    f'Or simply set truncate_long_sequences = False to enable sliding window.'\n                )\n            else:\n                input_ids = self.sliding_window(input_ids, input_ids[-1] == self.sep_token_id)\n        if prefix_mask:\n            if cls_is_bos:\n                prefix_mask[0] = True\n            if sep_is_eos:\n                prefix_mask[-1] = True\n        outputs = [input_ids]\n        if self.ret_mask_and_type:\n            # noinspection PyUnboundLocalVariable\n            outputs += [attention_mask, token_type_ids]\n        if self.ret_prefix_mask:","sourceCodeStart":482,"sourceCodeEnd":518,"githubUrl":"https://github.com/hankcs/HanLP/blob/ddb1299bddff079e447af52ec12549c50636bfa8/hanlp/transform/transformer_tokenizer.py#L482-L518","documentation":"The transformer tokenizer transform warns that input tokens exceed max_seq_length - 2 (room for [CLS]/[SEP]); the overflow is truncated and silently dropped, so predictions for the tail of the input will be missing unless sliding window is enabled.","triggerScenarios":"Passing sentences longer than the model's max sequence length (e.g. 512-token limit -> >510 tokens) through TransformerTokenizerTransform/__call__ with truncate_long_sequences=True (default).","commonSituations":"Long documents fed as a single sentence; token-count underestimation for languages that fragment into many subwords; downstream spans misaligned with input because the tail was cut.","solutions":["Split long text into sentences shorter than max_seq_length - 2 tokens beforehand","Set truncate_long_sequences=False on the component/transform to enable sliding window (full coverage)","Increase max_seq_length only if the model architecture supports longer positions (e.g. relative-position models)"],"exampleFix":"# before\nhanlp_pipeline(text=very_long_doc)  # tail truncated\n# after\ncomponent.transform.truncate_long_sequences = False  # sliding window\n# or: split text into <510-token sentences first","handlingStrategy":"validation","validationCode":"est = sum(max(1, len(pieces(w)) for w in tokens) for w in []) or 0\nif len(pieces(text)) > max_seq_length - 2:\n    text = split_into_sentences(text)  # keep each under budget","typeGuard":"def fits_limit(tokenized_len: int, max_seq_length: int) -> bool:\n    return tokenized_len <= max_seq_length - 2","tryCatchPattern":null,"preventionTips":["Sentence-split long documents before inference","Set truncate_long_sequences=False for full coverage"],"tags":["transformer","truncation","sequence-length"],"backgroundTag":"sequence-length-exceeded","analyzedSha":"ddb1299bddff079e447af52ec12549c50636bfa8","analyzedAt":"2026-08-27T03:36:54.287Z","schemaVersion":2},"datasetVersion":"2026-08-27T08:17:20.692Z"}