hankcs/HanLP · warning

Input tokens exceed the max sequence length of . The…

Error message

Input tokens {input_tokens} exceed the max sequence length of {self.max_seq_length - 2}. The exceeded part will be truncated and ignored. You are recommended to split your long text into several sentences within {self.max_seq_length - 2} tokens beforehand.Or simply set truncate_long_sequences = False to enable sliding window.

What it means

The transformer tokenizer transform warns that input tokens exceed max_seq_length - 2 (room for [CLS]/[SEP]); the overflow is truncated and silently dropped, so predictions for the tail of the input will be missing unless sliding window is enabled.

Solutions

  1. Split long text into sentences shorter than max_seq_length - 2 tokens beforehand
  2. Set truncate_long_sequences=False on the component/transform to enable sliding window (full coverage)
  3. Increase max_seq_length only if the model architecture supports longer positions (e.g. relative-position models)

Example fix

# before
hanlp_pipeline(text=very_long_doc)  # tail truncated
# after
component.transform.truncate_long_sequences = False  # sliding window
# or: split text into <510-token sentences first
Defensive patterns

Strategy: validation

Validate before calling

est = sum(max(1, len(pieces(w)) for w in tokens) for w in []) or 0
if len(pieces(text)) > max_seq_length - 2:
    text = split_into_sentences(text)  # keep each under budget

Type guard

def fits_limit(tokenized_len: int, max_seq_length: int) -> bool:
    return tokenized_len <= max_seq_length - 2

Prevention

When it happens

Trigger: Passing sentences longer than the model's max sequence length (e.g. 512-token limit -> >510 tokens) through TransformerTokenizerTransform/__call__ with truncate_long_sequences=True (default).

Common situations: Long documents fed as a single sentence; token-count underestimation for languages that fragment into many subwords; downstream spans misaligned with input because the tail was cut.

Related errors

AI-assisted analysis of hankcs/HanLP@ddb1299bdd (2026-08-27). Data as JSON: /api/errors/b9c089781b85ba64. Report an issue: GitHub.

Appendix: source

Thrown at hanlp/transform/transformer_tokenizer.py:500

                                             sep_token_extra=self.sep_token_extra,
                                             # roberta uses an extra separator b/w pairs of sentences, cf. github.com/pytorch/fairseq/commit/1684e166e3da03f5b600dbb7855cb98ddfcd0805
                                             pad_on_left=self.pad_on_left,
                                             # pad on the left for xlnet
                                             pad_token_id=self.pad_token_id,
                                             pad_token_segment_id=self.pad_token_segment_id,
                                             pad_token_label_id=0,
                                             do_padding=self.do_padding)
        if len(input_ids) > self.max_seq_length:
            if self.truncate_long_sequences:
                # raise SequenceTooLong(
                #     f'Input tokens {input_tokens} exceed the max sequence length of {self.max_seq_length - 2}. '
                #     f'For sequence tasks, truncate_long_sequences = True is not supported.'
                #     f'You are recommended to split your long text into several sentences within '
                #     f'{self.max_seq_length - 2} tokens beforehand. '
                #     f'Or simply set truncate_long_sequences = False to enable sliding window.')
                input_ids = input_ids[:self.max_seq_length]
                prefix_mask = prefix_mask[:self.max_seq_length]
                warnings.warn(
                    f'Input tokens {input_tokens} exceed the max sequence length of {self.max_seq_length - 2}. '
                    f'The exceeded part will be truncated and ignored. '
                    f'You are recommended to split your long text into several sentences within '
                    f'{self.max_seq_length - 2} tokens beforehand.'
                    f'Or simply set truncate_long_sequences = False to enable sliding window.'
                )
            else:
                input_ids = self.sliding_window(input_ids, input_ids[-1] == self.sep_token_id)
        if prefix_mask:
            if cls_is_bos:
                prefix_mask[0] = True
            if sep_is_eos:
                prefix_mask[-1] = True
        outputs = [input_ids]
        if self.ret_mask_and_type:
            # noinspection PyUnboundLocalVariable
            outputs += [attention_mask, token_type_ids]
        if self.ret_prefix_mask:

View on GitHub (pinned to ddb1299bdd)