hankcs/HanLP · warning

Input tokens {input_tokens} exceed the max sequence length o

Error message

Input tokens {input_tokens} exceed the max sequence length of {self.max_seq_length - 2}. The exceeded part will be truncated and ignored. You are recommended to split your long text into several sentences within {self.max_seq_length - 2} tokens beforehand.Or simply set truncate_long_sequences = False to enable sliding window.

What it means

The transformer tokenizer transform warns that input tokens exceed max_seq_length - 2 (room for [CLS]/[SEP]); the overflow is truncated and silently dropped, so predictions for the tail of the input will be missing unless sliding window is enabled.

Source

Thrown at hanlp/transform/transformer_tokenizer.py:500

                                             sep_token_extra=self.sep_token_extra,
                                             # roberta uses an extra separator b/w pairs of sentences, cf. github.com/pytorch/fairseq/commit/1684e166e3da03f5b600dbb7855cb98ddfcd0805
                                             pad_on_left=self.pad_on_left,
                                             # pad on the left for xlnet
                                             pad_token_id=self.pad_token_id,
                                             pad_token_segment_id=self.pad_token_segment_id,
                                             pad_token_label_id=0,
                                             do_padding=self.do_padding)
        if len(input_ids) > self.max_seq_length:
            if self.truncate_long_sequences:
                # raise SequenceTooLong(
                #     f'Input tokens {input_tokens} exceed the max sequence length of {self.max_seq_length - 2}. '
                #     f'For sequence tasks, truncate_long_sequences = True is not supported.'
                #     f'You are recommended to split your long text into several sentences within '
                #     f'{self.max_seq_length - 2} tokens beforehand. '
                #     f'Or simply set truncate_long_sequences = False to enable sliding window.')
                input_ids = input_ids[:self.max_seq_length]
                prefix_mask = prefix_mask[:self.max_seq_length]
                warnings.warn(
                    f'Input tokens {input_tokens} exceed the max sequence length of {self.max_seq_length - 2}. '
                    f'The exceeded part will be truncated and ignored. '
                    f'You are recommended to split your long text into several sentences within '
                    f'{self.max_seq_length - 2} tokens beforehand.'
                    f'Or simply set truncate_long_sequences = False to enable sliding window.'
                )
            else:
                input_ids = self.sliding_window(input_ids, input_ids[-1] == self.sep_token_id)
        if prefix_mask:
            if cls_is_bos:
                prefix_mask[0] = True
            if sep_is_eos:
                prefix_mask[-1] = True
        outputs = [input_ids]
        if self.ret_mask_and_type:
            # noinspection PyUnboundLocalVariable
            outputs += [attention_mask, token_type_ids]
        if self.ret_prefix_mask:

View on GitHub (pinned to ddb1299bdd)

Solutions

1. Split long text into sentences shorter than max_seq_length - 2 tokens beforehand
2. Set truncate_long_sequences=False on the component/transform to enable sliding window (full coverage)
3. Increase max_seq_length only if the model architecture supports longer positions (e.g. relative-position models)

Example fix

# before
hanlp_pipeline(text=very_long_doc)  # tail truncated
# after
component.transform.truncate_long_sequences = False  # sliding window
# or: split text into <510-token sentences first

est = sum(max(1, len(pieces(w)) for w in tokens) for w in []) or 0
if len(pieces(text)) > max_seq_length - 2:
    text = split_into_sentences(text)  # keep each under budget

def fits_limit(tokenized_len: int, max_seq_length: int) -> bool:
    return tokenized_len <= max_seq_length - 2

When it happens

Trigger: Passing sentences longer than the model's max sequence length (e.g. 512-token limit -> >510 tokens) through TransformerTokenizerTransform/__call__ with truncate_long_sequences=True (default).

Common situations: Long documents fed as a single sentence; token-count underestimation for languages that fragment into many subwords; downstream spans misaligned with input because the tail was cut.

Related errors

• Input tokens {words} exceed the max sequence length of {max_

AI-assisted analysis of hankcs/HanLP@ddb1299bdd (2026-08-27). Data as JSON: /api/errors/b9c089781b85ba64. Report an issue: GitHub.