hankcs/HanLP · warning

Input tokens {words} exceed the max sequence length of {max_

Error message

Input tokens {words} exceed the max sequence length of {max_seq_length - special_tokens_count}. The exceeded part will be truncated and ignored. You are recommended to split your long text into several sentences within {max_seq_length - special_tokens_count} tokens beforehand.

What it means

convert_examples_to_features warns when tokenized words exceed max_seq_length - special_tokens_count (2 for BERT-style, 3 with sep_token_extra for RoBERTa/XLNet); tokens and label ids are truncated, so trailing tokens/labels are dropped from features.

Source

Thrown at hanlp/transform/transformer_tokenizer.py:624

    if not labels:
        labels = words
        pad_token_label_id = False

    tokens = []
    label_ids = []
    for word, label in zip(words, labels):
        word_tokens = tokenizer.tokenize(word)
        if not word_tokens:
            # some wired chars cause the tagger to return empty list
            word_tokens = [unk_token] * len(word)
        tokens.extend(word_tokens)
        # Use the real label id for the first token of the word, and padding ids for the remaining tokens
        label_ids.extend([label_map[label] if label_map else True] + [pad_token_label_id] * (len(word_tokens) - 1))

    # Account for [CLS] and [SEP] with "- 2" and with "- 3" for RoBERTa.
    special_tokens_count = 3 if sep_token_extra else 2
    if max_seq_length and len(tokens) > max_seq_length - special_tokens_count:
        warnings.warn(
            f'Input tokens {words} exceed the max sequence length of {max_seq_length - special_tokens_count}. '
            f'The exceeded part will be truncated and ignored. '
            f'You are recommended to split your long text into several sentences within '
            f'{max_seq_length - special_tokens_count} tokens beforehand.')
        tokens = tokens[: (max_seq_length - special_tokens_count)]
        label_ids = label_ids[: (max_seq_length - special_tokens_count)]

    # The convention in BERT is:
    # (a) For sequence pairs:
    #  tokens:   [CLS] is this jack ##son ##ville ? [SEP] no it is not . [SEP]
    #  token_type_ids:   0   0  0    0    0     0       0   0   1  1  1  1   1   1
    # (b) For single sequences:
    #  tokens:   [CLS] the dog is hairy . [SEP]
    #  token_type_ids:   0   0   0   0  0     0   0
    #
    # Where "token_type_ids" are used to indicate whether this is the first
    # sequence or the second sequence. The embedding vectors for `type=0` and
    # `type=1` were learned during pre-training and are added to the wordpiece

View on GitHub (pinned to ddb1299bdd)

Solutions

1. Pre-split long sentences so subword counts stay under max_seq_length - special_tokens_count
2. Enable sliding-window handling (truncate_long_sequences=False) if the calling transform supports it
3. For RoBERTa-style models remember the budget is max_seq_length - 3, not - 2

Example fix

# before
feats = tokenizer.convert_examples_to_features(long_words, max_seq_length=512)  # warns, truncates
# after
chunks = [long_words[i:i+200] for i in range(0, len(long_words), 200)]
feats = [tokenizer.convert_examples_to_features(c, max_seq_length=512) for c in chunks]

budget = max_seq_length - (3 if sep_token_extra else 2)
assert sum(len(tok.tokenize(w)) for w in words) <= budget, 'example too long — split first'

def example_fits(words: List[str], tokenizer, max_seq_length: int, sep_token_extra: bool = False) -> bool:
    budget = max_seq_length - (3 if sep_token_extra else 2)
    return sum(len(tokenizer.tokenize(w)) for w in words) <= budget

When it happens

Trigger: Tokenizing examples whose subword length exceeds the budget — e.g. a 400-word sentence that explodes past 510 subword pieces — via TransformerTokenizerTransform used by batched_inputs_to_batches or __call__.

Common situations: Agglutinative/character-split languages inflating subword counts; datasets with occasional very long lines; label misalignment downstream because label_ids were cut.

Related errors

• Input tokens {input_tokens} exceed the max sequence length o

AI-assisted analysis of hankcs/HanLP@ddb1299bdd (2026-08-27). Data as JSON: /api/errors/1e5bb0134dfe145d. Report an issue: GitHub.