{"record":{"id":"1e5bb0134dfe145d","repo":"hankcs/HanLP","slug":"input-tokens-words-exceed-the-max-sequence-lengt","errorCode":null,"errorMessage":"Input tokens {words} exceed the max sequence length of {max_seq_length - special_tokens_count}. The exceeded part will be truncated and ignored. You are recommended to split your long text into several sentences within {max_seq_length - special_tokens_count} tokens beforehand.","messagePattern":"Input tokens (.+?) exceed the max sequence length of (.+?)\\. The exceeded part will be truncated and ignored\\. You are recommended to split your long text into several sentences within (.+?) tokens beforehand\\.","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"hanlp/transform/transformer_tokenizer.py","lineNumber":624,"sourceCode":"    if not labels:\n        labels = words\n        pad_token_label_id = False\n\n    tokens = []\n    label_ids = []\n    for word, label in zip(words, labels):\n        word_tokens = tokenizer.tokenize(word)\n        if not word_tokens:\n            # some wired chars cause the tagger to return empty list\n            word_tokens = [unk_token] * len(word)\n        tokens.extend(word_tokens)\n        # Use the real label id for the first token of the word, and padding ids for the remaining tokens\n        label_ids.extend([label_map[label] if label_map else True] + [pad_token_label_id] * (len(word_tokens) - 1))\n\n    # Account for [CLS] and [SEP] with \"- 2\" and with \"- 3\" for RoBERTa.\n    special_tokens_count = 3 if sep_token_extra else 2\n    if max_seq_length and len(tokens) > max_seq_length - special_tokens_count:\n        warnings.warn(\n            f'Input tokens {words} exceed the max sequence length of {max_seq_length - special_tokens_count}. '\n            f'The exceeded part will be truncated and ignored. '\n            f'You are recommended to split your long text into several sentences within '\n            f'{max_seq_length - special_tokens_count} tokens beforehand.')\n        tokens = tokens[: (max_seq_length - special_tokens_count)]\n        label_ids = label_ids[: (max_seq_length - special_tokens_count)]\n\n    # The convention in BERT is:\n    # (a) For sequence pairs:\n    #  tokens:   [CLS] is this jack ##son ##ville ? [SEP] no it is not . [SEP]\n    #  token_type_ids:   0   0  0    0    0     0       0   0   1  1  1  1   1   1\n    # (b) For single sequences:\n    #  tokens:   [CLS] the dog is hairy . [SEP]\n    #  token_type_ids:   0   0   0   0  0     0   0\n    #\n    # Where \"token_type_ids\" are used to indicate whether this is the first\n    # sequence or the second sequence. The embedding vectors for `type=0` and\n    # `type=1` were learned during pre-training and are added to the wordpiece","sourceCodeStart":606,"sourceCodeEnd":642,"githubUrl":"https://github.com/hankcs/HanLP/blob/ddb1299bddff079e447af52ec12549c50636bfa8/hanlp/transform/transformer_tokenizer.py#L606-L642","documentation":"convert_examples_to_features warns when tokenized words exceed max_seq_length - special_tokens_count (2 for BERT-style, 3 with sep_token_extra for RoBERTa/XLNet); tokens and label ids are truncated, so trailing tokens/labels are dropped from features.","triggerScenarios":"Tokenizing examples whose subword length exceeds the budget — e.g. a 400-word sentence that explodes past 510 subword pieces — via TransformerTokenizerTransform used by batched_inputs_to_batches or __call__.","commonSituations":"Agglutinative/character-split languages inflating subword counts; datasets with occasional very long lines; label misalignment downstream because label_ids were cut.","solutions":["Pre-split long sentences so subword counts stay under max_seq_length - special_tokens_count","Enable sliding-window handling (truncate_long_sequences=False) if the calling transform supports it","For RoBERTa-style models remember the budget is max_seq_length - 3, not - 2"],"exampleFix":"# before\nfeats = tokenizer.convert_examples_to_features(long_words, max_seq_length=512)  # warns, truncates\n# after\nchunks = [long_words[i:i+200] for i in range(0, len(long_words), 200)]\nfeats = [tokenizer.convert_examples_to_features(c, max_seq_length=512) for c in chunks]","handlingStrategy":"validation","validationCode":"budget = max_seq_length - (3 if sep_token_extra else 2)\nassert sum(len(tok.tokenize(w)) for w in words) <= budget, 'example too long — split first'","typeGuard":"def example_fits(words: List[str], tokenizer, max_seq_length: int, sep_token_extra: bool = False) -> bool:\n    budget = max_seq_length - (3 if sep_token_extra else 2)\n    return sum(len(tokenizer.tokenize(w)) for w in words) <= budget","tryCatchPattern":null,"preventionTips":["Budget is max_seq_length minus 2 (BERT) or 3 (RoBERTa/XLNet) subword pieces","Chunk long sentences and merge spans after prediction"],"tags":["transformer","truncation","sequence-length"],"backgroundTag":"sequence-length-exceeded","analyzedSha":"ddb1299bddff079e447af52ec12549c50636bfa8","analyzedAt":"2026-08-27T03:36:54.287Z","schemaVersion":2},"datasetVersion":"2026-08-27T08:17:20.692Z"}