hankcs/HanLP · warning

`do_basic_tokenize=False` might not work when `use_fast=True

Error message

`do_basic_tokenize=False` might not work when `use_fast=True`

What it means

Warns that transformers' fast (Rust) tokenizers ignore do_basic_tokenize=False for some Chinese BERT tokenizers — the word-segmentation flag may silently not apply, changing tokenization of Chinese text.

Source

Thrown at hanlp/layers/transformers/pt_imports.py:64

            transformer = pretrained_model_name_or_path.transformer
        additional_config = dict()
        if transformer.startswith('voidful/albert_chinese_') or transformer.startswith('uer/albert'):
            cls = BertTokenizer
        elif transformer == 'cl-tohoku/bert-base-japanese-char':
            # Since it's char level model, it's OK to use char level tok instead of fugashi
            # from hanlp.utils.lang.ja.bert_tok import BertJapaneseTokenizerFast
            # cls = BertJapaneseTokenizerFast
            from transformers import BertJapaneseTokenizer
            cls = BertJapaneseTokenizer
            # from transformers import BertTokenizerFast
            # cls = BertTokenizerFast
            additional_config['word_tokenizer_type'] = 'basic'
        elif transformer == "Langboat/mengzi-bert-base":
            cls = BertTokenizerFast if use_fast else BertTokenizer
        else:
            cls = AutoTokenizer
        if use_fast and not do_basic_tokenize:
            warnings.warn('`do_basic_tokenize=False` might not work when `use_fast=True`')
        tokenizer = cls.from_pretrained(get_tokenizer_mirror(transformer), use_fast=use_fast,
                                        do_basic_tokenize=do_basic_tokenize,
                                        **additional_config)
        tokenizer.name_or_path = transformer
        return tokenizer

View on GitHub (pinned to ddb1299bdd)

Solutions

1. Use use_fast=False when do_basic_tokenize=False matters
2. Or pre-tokenize/pre-segment text yourself and rely only on subword tokenization
3. If basic tokenization is fine, set do_basic_tokenize=True and ignore the warning

Example fix

# before
tok = Tokenizer.from_pretrained('bert-base-chinese', use_fast=True, do_basic_tokenize=False)
# after
tok = Tokenizer.from_pretrained('bert-base-chinese', use_fast=False, do_basic_tokenize=False)

if use_fast and not do_basic_tokenize:
    use_fast = False  # flag only honored by slow tokenizers

def needs_slow_tokenizer(do_basic_tokenize: bool, use_fast: bool) -> bool:
    return do_basic_tokenize is False and use_fast

When it happens

Trigger: Calling get_tokenizer/Tokenizer.from_pretrained with use_fast=True and do_basic_tokenize=False, e.g. for word-level Chinese BERT pipelines that rely on pre-segmented input.

Common situations: Switching a pipeline from slow to fast tokenizers for speed; trying to make the tokenizer treat whole words/sentences as single tokens for Chinese BERT.

Related errors

• transformers has its own tagger, not need to convert idx for
• transformers has its own tagger, not need to convert idx for
• Hugging Face 🤗 Transformers failed to download because your
• The class `PretrainedBartModel` has been depreciated, please

AI-assisted analysis of hankcs/HanLP@ddb1299bdd (2026-08-27). Data as JSON: /api/errors/234f951d3bf5b7e2. Report an issue: GitHub.