{"record":{"id":"234f951d3bf5b7e2","repo":"hankcs/HanLP","slug":"do-basic-tokenize-false-might-not-work-when-use","errorCode":null,"errorMessage":"`do_basic_tokenize=False` might not work when `use_fast=True`","messagePattern":"`do_basic_tokenize=False` might not work when `use_fast=True`","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"hanlp/layers/transformers/pt_imports.py","lineNumber":64,"sourceCode":"            transformer = pretrained_model_name_or_path.transformer\n        additional_config = dict()\n        if transformer.startswith('voidful/albert_chinese_') or transformer.startswith('uer/albert'):\n            cls = BertTokenizer\n        elif transformer == 'cl-tohoku/bert-base-japanese-char':\n            # Since it's char level model, it's OK to use char level tok instead of fugashi\n            # from hanlp.utils.lang.ja.bert_tok import BertJapaneseTokenizerFast\n            # cls = BertJapaneseTokenizerFast\n            from transformers import BertJapaneseTokenizer\n            cls = BertJapaneseTokenizer\n            # from transformers import BertTokenizerFast\n            # cls = BertTokenizerFast\n            additional_config['word_tokenizer_type'] = 'basic'\n        elif transformer == \"Langboat/mengzi-bert-base\":\n            cls = BertTokenizerFast if use_fast else BertTokenizer\n        else:\n            cls = AutoTokenizer\n        if use_fast and not do_basic_tokenize:\n            warnings.warn('`do_basic_tokenize=False` might not work when `use_fast=True`')\n        tokenizer = cls.from_pretrained(get_tokenizer_mirror(transformer), use_fast=use_fast,\n                                        do_basic_tokenize=do_basic_tokenize,\n                                        **additional_config)\n        tokenizer.name_or_path = transformer\n        return tokenizer\n","sourceCodeStart":46,"sourceCodeEnd":70,"githubUrl":"https://github.com/hankcs/HanLP/blob/ddb1299bddff079e447af52ec12549c50636bfa8/hanlp/layers/transformers/pt_imports.py#L46-L70","documentation":"Warns that transformers' fast (Rust) tokenizers ignore do_basic_tokenize=False for some Chinese BERT tokenizers — the word-segmentation flag may silently not apply, changing tokenization of Chinese text.","triggerScenarios":"Calling get_tokenizer/Tokenizer.from_pretrained with use_fast=True and do_basic_tokenize=False, e.g. for word-level Chinese BERT pipelines that rely on pre-segmented input.","commonSituations":"Switching a pipeline from slow to fast tokenizers for speed; trying to make the tokenizer treat whole words/sentences as single tokens for Chinese BERT.","solutions":["Use use_fast=False when do_basic_tokenize=False matters","Or pre-tokenize/pre-segment text yourself and rely only on subword tokenization","If basic tokenization is fine, set do_basic_tokenize=True and ignore the warning"],"exampleFix":"# before\ntok = Tokenizer.from_pretrained('bert-base-chinese', use_fast=True, do_basic_tokenize=False)\n# after\ntok = Tokenizer.from_pretrained('bert-base-chinese', use_fast=False, do_basic_tokenize=False)","handlingStrategy":"validation","validationCode":"if use_fast and not do_basic_tokenize:\n    use_fast = False  # flag only honored by slow tokenizers","typeGuard":"def needs_slow_tokenizer(do_basic_tokenize: bool, use_fast: bool) -> bool:\n    return do_basic_tokenize is False and use_fast","tryCatchPattern":null,"preventionTips":["Use use_fast=False when do_basic_tokenize=False","Pre-segment Chinese text upstream instead of relying on tokenizer flags"],"tags":["tokenizer","transformers","chinese"],"backgroundTag":"tokenizer-flag-ignored-warning","analyzedSha":"ddb1299bddff079e447af52ec12549c50636bfa8","analyzedAt":"2026-08-27T03:36:54.287Z","schemaVersion":2},"datasetVersion":"2026-08-27T08:17:20.692Z"}