{"record":{"id":"20e3c566d68aee16","repo":"hankcs/HanLP","slug":"the-tag-scheme-for-self-vocabs-tag-idx-to-token","errorCode":null,"errorMessage":"The tag scheme for {self.vocabs.tag.idx_to_token} might be IOB1 or IOB2 but we are using IOB2 by default. Please set tagging_scheme=\"IOB1\" or tagging_scheme=\"BIO\" to get rid of this warning.","messagePattern":"The tag scheme for (.+?) might be IOB1 or IOB2 but we are using IOB2 by default\\. Please set tagging_scheme=\"IOB1\" or tagging_scheme=\"BIO\" to get rid of this warning\\.","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"hanlp/components/taggers/tagger.py","lineNumber":199,"sourceCode":"            pred_ids = pred_ids.tolist()\n        sents = batch.get(f'{self.config.token_key}_')\n        if not sents:\n            sents = batch[self.config.token_key]\n        dict_tags: DictInterface = self.dict_tags\n        for each, sent in zip(pred_ids, sents):\n            tags = [vocab[id] for id in each[:len(sent)]]\n            if dict_tags:\n                for begin, end, label in dict_tags.tokenize(sent):\n                    tags[begin:end] = label\n            yield tags\n\n    @property\n    def tagging_scheme(self):\n        tagging_scheme = self.config.tagging_scheme\n        if not tagging_scheme:\n            self.config.tagging_scheme = tagging_scheme = guess_tagging_scheme(self.vocabs.tag.idx_to_token)\n            if tagging_scheme == 'BIO':\n                warnings.warn(f'The tag scheme for {self.vocabs.tag.idx_to_token} might be IOB1 or IOB2 '\n                              f'but we are using IOB2 by default. Please set tagging_scheme=\"IOB1\" or tagging_scheme=\"BIO\" '\n                              f'to get rid of this warning.')\n        return tagging_scheme\n\n    @property\n    def dict_tags(self) -> DictInterface:\n        r\"\"\" A custom dictionary to override predicted tags by performing longest-prefix-matching.\n\n        Examples:\n            >>> pos.dict_tags = {'HanLP': 'state-of-the-art-tool'} # Force 'HanLP' to be 'state-of-the-art-tool'\n            >>> tagger(\"HanLP为生产环境带来次世代最先进的多语种NLP技术。\")\n                # HanLP/state-of-the-art-tool 为/P 生产/NN 环境/NN 带来/VV 次世代/NN 最/AD 先进/VA 的/DEC 多语种/NN NLP/NR 技术/NN 。/PU\n            >>> pos.dict_tags = {('的', '希望'): ('补语成分', '名词'), '希望': '动词'} # Conditional matching\n            >>> tagger(\"我的希望是希望张晚霞的背影被晚霞映红。\")\n                # 我/PN 的/补语成分 希望/名词 是/VC 希望/动词 张晚霞/NR 的/DEG 背影/NN 被/LB 晚霞/NN 映红/VV 。/PU\n        \"\"\"\n        return self.config.get('dict_tags', None)\n","sourceCodeStart":181,"sourceCodeEnd":217,"githubUrl":"https://github.com/hankcs/HanLP/blob/ddb1299bddff079e447af52ec12549c50636bfa8/hanlp/components/taggers/tagger.py#L181-L217","documentation":"The tagger lazily guesses the tagging scheme from the tag vocab; when the guessed scheme is 'BIO' (vocab has B- and I- but no clear IOB1 markers) it warns that the data might actually be IOB1 and decoding will assume IOB2 by default — which can mis-merge adjacent same-type spans.","triggerScenarios":"Loading a tagger whose vocabs.tag.idx_to_token contains B-/I- tags (IOB1-ambiguous) without an explicit tagging_scheme in config; the warning fires on first access of the tagging_scheme property (e.g. at predict time).","commonSituations":"Using a model trained on IOB1-annotated data (e.g. OntoNotes-style) with default settings; fine-tuned checkpoints missing the tagging_scheme config key.","solutions":["Determine your training data's scheme (IOB1 vs IOB2/BIO) and set tagging_scheme explicitly on the component/config to silence and disambiguate","If predictions incorrectly merge adjacent same-type entities, switch to tagging_scheme='IOB1'","Ignore the warning if IOB2 decoding gives correct spans for your data"],"exampleFix":"# before\ntagger = hanlp.load(hanlp.pretrained.ner.X)  # warns, assumes IOB2\n# after\ntagger.config.tagging_scheme = 'IOB1'  # or 'BIO'","handlingStrategy":"validation","validationCode":"from hanlp.utils.span_util import guess_tagging_scheme\nscheme = guess_tagging_scheme(vocab.idx_to_token)\nif scheme == 'BIO':\n    print('Ambiguous IOB1/IOB2 — verify training data annotation')","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Set tagging_scheme in config when training/saving taggers","Check whether adjacent same-type entities merge incorrectly — sign of IOB1 data"],"tags":["ner","tagging-scheme","decoding"],"backgroundTag":"ambiguous-tagging-scheme-warning","analyzedSha":"ddb1299bddff079e447af52ec12549c50636bfa8","analyzedAt":"2026-08-27T03:36:54.287Z","schemaVersion":2},"datasetVersion":"2026-08-27T08:17:20.692Z"}