hankcs/HanLP · warning

The tag scheme for {self.vocabs.tag.idx_to_token} might be I

Error message

The tag scheme for {self.vocabs.tag.idx_to_token} might be IOB1 or IOB2 but we are using IOB2 by default. Please set tagging_scheme="IOB1" or tagging_scheme="BIO" to get rid of this warning.

What it means

The tagger lazily guesses the tagging scheme from the tag vocab; when the guessed scheme is 'BIO' (vocab has B- and I- but no clear IOB1 markers) it warns that the data might actually be IOB1 and decoding will assume IOB2 by default — which can mis-merge adjacent same-type spans.

Source

Thrown at hanlp/components/taggers/tagger.py:199

            pred_ids = pred_ids.tolist()
        sents = batch.get(f'{self.config.token_key}_')
        if not sents:
            sents = batch[self.config.token_key]
        dict_tags: DictInterface = self.dict_tags
        for each, sent in zip(pred_ids, sents):
            tags = [vocab[id] for id in each[:len(sent)]]
            if dict_tags:
                for begin, end, label in dict_tags.tokenize(sent):
                    tags[begin:end] = label
            yield tags

    @property
    def tagging_scheme(self):
        tagging_scheme = self.config.tagging_scheme
        if not tagging_scheme:
            self.config.tagging_scheme = tagging_scheme = guess_tagging_scheme(self.vocabs.tag.idx_to_token)
            if tagging_scheme == 'BIO':
                warnings.warn(f'The tag scheme for {self.vocabs.tag.idx_to_token} might be IOB1 or IOB2 '
                              f'but we are using IOB2 by default. Please set tagging_scheme="IOB1" or tagging_scheme="BIO" '
                              f'to get rid of this warning.')
        return tagging_scheme

    @property
    def dict_tags(self) -> DictInterface:
        r""" A custom dictionary to override predicted tags by performing longest-prefix-matching.

        Examples:
            >>> pos.dict_tags = {'HanLP': 'state-of-the-art-tool'} # Force 'HanLP' to be 'state-of-the-art-tool'
            >>> tagger("HanLP为生产环境带来次世代最先进的多语种NLP技术。")
                # HanLP/state-of-the-art-tool 为/P 生产/NN 环境/NN 带来/VV 次世代/NN 最/AD 先进/VA 的/DEC 多语种/NN NLP/NR 技术/NN 。/PU
            >>> pos.dict_tags = {('的', '希望'): ('补语成分', '名词'), '希望': '动词'} # Conditional matching
            >>> tagger("我的希望是希望张晚霞的背影被晚霞映红。")
                # 我/PN 的/补语成分 希望/名词 是/VC 希望/动词 张晚霞/NR 的/DEG 背影/NN 被/LB 晚霞/NN 映红/VV 。/PU
        """
        return self.config.get('dict_tags', None)

View on GitHub (pinned to ddb1299bdd)

Solutions

1. Determine your training data's scheme (IOB1 vs IOB2/BIO) and set tagging_scheme explicitly on the component/config to silence and disambiguate
2. If predictions incorrectly merge adjacent same-type entities, switch to tagging_scheme='IOB1'
3. Ignore the warning if IOB2 decoding gives correct spans for your data

Example fix

# before
tagger = hanlp.load(hanlp.pretrained.ner.X)  # warns, assumes IOB2
# after
tagger.config.tagging_scheme = 'IOB1'  # or 'BIO'

from hanlp.utils.span_util import guess_tagging_scheme
scheme = guess_tagging_scheme(vocab.idx_to_token)
if scheme == 'BIO':
    print('Ambiguous IOB1/IOB2 — verify training data annotation')

When it happens

Trigger: Loading a tagger whose vocabs.tag.idx_to_token contains B-/I- tags (IOB1-ambiguous) without an explicit tagging_scheme in config; the warning fires on first access of the tagging_scheme property (e.g. at predict time).

Common situations: Using a model trained on IOB1-annotated data (e.g. OntoNotes-style) with default settings; fine-tuned checkpoints missing the tagging_scheme config key.

Related errors

• Invalid encoding {encoding} passed to 'to_bioul'.
• Unsupported tagging scheme {tagging_scheme}.
• " ".join(tag_sequence)

AI-assisted analysis of hankcs/HanLP@ddb1299bdd (2026-08-27). Data as JSON: /api/errors/20e3c566d68aee16. Report an issue: GitHub.