hankcs/HanLP · error · NotImplementedError

Unsupported tagging scheme {tagging_scheme}.

Error message

Unsupported tagging scheme {tagging_scheme}.

What it means

When building tokenization training data, generate_tags_for_subtokens only implements the BMES and BI tagging schemes for grouping subtokens into words. Any other scheme string (e.g. 'BIO', 'BIOES', 'bieos') raises NotImplementedError.

Source

Thrown at hanlp/datasets/tokenization/loaders/txt.py:111

         subtoken offsets grouped by each token.
        tagging_scheme:

    Returns:

    """
    # We could use token_token_span but we don't want token_token_span in the batch
    subtokens_group = sample.get('token_subtoken_offsets_group', None)
    sample['raw_token'] = sample['token']
    tokens = sample.get('token_') or sample['token']

    if subtokens_group:
        sample['token'] = subtokens_group_to_subtokens(tokens, subtokens_group)
        if tagging_scheme == 'BMES':
            sample['tag'] = words_to_bmes(subtokens_group)
        elif tagging_scheme == 'BI':
            sample['tag'] = words_to_bi(subtokens_group)
        else:
            raise NotImplementedError(f'Unsupported tagging scheme {tagging_scheme}.')
    else:
        sample['token'] = subtoken_offsets_to_subtokens(tokens, sample['token_subtoken_offsets'])
    return sample


def subtoken_offsets_to_subtokens(text, token_subtoken_offsets):
    results = []
    for b, e in token_subtoken_offsets:
        results.append(text[b:e])
    return results


def subtokens_group_to_subtokens(tokens, subtoken_offsets_group):
    results = []
    for subtoken_offsets, token in zip(subtoken_offsets_group, tokens):
        for b, e in subtoken_offsets:
            results.append(token[b:e])
    return results

View on GitHub (pinned to ddb1299bdd)

Solutions

1. Use tagging_scheme='BMES' (HanLP tokenization default) or 'BI'
2. If you need another scheme, convert tags post-hoc from BMES (B/M/E/S → BIO/BIOES mapping is mechanical) or patch the loader

Example fix

# before
sample = generate_tags_for_subtokens(sample, tagging_scheme='BIOES')
# after
sample = generate_tags_for_subtokens(sample, tagging_scheme='BMES')

assert tagging_scheme in ('BMES', 'BI'), f'unsupported tagging scheme {tagging_scheme}'

When it happens

Trigger: Calling generate_tags_for_subtokens (via txt dataset loading for tokenization) with tagging_scheme set to a scheme other than 'BMES' or 'BI', such as the common 'BIO'/'BIOES' used in NER.

Common situations: Porting NER-style scheme names into tokenization config; assuming BIOES is supported because other HanLP components use it.

Related errors

• transformers has its own tagger, not need to convert idx for
• transformers has its own tagger, not need to convert idx for
• error
• output ({}) must be of type bool or str
• Call fit or load before evaluate.

AI-assisted analysis of hankcs/HanLP@ddb1299bdd (2026-08-27). Data as JSON: /api/errors/1da23f5c63b66660. Report an issue: GitHub.