{"record":{"id":"1da23f5c63b66660","repo":"hankcs/HanLP","slug":"unsupported-tagging-scheme-tagging-scheme","errorCode":null,"errorMessage":"Unsupported tagging scheme {tagging_scheme}.","messagePattern":"Unsupported tagging scheme (.+?)\\.","errorType":"exception","errorClass":"NotImplementedError","httpStatus":null,"severity":"error","filePath":"hanlp/datasets/tokenization/loaders/txt.py","lineNumber":111,"sourceCode":"         subtoken offsets grouped by each token.\n        tagging_scheme:\n\n    Returns:\n\n    \"\"\"\n    # We could use token_token_span but we don't want token_token_span in the batch\n    subtokens_group = sample.get('token_subtoken_offsets_group', None)\n    sample['raw_token'] = sample['token']\n    tokens = sample.get('token_') or sample['token']\n\n    if subtokens_group:\n        sample['token'] = subtokens_group_to_subtokens(tokens, subtokens_group)\n        if tagging_scheme == 'BMES':\n            sample['tag'] = words_to_bmes(subtokens_group)\n        elif tagging_scheme == 'BI':\n            sample['tag'] = words_to_bi(subtokens_group)\n        else:\n            raise NotImplementedError(f'Unsupported tagging scheme {tagging_scheme}.')\n    else:\n        sample['token'] = subtoken_offsets_to_subtokens(tokens, sample['token_subtoken_offsets'])\n    return sample\n\n\ndef subtoken_offsets_to_subtokens(text, token_subtoken_offsets):\n    results = []\n    for b, e in token_subtoken_offsets:\n        results.append(text[b:e])\n    return results\n\n\ndef subtokens_group_to_subtokens(tokens, subtoken_offsets_group):\n    results = []\n    for subtoken_offsets, token in zip(subtoken_offsets_group, tokens):\n        for b, e in subtoken_offsets:\n            results.append(token[b:e])\n    return results","sourceCodeStart":93,"sourceCodeEnd":129,"githubUrl":"https://github.com/hankcs/HanLP/blob/ddb1299bddff079e447af52ec12549c50636bfa8/hanlp/datasets/tokenization/loaders/txt.py#L93-L129","documentation":"When building tokenization training data, generate_tags_for_subtokens only implements the BMES and BI tagging schemes for grouping subtokens into words. Any other scheme string (e.g. 'BIO', 'BIOES', 'bieos') raises NotImplementedError.","triggerScenarios":"Calling generate_tags_for_subtokens (via txt dataset loading for tokenization) with tagging_scheme set to a scheme other than 'BMES' or 'BI', such as the common 'BIO'/'BIOES' used in NER.","commonSituations":"Porting NER-style scheme names into tokenization config; assuming BIOES is supported because other HanLP components use it.","solutions":["Use tagging_scheme='BMES' (HanLP tokenization default) or 'BI'","If you need another scheme, convert tags post-hoc from BMES (B/M/E/S → BIO/BIOES mapping is mechanical) or patch the loader"],"exampleFix":"# before\nsample = generate_tags_for_subtokens(sample, tagging_scheme='BIOES')\n# after\nsample = generate_tags_for_subtokens(sample, tagging_scheme='BMES')","handlingStrategy":"validation","validationCode":"assert tagging_scheme in ('BMES', 'BI'), f'unsupported tagging scheme {tagging_scheme}'","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Stick to BMES/BI for HanLP tokenization data; convert other schemes to BMES upstream"],"tags":["python","tokenization","tagging-scheme","not-implemented"],"backgroundTag":"unsupported-tagging-scheme","analyzedSha":"ddb1299bddff079e447af52ec12549c50636bfa8","analyzedAt":"2026-08-27T03:36:54.287Z","schemaVersion":2},"datasetVersion":"2026-08-27T08:17:20.692Z"}