{"record":{"id":"f98ad529ea0dbf75","repo":"hankcs/HanLP","slug":"contains-none-or-zero-length-word","errorCode":null,"errorMessage":"{} contains None or zero-length word {}","messagePattern":"(.+?) contains None or zero-length word (.+?)","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"hanlp/transform/txt_tf.py","lineNumber":31,"sourceCode":"from hanlp.utils.lang.zh.char_table import CharTable\nfrom hanlp.utils.span_util import bmes_of, bmes_to_words\nfrom hanlp.utils.string_util import split_long_sent\n\n\ndef generate_words_per_line(file_path):\n    with open(file_path, encoding='utf-8') as src:\n        for line in src:\n            cells = line.strip().split()\n            if not cells:\n                continue\n            yield cells\n\n\ndef words_to_bmes(words):\n    tags = []\n    for w in words:\n        if not w:\n            raise ValueError('{} contains None or zero-length word {}'.format(str(words), w))\n        if len(w) == 1:\n            tags.append('S')\n        else:\n            tags.extend(['B'] + ['M'] * (len(w) - 2) + ['E'])\n    return tags\n\n\ndef extract_ngram_features_and_tags(sentence, bigram_only=False, window_size=4, segmented=True):\n    \"\"\"\n    Feature extraction for windowed approaches\n    See Also https://github.com/chqiwang/convseg/\n    Parameters\n    ----------\n    sentence\n    bigram_only\n    window_size\n    segmented\n","sourceCodeStart":13,"sourceCodeEnd":49,"githubUrl":"https://github.com/hankcs/HanLP/blob/ddb1299bddff079e447af52ec12549c50636bfa8/hanlp/transform/txt_tf.py#L13-L49","documentation":"words_to_bmes builds BMES segmentation tags from a list of words; an empty string or None word has no valid tag sequence, so it raises ValueError naming the offending word and its containing list.","triggerScenarios":"Calling words_to_bmes (or a transform that uses it, e.g. building BMES training data for CWS) with a sentence containing '' or None as a word — often from a bad corpus split, an empty regex capture, or a whitespace tokenization artifact.","commonSituations":"Dirty corpora with empty tokens between consecutive delimiters; tokens stripped to '' by normalization; None from a failed lookup in a preprocessing step.","solutions":["Find and fix the empty/None tokens in the source corpus at the quoted location.","Pre-filter sentences: drop or merge empty words before conversion ([w for w in words if w]).","If the corpus is trusted, audit your preprocessing (split/regex/strip) for producing empties."],"exampleFix":"# before\ntags = words_to_bmes(words)\n# after\nwords = [w for w in words if w]\ntags = words_to_bmes(words)","handlingStrategy":"validation","validationCode":"assert all(isinstance(w, str) and w for w in words), f'bad word list: {words}'","typeGuard":"def valid_words(words):\n    return all(isinstance(w, str) and len(w) > 0 for w in words)","tryCatchPattern":null,"preventionTips":["Filter empty tokens at corpus load time.","Run a corpus lint pass (empty-token check) before training preprocessing."],"tags":["data-quality","segmentation","validation"],"backgroundTag":"empty-or-null-value","analyzedSha":"ddb1299bddff079e447af52ec12549c50636bfa8","analyzedAt":"2026-08-27T03:36:54.287Z","schemaVersion":2},"datasetVersion":"2026-08-27T08:17:20.692Z"}