hankcs/HanLP · error · ValueError

{} contains None or zero-length word {}

Error message

{} contains None or zero-length word {}

What it means

words_to_bmes builds BMES segmentation tags from a list of words; an empty string or None word has no valid tag sequence, so it raises ValueError naming the offending word and its containing list.

Source

Thrown at hanlp/transform/txt_tf.py:31

from hanlp.utils.lang.zh.char_table import CharTable
from hanlp.utils.span_util import bmes_of, bmes_to_words
from hanlp.utils.string_util import split_long_sent


def generate_words_per_line(file_path):
    with open(file_path, encoding='utf-8') as src:
        for line in src:
            cells = line.strip().split()
            if not cells:
                continue
            yield cells


def words_to_bmes(words):
    tags = []
    for w in words:
        if not w:
            raise ValueError('{} contains None or zero-length word {}'.format(str(words), w))
        if len(w) == 1:
            tags.append('S')
        else:
            tags.extend(['B'] + ['M'] * (len(w) - 2) + ['E'])
    return tags


def extract_ngram_features_and_tags(sentence, bigram_only=False, window_size=4, segmented=True):
    """
    Feature extraction for windowed approaches
    See Also https://github.com/chqiwang/convseg/
    Parameters
    ----------
    sentence
    bigram_only
    window_size
    segmented

View on GitHub (pinned to ddb1299bdd)

Solutions

1. Find and fix the empty/None tokens in the source corpus at the quoted location.
2. Pre-filter sentences: drop or merge empty words before conversion ([w for w in words if w]).
3. If the corpus is trusted, audit your preprocessing (split/regex/strip) for producing empties.

Example fix

# before
tags = words_to_bmes(words)
# after
words = [w for w in words if w]
tags = words_to_bmes(words)

assert all(isinstance(w, str) and w for w in words), f'bad word list: {words}'

def valid_words(words):
    return all(isinstance(w, str) and len(w) > 0 for w in words)

When it happens

Trigger: Calling words_to_bmes (or a transform that uses it, e.g. building BMES training data for CWS) with a sentence containing '' or None as a word — often from a bad corpus split, an empty regex capture, or a whitespace tokenization artifact.

Common situations: Dirty corpora with empty tokens between consecutive delimiters; tokens stripped to '' by normalization; None from a failed lookup in a preprocessing step.

Related errors

• {} contains None or zero-length word {}
• the first two dimensions of emissions and tags must match, g
• Unrecognized label encoding {self.label_encoding}
• Expect X to be 2 or 3 elements but got {repr(X)}
• Unknown data arrangement

AI-assisted analysis of hankcs/HanLP@ddb1299bdd (2026-08-27). Data as JSON: /api/errors/f98ad529ea0dbf75. Report an issue: GitHub.