hankcs/HanLP · error · ValueError
{} contains None or zero-length word {}
Error message
{} contains None or zero-length word {} What it means
Identical helper to the one in txt_tf: span_util.words_to_bmes converts a word list to BMES tags and rejects any None or empty-string word, quoting both the word and the full list.
Source
Thrown at hanlp/utils/span_util.py:21
# Date: 2020-06-12 20:34
import warnings
from typing import Dict, List, Tuple, Callable, Set, Optional
def generate_words_per_line(file_path):
with open(file_path, encoding='utf-8') as src:
for line in src:
cells = line.strip().split()
if not cells:
continue
yield cells
def words_to_bmes(words):
tags = []
for w in words:
if not w:
raise ValueError('{} contains None or zero-length word {}'.format(str(words), w))
if len(w) == 1:
tags.append('S')
else:
tags.extend(['B'] + ['M'] * (len(w) - 2) + ['E'])
return tags
def words_to_bi(words):
tags = []
for w in words:
if not w:
raise ValueError('{} contains None or zero-length word {}'.format(str(words), w))
tags.extend(['B'] + ['I'] * (len(w) - 1))
return tags
def bmes_to_words(chars, tags):
result = []View on GitHub (pinned to ddb1299bdd)
Solutions
- Pre-filter: [w for w in words if w] before conversion.
- Fix the upstream tokenizer/corpus so empty tokens are never emitted.
- Log the offending sentence (it is included in the message) to locate the corpus line and repair it.
Example fix
# before tags = words_to_bmes(words) # after tags = words_to_bmes([w for w in words if w])
Defensive patterns
Strategy: validation
Validate before calling
assert all(isinstance(w, str) and w for w in words)
Type guard
def valid_words(words):
return all(isinstance(w, str) and len(w) > 0 for w in words) Prevention
- Filter empty tokens at ingestion.
- Lint corpora for empty tokens in a data-validation step.
When it happens
Trigger: Calling words_to_bmes (or code paths like span tag generation) with a token list containing '' or None — typically from tokenization that produced empty matches.
Common situations: Empty regex captures, consecutive separators in corpora, normalization reducing tokens to '', None from failed dict lookups during preprocessing.
Related errors
AI-assisted analysis of hankcs/HanLP@ddb1299bdd (2026-08-27).
Data as JSON: /api/errors/b4f6668059b56c80.
Report an issue: GitHub.