{"record":{"id":"b4f6668059b56c80","repo":"hankcs/HanLP","slug":"contains-none-or-zero-length-word-b4f666","errorCode":null,"errorMessage":"{} contains None or zero-length word {}","messagePattern":"(.+?) contains None or zero-length word (.+?)","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"hanlp/utils/span_util.py","lineNumber":21,"sourceCode":"# Date: 2020-06-12 20:34\nimport warnings\nfrom typing import Dict, List, Tuple, Callable, Set, Optional\n\n\ndef generate_words_per_line(file_path):\n    with open(file_path, encoding='utf-8') as src:\n        for line in src:\n            cells = line.strip().split()\n            if not cells:\n                continue\n            yield cells\n\n\ndef words_to_bmes(words):\n    tags = []\n    for w in words:\n        if not w:\n            raise ValueError('{} contains None or zero-length word {}'.format(str(words), w))\n        if len(w) == 1:\n            tags.append('S')\n        else:\n            tags.extend(['B'] + ['M'] * (len(w) - 2) + ['E'])\n    return tags\n\n\ndef words_to_bi(words):\n    tags = []\n    for w in words:\n        if not w:\n            raise ValueError('{} contains None or zero-length word {}'.format(str(words), w))\n        tags.extend(['B'] + ['I'] * (len(w) - 1))\n    return tags\n\n\ndef bmes_to_words(chars, tags):\n    result = []","sourceCodeStart":3,"sourceCodeEnd":39,"githubUrl":"https://github.com/hankcs/HanLP/blob/ddb1299bddff079e447af52ec12549c50636bfa8/hanlp/utils/span_util.py#L3-L39","documentation":"Identical helper to the one in txt_tf: span_util.words_to_bmes converts a word list to BMES tags and rejects any None or empty-string word, quoting both the word and the full list.","triggerScenarios":"Calling words_to_bmes (or code paths like span tag generation) with a token list containing '' or None — typically from tokenization that produced empty matches.","commonSituations":"Empty regex captures, consecutive separators in corpora, normalization reducing tokens to '', None from failed dict lookups during preprocessing.","solutions":["Pre-filter: [w for w in words if w] before conversion.","Fix the upstream tokenizer/corpus so empty tokens are never emitted.","Log the offending sentence (it is included in the message) to locate the corpus line and repair it."],"exampleFix":"# before\ntags = words_to_bmes(words)\n# after\ntags = words_to_bmes([w for w in words if w])","handlingStrategy":"validation","validationCode":"assert all(isinstance(w, str) and w for w in words)","typeGuard":"def valid_words(words):\n    return all(isinstance(w, str) and len(w) > 0 for w in words)","tryCatchPattern":null,"preventionTips":["Filter empty tokens at ingestion.","Lint corpora for empty tokens in a data-validation step."],"tags":["data-quality","segmentation","span"],"backgroundTag":"empty-or-null-value","analyzedSha":"ddb1299bddff079e447af52ec12549c50636bfa8","analyzedAt":"2026-08-27T03:36:54.287Z","schemaVersion":2},"datasetVersion":"2026-08-27T08:17:20.692Z"}