fxsjy/jieba · critical · ValueError

invalid POS dictionary entry in %s at Line %s: %s

Error message

invalid POS dictionary entry in %s at Line %s: %s

What it means

posseg's load_word_tag parses the POS dictionary expecting exactly 'word tag' separated by a single space per line (line.split(' ')). Any line that raises during strip/decode/split — wrong encoding, more than one space, or split not yielding 3 values — is reported as an invalid POS dictionary entry with file, line number, and content.

Source

Thrown at jieba/posseg/__init__.py:108

            raise NotImplementedError
        return getattr(self.tokenizer, name)

    def initialize(self, dictionary=None):
        self.tokenizer.initialize(dictionary)
        self.load_word_tag(self.tokenizer.get_dict_file())

    def load_word_tag(self, f):
        self.word_tag_tab = {}
        f_name = resolve_filename(f)
        for lineno, line in enumerate(f, 1):
            try:
                line = line.strip().decode("utf-8")
                if not line:
                    continue
                word, _, tag = line.split(" ")
                self.word_tag_tab[word] = tag
            except Exception:
                raise ValueError(
                    'invalid POS dictionary entry in %s at Line %s: %s' % (f_name, lineno, line))
        f.close()

    def makesure_userdict_loaded(self):
        if self.tokenizer.user_word_tag_tab:
            self.word_tag_tab.update(self.tokenizer.user_word_tag_tab)
            self.tokenizer.user_word_tag_tab = {}

    def __cut(self, sentence):
        prob, pos_list = viterbi(
            sentence, char_state_tab_P, start_P, trans_P, emit_P)
        begin, nexti = 0, 0

        for i, char in enumerate(sentence):
            pos = pos_list[i][0]
            if pos == 'B':
                begin = i
            elif pos == 'E':

View on GitHub (pinned to 67fa2e36e7)

Solutions

  1. Fix the reported line: exactly 'word TAG', single space separator, UTF-8, no BOM
  2. Normalize the file programmatically: split on any whitespace and re-join with a single space
  3. Restore the default dict.txt.pos via pip install --force-reinstall jieba
  4. Never pass a plain frequency dict (word freq) to posseg.initialize — it must include POS tags

Example fix

# before
# posdict line: '北京  ns  extra'  -> ValueError
jieba.posseg.initialize('posdict.txt')

# after
with open('posdict.txt', encoding='utf-8') as f:
    rows = [ln.split()[:2] for ln in f if ln.strip()]
with open('posdict_fixed.txt', 'w', encoding='utf-8') as f:
    f.write('\n'.join('%s %s' % (w, t) for w, t in rows))
jieba.posseg.initialize('posdict_fixed.txt')
Defensive patterns

Strategy: validation

Validate before calling

import re
def posdict_ok(path):
    pat = re.compile(r'^\S+ [a-z]+\s*$', re.IGNORECASE)
    with open(path, encoding='utf-8') as f:
        for i, ln in enumerate(f, 1):
            if ln.strip() and not pat.match(ln):
                return False, i, ln
    return True, None, None

Try / catch

try:
    jieba.posseg.initialize(p)
except ValueError as e:
    # message contains file, line number, offending line
    fix_pos_line(e); jieba.posseg.initialize(p)

Prevention

When it happens

Trigger: Initializing jieba.posseg with a custom dictionary whose lines aren't 'word TAG' (single space), contain multiple spaces, are non-UTF-8, or where the default dict.txt.pos is corrupted. Triggered on first posseg call or posseg.initialize(dictionary).

Common situations: Custom dictionaries generated with tabs or multiple spaces; GBK-encoded POS dicts; tabs instead of spaces; corrupted default dict.txt.pos after a bad install; reusing a plain jieba dict (no tag column) as a POS dict.

Related errors

AI-assisted analysis of fxsjy/jieba@67fa2e36e7 (2026-08-27). Data as JSON: /api/errors/e4942f13ff58e0c2. Report an issue: GitHub.