fxsjy/jieba · critical · ValueError

invalid dictionary entry in %s at Line %s: %s

Error message

invalid dictionary entry in %s at Line %s: %s

What it means

Raised by jieba's gen_pfdict while parsing the main dictionary file. Each dictionary line must be 'word frequency [tag]'; if word/freq parsing or int(freq) raises ValueError, jieba reports the offending file, line number, and raw line. It almost always means the dict.txt (or a custom dictionary passed to initialize/set_dictionary) is malformed or has been corrupted/truncated.

Source

Thrown at jieba/__init__.py:88

    @staticmethod
    def gen_pfdict(f):
        lfreq = {}
        ltotal = 0
        f_name = resolve_filename(f)
        for lineno, line in enumerate(f, 1):
            try:
                line = line.strip().decode('utf-8')
                word, freq = line.split(' ')[:2]
                freq = int(freq)
                lfreq[word] = freq
                ltotal += freq
                for ch in xrange(len(word)):
                    wfrag = word[:ch + 1]
                    if wfrag not in lfreq:
                        lfreq[wfrag] = 0
            except ValueError:
                raise ValueError(
                    'invalid dictionary entry in %s at Line %s: %s' % (f_name, lineno, line))
        f.close()
        return lfreq, ltotal

    def initialize(self, dictionary=None):
        if dictionary:
            abs_path = _get_abs_path(dictionary)
            if self.dictionary == abs_path and self.initialized:
                return
            else:
                self.dictionary = abs_path
                self.initialized = False
        else:
            abs_path = self.dictionary

        with self.lock:
            try:
                with DICT_WRITING[abs_path]:

View on GitHub (pinned to 67fa2e36e7)

Solutions

1. Open the named file at the reported line and fix/remove the malformed entry (must be 'word freq' with an integer frequency)
2. If editing caused it, re-check line endings and encoding (UTF-8, no BOM) and remove comment/blank lines with content
3. Reinstall jieba (pip install --force-reinstall jieba) to restore a pristine dict.txt
4. If using a custom dict, validate every line with a regex like ^(\S+)\s+(\d+)(\s+\S+)?$ before passing it

Example fix

# before
jieba.set_dictionary('my_broken_dict.txt')

# after
import re
with open('my_broken_dict.txt', encoding='utf-8') as f:
    for i, ln in enumerate(f, 1):
        assert re.match(r'^\S+\s+\d+(\s+\S+)?\s*$', ln), (i, ln)
jieba.set_dictionary('my_broken_dict.txt')

import re
def dict_ok(path):
    pat = re.compile(r'^\S+\s+\d+(\s+\S+)?\s*$')
    with open(path, encoding='utf-8') as f:
        for i, ln in enumerate(f, 1):
            if not pat.match(ln):
                return False, i, ln
    return True, None, None

try:
    jieba.initialize(mydict)
except ValueError as e:
    # e message contains file, line number, and offending line
    lineno, line = parse_from_message(e)
    fix_line(mydict, lineno); jieba.initialize(mydict)

When it happens

Trigger: Calling dt.initialize(dictionary) or jieba.set_dictionary() with a custom dict whose lines are not 'word freq' pairs, have non-integer frequencies, contain stray characters, or a file with wrong line endings/embedded content. Also triggered by a truncated download of dict.txt.

Common situations: Bundled dict.txt corrupted by a partial pip install or git checkout; hand-edited dictionaries with blank-but-whitespace lines or comments; dictionaries saved with BOM or GBK encoding producing garbage tokens; wrong file passed to set_dictionary.

Related errors

• invalid POS dictionary entry in %s at Line %s: %s
• jieba: the input parameter should be unicode.
• jieba: file does not exist: %s
• dictionary file %s must be utf-8
• jieba: parallel mode only supports posix system

AI-assisted analysis of fxsjy/jieba@67fa2e36e7 (2026-08-27). Data as JSON: /api/errors/67e16380cbc20424. Report an issue: GitHub.