{"record":{"id":"67e16380cbc20424","repo":"fxsjy/jieba","slug":"invalid-dictionary-entry-in-s-at-line-s-s","errorCode":null,"errorMessage":"invalid dictionary entry in %s at Line %s: %s","messagePattern":"invalid dictionary entry in (.+?) at Line (.+?): (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"critical","filePath":"jieba/__init__.py","lineNumber":88,"sourceCode":"\n    @staticmethod\n    def gen_pfdict(f):\n        lfreq = {}\n        ltotal = 0\n        f_name = resolve_filename(f)\n        for lineno, line in enumerate(f, 1):\n            try:\n                line = line.strip().decode('utf-8')\n                word, freq = line.split(' ')[:2]\n                freq = int(freq)\n                lfreq[word] = freq\n                ltotal += freq\n                for ch in xrange(len(word)):\n                    wfrag = word[:ch + 1]\n                    if wfrag not in lfreq:\n                        lfreq[wfrag] = 0\n            except ValueError:\n                raise ValueError(\n                    'invalid dictionary entry in %s at Line %s: %s' % (f_name, lineno, line))\n        f.close()\n        return lfreq, ltotal\n\n    def initialize(self, dictionary=None):\n        if dictionary:\n            abs_path = _get_abs_path(dictionary)\n            if self.dictionary == abs_path and self.initialized:\n                return\n            else:\n                self.dictionary = abs_path\n                self.initialized = False\n        else:\n            abs_path = self.dictionary\n\n        with self.lock:\n            try:\n                with DICT_WRITING[abs_path]:","sourceCodeStart":70,"sourceCodeEnd":106,"githubUrl":"https://github.com/fxsjy/jieba/blob/67fa2e36e72f69d9134b8a1037b83fbb070b9775/jieba/__init__.py#L70-L106","documentation":"Raised by jieba's gen_pfdict while parsing the main dictionary file. Each dictionary line must be 'word frequency [tag]'; if word/freq parsing or int(freq) raises ValueError, jieba reports the offending file, line number, and raw line. It almost always means the dict.txt (or a custom dictionary passed to initialize/set_dictionary) is malformed or has been corrupted/truncated.","triggerScenarios":"Calling dt.initialize(dictionary) or jieba.set_dictionary() with a custom dict whose lines are not 'word freq' pairs, have non-integer frequencies, contain stray characters, or a file with wrong line endings/embedded content. Also triggered by a truncated download of dict.txt.","commonSituations":"Bundled dict.txt corrupted by a partial pip install or git checkout; hand-edited dictionaries with blank-but-whitespace lines or comments; dictionaries saved with BOM or GBK encoding producing garbage tokens; wrong file passed to set_dictionary.","solutions":["Open the named file at the reported line and fix/remove the malformed entry (must be 'word freq' with an integer frequency)","If editing caused it, re-check line endings and encoding (UTF-8, no BOM) and remove comment/blank lines with content","Reinstall jieba (pip install --force-reinstall jieba) to restore a pristine dict.txt","If using a custom dict, validate every line with a regex like ^(\\S+)\\s+(\\d+)(\\s+\\S+)?$ before passing it"],"exampleFix":"# before\njieba.set_dictionary('my_broken_dict.txt')\n\n# after\nimport re\nwith open('my_broken_dict.txt', encoding='utf-8') as f:\n    for i, ln in enumerate(f, 1):\n        assert re.match(r'^\\S+\\s+\\d+(\\s+\\S+)?\\s*$', ln), (i, ln)\njieba.set_dictionary('my_broken_dict.txt')","handlingStrategy":"validation","validationCode":"import re\ndef dict_ok(path):\n    pat = re.compile(r'^\\S+\\s+\\d+(\\s+\\S+)?\\s*$')\n    with open(path, encoding='utf-8') as f:\n        for i, ln in enumerate(f, 1):\n            if not pat.match(ln):\n                return False, i, ln\n    return True, None, None","typeGuard":null,"tryCatchPattern":"try:\n    jieba.initialize(mydict)\nexcept ValueError as e:\n    # e message contains file, line number, and offending line\n    lineno, line = parse_from_message(e)\n    fix_line(mydict, lineno); jieba.initialize(mydict)","preventionTips":["Generate dictionaries programmatically with '%s %d' % (word, freq) formatting","Keep dictionary files UTF-8 without BOM","Validate dict files in CI before deployment"],"tags":["jieba","dictionary","validation","unicode"],"backgroundTag":"malformed-data-file","analyzedSha":"67fa2e36e72f69d9134b8a1037b83fbb070b9775","analyzedAt":"2026-08-27T11:28:25.101Z","schemaVersion":2},"datasetVersion":"2026-08-27T13:17:12.746Z"}