{"record":{"id":"e4942f13ff58e0c2","repo":"fxsjy/jieba","slug":"invalid-pos-dictionary-entry-in-s-at-line-s-s","errorCode":null,"errorMessage":"invalid POS dictionary entry in %s at Line %s: %s","messagePattern":"invalid POS dictionary entry in (.+?) at Line (.+?): (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"critical","filePath":"jieba/posseg/__init__.py","lineNumber":108,"sourceCode":"            raise NotImplementedError\n        return getattr(self.tokenizer, name)\n\n    def initialize(self, dictionary=None):\n        self.tokenizer.initialize(dictionary)\n        self.load_word_tag(self.tokenizer.get_dict_file())\n\n    def load_word_tag(self, f):\n        self.word_tag_tab = {}\n        f_name = resolve_filename(f)\n        for lineno, line in enumerate(f, 1):\n            try:\n                line = line.strip().decode(\"utf-8\")\n                if not line:\n                    continue\n                word, _, tag = line.split(\" \")\n                self.word_tag_tab[word] = tag\n            except Exception:\n                raise ValueError(\n                    'invalid POS dictionary entry in %s at Line %s: %s' % (f_name, lineno, line))\n        f.close()\n\n    def makesure_userdict_loaded(self):\n        if self.tokenizer.user_word_tag_tab:\n            self.word_tag_tab.update(self.tokenizer.user_word_tag_tab)\n            self.tokenizer.user_word_tag_tab = {}\n\n    def __cut(self, sentence):\n        prob, pos_list = viterbi(\n            sentence, char_state_tab_P, start_P, trans_P, emit_P)\n        begin, nexti = 0, 0\n\n        for i, char in enumerate(sentence):\n            pos = pos_list[i][0]\n            if pos == 'B':\n                begin = i\n            elif pos == 'E':","sourceCodeStart":90,"sourceCodeEnd":126,"githubUrl":"https://github.com/fxsjy/jieba/blob/67fa2e36e72f69d9134b8a1037b83fbb070b9775/jieba/posseg/__init__.py#L90-L126","documentation":"posseg's load_word_tag parses the POS dictionary expecting exactly 'word tag' separated by a single space per line (line.split(' ')). Any line that raises during strip/decode/split — wrong encoding, more than one space, or split not yielding 3 values — is reported as an invalid POS dictionary entry with file, line number, and content.","triggerScenarios":"Initializing jieba.posseg with a custom dictionary whose lines aren't 'word TAG' (single space), contain multiple spaces, are non-UTF-8, or where the default dict.txt.pos is corrupted. Triggered on first posseg call or posseg.initialize(dictionary).","commonSituations":"Custom dictionaries generated with tabs or multiple spaces; GBK-encoded POS dicts; tabs instead of spaces; corrupted default dict.txt.pos after a bad install; reusing a plain jieba dict (no tag column) as a POS dict.","solutions":["Fix the reported line: exactly 'word TAG', single space separator, UTF-8, no BOM","Normalize the file programmatically: split on any whitespace and re-join with a single space","Restore the default dict.txt.pos via pip install --force-reinstall jieba","Never pass a plain frequency dict (word freq) to posseg.initialize — it must include POS tags"],"exampleFix":"# before\n# posdict line: '北京  ns  extra'  -> ValueError\njieba.posseg.initialize('posdict.txt')\n\n# after\nwith open('posdict.txt', encoding='utf-8') as f:\n    rows = [ln.split()[:2] for ln in f if ln.strip()]\nwith open('posdict_fixed.txt', 'w', encoding='utf-8') as f:\n    f.write('\\n'.join('%s %s' % (w, t) for w, t in rows))\njieba.posseg.initialize('posdict_fixed.txt')","handlingStrategy":"validation","validationCode":"import re\ndef posdict_ok(path):\n    pat = re.compile(r'^\\S+ [a-z]+\\s*$', re.IGNORECASE)\n    with open(path, encoding='utf-8') as f:\n        for i, ln in enumerate(f, 1):\n            if ln.strip() and not pat.match(ln):\n                return False, i, ln\n    return True, None, None","typeGuard":null,"tryCatchPattern":"try:\n    jieba.posseg.initialize(p)\nexcept ValueError as e:\n    # message contains file, line number, offending line\n    fix_pos_line(e); jieba.posseg.initialize(p)","preventionTips":["Generate POS dicts as 'word TAG' with exactly one space","Never reuse plain frequency dicts for posseg","Keep files UTF-8; validate in CI"],"tags":["jieba","posseg","dictionary","validation","unicode"],"backgroundTag":"malformed-data-file","analyzedSha":"67fa2e36e72f69d9134b8a1037b83fbb070b9775","analyzedAt":"2026-08-27T11:28:25.101Z","schemaVersion":2},"datasetVersion":"2026-08-27T13:17:12.746Z"}