{"record":{"id":"c4c9f8a75981862d","repo":"fxsjy/jieba","slug":"dictionary-file-s-must-be-utf-8","errorCode":null,"errorMessage":"dictionary file %s must be utf-8","messagePattern":"dictionary file (.+?) must be utf-8","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"jieba/__init__.py","lineNumber":407,"sourceCode":"        Structure of dict file:\n        word1 freq1 word_type1\n        word2 freq2 word_type2\n        ...\n        Word type may be ignored\n        '''\n        self.check_initialized()\n        if isinstance(f, string_types):\n            f_name = f\n            f = open(f, 'rb')\n        else:\n            f_name = resolve_filename(f)\n        for lineno, ln in enumerate(f, 1):\n            line = ln.strip()\n            if not isinstance(line, text_type):\n                try:\n                    line = line.decode('utf-8').lstrip('\\ufeff')\n                except UnicodeDecodeError:\n                    raise ValueError('dictionary file %s must be utf-8' % f_name)\n            if not line:\n                continue\n            # match won't be None because there's at least one character\n            word, freq, tag = re_userdict.match(line).groups()\n            if freq is not None:\n                freq = freq.strip()\n            if tag is not None:\n                tag = tag.strip()\n            self.add_word(word, freq, tag)\n\n    def add_word(self, word, freq=None, tag=None):\n        \"\"\"\n        Add a word to dictionary.\n\n        freq and tag can be omitted, freq defaults to be a calculated value\n        that ensures the word can be cut out.\n        \"\"\"\n        self.check_initialized()","sourceCodeStart":389,"sourceCodeEnd":425,"githubUrl":"https://github.com/fxsjy/jieba/blob/67fa2e36e72f69d9134b8a1037b83fbb070b9775/jieba/__init__.py#L389-L425","documentation":"load_userdict tries to decode each line of a user dictionary as UTF-8; if decoding fails it raises ValueError stating the file must be UTF-8. User dictionaries must be UTF-8 encoded plain text, one 'word [freq] [tag]' per line.","triggerScenarios":"Calling jieba.load_userdict(path) where the file is encoded in GBK/GB2312/Big5/Latin-1 or is binary. Python 2 byte strings that aren't valid UTF-8 also trigger it.","commonSituations":"Chinese word lists exported from Excel or Windows tools default to GBK; files created on Windows with ANSI encoding; downloading a userdict from the web that is actually GBK.","solutions":["Convert the file to UTF-8: iconv -f GBK -t UTF-8 userdict.txt > userdict.utf8.txt and load the converted file","Ensure the editor saves the file as 'UTF-8 (no BOM)' or strip the BOM (the code lstrips \\ufeff only for valid UTF-8)","If stuck with GBK data, decode/re-encode in Python before writing a temp UTF-8 file and loading it"],"exampleFix":"# before\njieba.load_userdict('words_gbk.txt')  # GBK-encoded -> ValueError\n\n# after\nimport codecs\nwith codecs.open('words_gbk.txt', encoding='gbk') as src, \\\n     codecs.open('words_utf8.txt', 'w', encoding='utf-8') as dst:\n    dst.write(src.read())\njieba.load_userdict('words_utf8.txt')","handlingStrategy":"validation","validationCode":"def ensure_utf8(path):\n    with open(path, 'rb') as f:\n        f.read().decode('utf-8')  # raises if not UTF-8\n    return path\n\nensure_utf8('userdict.txt')\njieba.load_userdict('userdict.txt')","typeGuard":null,"tryCatchPattern":"try:\n    jieba.load_userdict(path)\nexcept ValueError:\n    subprocess.run(['iconv', '-f', 'GBK', '-t', 'UTF-8', path, '-o', path + '.utf8'])\n    jieba.load_userdict(path + '.utf8')","preventionTips":["Configure editors/exports to always write UTF-8","For Excel exports, choose 'UTF-8 CSV' explicitly","Strip BOMs after conversion"],"tags":["jieba","encoding","utf-8","userdict"],"backgroundTag":"unicode-decode-error","analyzedSha":"67fa2e36e72f69d9134b8a1037b83fbb070b9775","analyzedAt":"2026-08-27T11:28:25.101Z","schemaVersion":2},"datasetVersion":"2026-08-27T13:17:12.746Z"}