fxsjy/jieba · error · ValueError

dictionary file %s must be utf-8

Error message

dictionary file %s must be utf-8

What it means

load_userdict tries to decode each line of a user dictionary as UTF-8; if decoding fails it raises ValueError stating the file must be UTF-8. User dictionaries must be UTF-8 encoded plain text, one 'word [freq] [tag]' per line.

Source

Thrown at jieba/__init__.py:407

        Structure of dict file:
        word1 freq1 word_type1
        word2 freq2 word_type2
        ...
        Word type may be ignored
        '''
        self.check_initialized()
        if isinstance(f, string_types):
            f_name = f
            f = open(f, 'rb')
        else:
            f_name = resolve_filename(f)
        for lineno, ln in enumerate(f, 1):
            line = ln.strip()
            if not isinstance(line, text_type):
                try:
                    line = line.decode('utf-8').lstrip('\ufeff')
                except UnicodeDecodeError:
                    raise ValueError('dictionary file %s must be utf-8' % f_name)
            if not line:
                continue
            # match won't be None because there's at least one character
            word, freq, tag = re_userdict.match(line).groups()
            if freq is not None:
                freq = freq.strip()
            if tag is not None:
                tag = tag.strip()
            self.add_word(word, freq, tag)

    def add_word(self, word, freq=None, tag=None):
        """
        Add a word to dictionary.

        freq and tag can be omitted, freq defaults to be a calculated value
        that ensures the word can be cut out.
        """
        self.check_initialized()

View on GitHub (pinned to 67fa2e36e7)

Solutions

  1. Convert the file to UTF-8: iconv -f GBK -t UTF-8 userdict.txt > userdict.utf8.txt and load the converted file
  2. Ensure the editor saves the file as 'UTF-8 (no BOM)' or strip the BOM (the code lstrips \ufeff only for valid UTF-8)
  3. If stuck with GBK data, decode/re-encode in Python before writing a temp UTF-8 file and loading it

Example fix

# before
jieba.load_userdict('words_gbk.txt')  # GBK-encoded -> ValueError

# after
import codecs
with codecs.open('words_gbk.txt', encoding='gbk') as src, \
     codecs.open('words_utf8.txt', 'w', encoding='utf-8') as dst:
    dst.write(src.read())
jieba.load_userdict('words_utf8.txt')
Defensive patterns

Strategy: validation

Validate before calling

def ensure_utf8(path):
    with open(path, 'rb') as f:
        f.read().decode('utf-8')  # raises if not UTF-8
    return path

ensure_utf8('userdict.txt')
jieba.load_userdict('userdict.txt')

Try / catch

try:
    jieba.load_userdict(path)
except ValueError:
    subprocess.run(['iconv', '-f', 'GBK', '-t', 'UTF-8', path, '-o', path + '.utf8'])
    jieba.load_userdict(path + '.utf8')

Prevention

When it happens

Trigger: Calling jieba.load_userdict(path) where the file is encoded in GBK/GB2312/Big5/Latin-1 or is binary. Python 2 byte strings that aren't valid UTF-8 also trigger it.

Common situations: Chinese word lists exported from Excel or Windows tools default to GBK; files created on Windows with ANSI encoding; downloading a userdict from the web that is actually GBK.

Related errors

AI-assisted analysis of fxsjy/jieba@67fa2e36e7 (2026-08-27). Data as JSON: /api/errors/c4c9f8a75981862d. Report an issue: GitHub.