What causes "jieba: the input parameter should be unicode." in fxsjy/jieba?

Decode before tokenizing: sentence.decode('utf-8') (Python 2) or pass the str directly in Python 3

fxsjy/jieba · error · ValueError

jieba: the input parameter should be unicode.

Error message

jieba: the input parameter should be unicode.

What it means

jieba's tokenize (and anything using unicode positions) requires the sentence to be a unicode object (str on Python 3, unicode on Python 2). Passing bytes or a non-string raises ValueError immediately. The returned (word, start, end) offsets only make sense for unicode code points.

Source

Thrown at jieba/__init__.py:486

            word = ''.join(segment)
            for seg in segment:
                freq *= self.FREQ.get(seg, 1) / ftotal
            freq = min(int(freq * self.total), self.FREQ.get(word, 0))
        if tune:
            self.add_word(word, freq)
        return freq

    def tokenize(self, unicode_sentence, mode="default", HMM=True):
        """
        Tokenize a sentence and yields tuples of (word, start, end)

        Parameter:
            - sentence: the str(unicode) to be segmented.
            - mode: "default" or "search", "search" is for finer segmentation.
            - HMM: whether to use the Hidden Markov Model.
        """
        if not isinstance(unicode_sentence, text_type):
            raise ValueError("jieba: the input parameter should be unicode.")
        start = 0
        if mode == 'default':
            for w in self.cut(unicode_sentence, HMM=HMM):
                width = len(w)
                yield (w, start, start + width)
                start += width
        else:
            for w in self.cut(unicode_sentence, HMM=HMM):
                width = len(w)
                if len(w) > 2:
                    for i in xrange(len(w) - 1):
                        gram2 = w[i:i + 2]
                        if self.FREQ.get(gram2):
                            yield (gram2, start + i, start + i + 2)
                if len(w) > 3:
                    for i in xrange(len(w) - 2):
                        gram3 = w[i:i + 3]
                        if self.FREQ.get(gram3):

View on GitHub (pinned to 67fa2e36e7)

Solutions

Decode before tokenizing: sentence.decode('utf-8') (Python 2) or pass the str directly in Python 3
If you have bytes in Python 3, do sentence.decode('utf-8') first
Wrap the call in a isinstance check against the library's text_type (six.string_types / str)

Example fix

# before
tokens = jieba.tokenize(resp.content)  # bytes -> ValueError

# after
tokens = jieba.tokenize(resp.content.decode('utf-8'))

Defensive patterns

Strategy: type-guard

Validate before calling

from six import text_type
if not isinstance(sentence, text_type):
    sentence = sentence.decode('utf-8')

Type guard

def is_text(s):
    import sys
    return isinstance(s, str if sys.version_info[0] >= 3 else unicode)

Prevention

Always decode bytes from files/network before passing to jieba
Open files with explicit encoding='utf-8'
Standardize on unicode/str at API boundaries

When it happens

Trigger: Calling jieba.tokenize(u_string) with a bytes object (Python 3 str from open('rb'), network payloads, .encode()'d strings) or any non-string type. On Python 2, passing a UTF-8 byte str instead of unicode.

Common situations: Reading text as bytes (open without encoding on Python 2, 'rb' mode), web-scraped content from requests.content (bytes), or Python 2 code passing str where unicode is required.

Related errors

AI-assisted analysis of fxsjy/jieba@67fa2e36e7 (2026-08-27). Data as JSON: /api/errors/ae3f505611c52aaf. Report an issue: GitHub.