{"record":{"id":"ae3f505611c52aaf","repo":"fxsjy/jieba","slug":"jieba-the-input-parameter-should-be-unicode","errorCode":null,"errorMessage":"jieba: the input parameter should be unicode.","messagePattern":"jieba: the input parameter should be unicode\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"jieba/__init__.py","lineNumber":486,"sourceCode":"            word = ''.join(segment)\n            for seg in segment:\n                freq *= self.FREQ.get(seg, 1) / ftotal\n            freq = min(int(freq * self.total), self.FREQ.get(word, 0))\n        if tune:\n            self.add_word(word, freq)\n        return freq\n\n    def tokenize(self, unicode_sentence, mode=\"default\", HMM=True):\n        \"\"\"\n        Tokenize a sentence and yields tuples of (word, start, end)\n\n        Parameter:\n            - sentence: the str(unicode) to be segmented.\n            - mode: \"default\" or \"search\", \"search\" is for finer segmentation.\n            - HMM: whether to use the Hidden Markov Model.\n        \"\"\"\n        if not isinstance(unicode_sentence, text_type):\n            raise ValueError(\"jieba: the input parameter should be unicode.\")\n        start = 0\n        if mode == 'default':\n            for w in self.cut(unicode_sentence, HMM=HMM):\n                width = len(w)\n                yield (w, start, start + width)\n                start += width\n        else:\n            for w in self.cut(unicode_sentence, HMM=HMM):\n                width = len(w)\n                if len(w) > 2:\n                    for i in xrange(len(w) - 1):\n                        gram2 = w[i:i + 2]\n                        if self.FREQ.get(gram2):\n                            yield (gram2, start + i, start + i + 2)\n                if len(w) > 3:\n                    for i in xrange(len(w) - 2):\n                        gram3 = w[i:i + 3]\n                        if self.FREQ.get(gram3):","sourceCodeStart":468,"sourceCodeEnd":504,"githubUrl":"https://github.com/fxsjy/jieba/blob/67fa2e36e72f69d9134b8a1037b83fbb070b9775/jieba/__init__.py#L468-L504","documentation":"jieba's tokenize (and anything using unicode positions) requires the sentence to be a unicode object (str on Python 3, unicode on Python 2). Passing bytes or a non-string raises ValueError immediately. The returned (word, start, end) offsets only make sense for unicode code points.","triggerScenarios":"Calling jieba.tokenize(u_string) with a bytes object (Python 3 str from open('rb'), network payloads, .encode()'d strings) or any non-string type. On Python 2, passing a UTF-8 byte str instead of unicode.","commonSituations":"Reading text as bytes (open without encoding on Python 2, 'rb' mode), web-scraped content from requests.content (bytes), or Python 2 code passing str where unicode is required.","solutions":["Decode before tokenizing: sentence.decode('utf-8') (Python 2) or pass the str directly in Python 3","If you have bytes in Python 3, do sentence.decode('utf-8') first","Wrap the call in a isinstance check against the library's text_type (six.string_types / str)"],"exampleFix":"# before\ntokens = jieba.tokenize(resp.content)  # bytes -> ValueError\n\n# after\ntokens = jieba.tokenize(resp.content.decode('utf-8'))","handlingStrategy":"type-guard","validationCode":"from six import text_type\nif not isinstance(sentence, text_type):\n    sentence = sentence.decode('utf-8')","typeGuard":"def is_text(s):\n    import sys\n    return isinstance(s, str if sys.version_info[0] >= 3 else unicode)","tryCatchPattern":null,"preventionTips":["Always decode bytes from files/network before passing to jieba","Open files with explicit encoding='utf-8'","Standardize on unicode/str at API boundaries"],"tags":["jieba","unicode","type-error","tokenize"],"backgroundTag":"wrong-argument-type","analyzedSha":"67fa2e36e72f69d9134b8a1037b83fbb070b9775","analyzedAt":"2026-08-27T11:28:25.101Z","schemaVersion":2},"datasetVersion":"2026-08-27T13:17:12.746Z"}