OpenBMB/VoxCPM · error · TypeError

Expected string input, got {type(text)}

Error message

Expected string input, got {type(text)}

What it means

CharTokenizerWrapper.tokenize requires a Python str; any other type (None, bytes, list) raises TypeError. This wrapper splits Hugging Face subword tokens into per-character tokens.

Source

Thrown at src/voxcpm/model/utils.py:96

            self.multichar_tokens = multichar_tokens

        def tokenize(self, text: str, **kwargs) -> List[str]:
            """Tokenize text and split multi-character Chinese tokens into single characters.

            Args:
                text: Input text to tokenize
                **kwargs: Additional arguments passed to the base tokenizer

            Returns:
                List of processed tokens with multi-character Chinese tokens split

            Example:
                >>> wrapper = CharTokenizerWrapper(tokenizer)
                >>> tokens = wrapper.tokenize("你好世界")
                >>> # Returns ["你", "好", "世", "界"] instead of ["你好", "世界"]
            """
            if not isinstance(text, str):
                raise TypeError(f"Expected string input, got {type(text)}")

            tokens = self.tokenizer.tokenize(text, **kwargs)
            processed = []

            for token in tokens:
                # Remove possible subword prefix
                clean_token = token.replace("▁", "")

                if clean_token in self.multichar_tokens:
                    # Split multi-character token into single characters
                    chars = list(clean_token)
                    processed.extend(chars)
                else:
                    processed.append(token)

            return processed

        def __call__(self, text: str, **kwargs) -> List[int]:

View on GitHub (pinned to f5a1c6a6b9)

Solutions

1. Ensure text is decoded to str before tokenization
2. Add an isinstance(text, str) guard upstream
3. Decode bytes with .decode('utf-8') before passing

Example fix

# before
tokens = tokenizer.tokenize(raw_bytes)
# after
tokens = tokenizer.tokenize(raw_bytes.decode("utf-8"))

assert isinstance(text, str) and text != ""

def is_str_input(x) -> bool:
    return isinstance(x, str)

When it happens

Trigger: Calling the wrapped tokenizer's tokenize/__call__ with non-string input, which propagates from _generate when text is mistyped (though core.py usually catches empty/None first).

Common situations: Programmatic pipelines feeding None or pre-tokenized lists into the tokenizer, or bytes from file reads not decoded.

Related errors

• target text must be a non-empty string
• Tokenization failed: {str(e)}

AI-assisted analysis of OpenBMB/VoxCPM@f5a1c6a6b9 (2026-08-27). Data as JSON: /api/errors/f9435d027d5749b9. Report an issue: GitHub.