OpenBMB/VoxCPM · error · ValueError

Tokenization failed: {str(e)}

Error message

Tokenization failed: {str(e)}

What it means

CharTokenizerWrapper.__call__ wraps tokenize + convert_tokens_to_ids and re-raises any exception as ValueError('Tokenization failed: ...'), preserving the cause via raise-from.

Source

Thrown at src/voxcpm/model/utils.py:136

            but with multi-character Chinese token handling.

            Args:
                text: Input text to tokenize
                **kwargs: Additional arguments passed to the base tokenizer

            Returns:
                List of token IDs

            Raises:
                TypeError: If input is not a string
                ValueError: If tokenization fails
            """
            try:
                tokens = self.tokenize(text, **kwargs)
                result = self.tokenizer.convert_tokens_to_ids(tokens)
                return result
            except Exception as e:
                raise ValueError(f"Tokenization failed: {str(e)}") from e

    return CharTokenizerWrapper(tokenizer)


def get_dtype(dtype: str):
    if dtype == "bfloat16":
        return torch.bfloat16
    elif dtype == "bf16":
        return torch.bfloat16
    elif dtype == "float16":
        return torch.float16
    elif dtype == "fp16":
        return torch.float16
    elif dtype == "float32":
        return torch.float32
    elif dtype == "fp32":
        return torch.float32
    else:

View on GitHub (pinned to f5a1c6a6b9)

Solutions

1. Read the chained original exception (raise-from cause) for the real reason
2. Sanitize/validate text before calling the model
3. Ensure tokenizer and model checkpoint versions match

Example fix

# before
ids = tokenizer(text)
# after
try:
    ids = tokenizer(text)
except ValueError as e:
    raise RuntimeError(f"tokenize failed for input: {text!r}") from e

if not isinstance(text, str):
    text = str(text)

try:
    ids = tokenizer(text)
except ValueError as e:
    cause = e.__cause__  # inspect real failure
    log.error('tokenization failed: %s (%r)', e, text[:100])
    raise

When it happens

Trigger: Any tokenizer-internal failure: unknown tokens that cannot convert to ids, encoding errors, or the TypeError from tokenize with non-str input bubbled through __call__.

Common situations: Corrupted/unusual unicode text, vocab mismatch after switching checkpoints, or non-string input slipping past earlier checks.

Related errors

• Unsupported architecture: {arch}
• Expected string input, got {type(text)}
• Unsupported dtype: {dtype}
• Unsupported device '{device}'. Supported values are 'auto',

AI-assisted analysis of OpenBMB/VoxCPM@f5a1c6a6b9 (2026-08-27). Data as JSON: /api/errors/ab188859de33d92d. Report an issue: GitHub.