{"record":{"id":"ab188859de33d92d","repo":"OpenBMB/VoxCPM","slug":"tokenization-failed-str-e","errorCode":null,"errorMessage":"Tokenization failed: {str(e)}","messagePattern":"Tokenization failed: (.+?)","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/voxcpm/model/utils.py","lineNumber":136,"sourceCode":"            but with multi-character Chinese token handling.\n\n            Args:\n                text: Input text to tokenize\n                **kwargs: Additional arguments passed to the base tokenizer\n\n            Returns:\n                List of token IDs\n\n            Raises:\n                TypeError: If input is not a string\n                ValueError: If tokenization fails\n            \"\"\"\n            try:\n                tokens = self.tokenize(text, **kwargs)\n                result = self.tokenizer.convert_tokens_to_ids(tokens)\n                return result\n            except Exception as e:\n                raise ValueError(f\"Tokenization failed: {str(e)}\") from e\n\n    return CharTokenizerWrapper(tokenizer)\n\n\ndef get_dtype(dtype: str):\n    if dtype == \"bfloat16\":\n        return torch.bfloat16\n    elif dtype == \"bf16\":\n        return torch.bfloat16\n    elif dtype == \"float16\":\n        return torch.float16\n    elif dtype == \"fp16\":\n        return torch.float16\n    elif dtype == \"float32\":\n        return torch.float32\n    elif dtype == \"fp32\":\n        return torch.float32\n    else:","sourceCodeStart":118,"sourceCodeEnd":154,"githubUrl":"https://github.com/OpenBMB/VoxCPM/blob/f5a1c6a6b901bc732e20f0d59a369f6829ad717a/src/voxcpm/model/utils.py#L118-L154","documentation":"CharTokenizerWrapper.__call__ wraps tokenize + convert_tokens_to_ids and re-raises any exception as ValueError('Tokenization failed: ...'), preserving the cause via raise-from.","triggerScenarios":"Any tokenizer-internal failure: unknown tokens that cannot convert to ids, encoding errors, or the TypeError from tokenize with non-str input bubbled through __call__.","commonSituations":"Corrupted/unusual unicode text, vocab mismatch after switching checkpoints, or non-string input slipping past earlier checks.","solutions":["Read the chained original exception (raise-from cause) for the real reason","Sanitize/validate text before calling the model","Ensure tokenizer and model checkpoint versions match"],"exampleFix":"# before\nids = tokenizer(text)\n# after\ntry:\n    ids = tokenizer(text)\nexcept ValueError as e:\n    raise RuntimeError(f\"tokenize failed for input: {text!r}\") from e","handlingStrategy":"try-catch","validationCode":"if not isinstance(text, str):\n    text = str(text)","typeGuard":null,"tryCatchPattern":"try:\n    ids = tokenizer(text)\nexcept ValueError as e:\n    cause = e.__cause__  # inspect real failure\n    log.error('tokenization failed: %s (%r)', e, text[:100])\n    raise","preventionTips":["Log the offending input snippet on failure","Keep tokenizer and checkpoint from the same release"],"tags":["tokenizer","error-chaining","value-error"],"backgroundTag":"tokenizer-failure","analyzedSha":"f5a1c6a6b901bc732e20f0d59a369f6829ad717a","analyzedAt":"2026-08-27T05:54:30.617Z","schemaVersion":2},"datasetVersion":"2026-08-27T08:17:20.692Z"}